SaaS Evaluation

Introduction

With more and more companies migrating to the cloud and implementing SaaS technology, the market is quickly adjusting to the increased demand by constantly offering new SaaS products that seem similar in function and features. However, if you look closely, “minor” details like an add-on feature, the subscription and renewal terms, or a security policy can determine whether the software is adding value to your company or just adding to a pile of hidden costs.

So, in order to choose the right cloud-based software that meets your organization’s needs, it is extremely important to conduct a proper SaaS evaluation before onboarding a new vendor. An evaluation starts when an employee has a software request or there is a new business need. Going through the evaluation process will ensure that aspects like functionality, compliance, security demands, service, and cost are met, saving you from future headaches that could have been avoided with a good SaaS evaluation matrix.

Read on and find out how SaaS is evaluated before procurement and how to create your own SaaS evaluation matrix.

How to evaluate a SaaS application?

With increasing digital literacy in any company, shadow IT has become a real issue within many organizations. This lack of oversight also leads to multiple SaaS products with overlapping functionalities.

Before SaaS procurement, make sure that the evaluated product isn’t already being used in your company or whether there is a similar one that could satisfy the respective business need. Sometimes, existing solutions let you add on additional features for a small price increase. In case there is still a need for the new SaaS application, you should establish an evaluation framework that will come in handy for any future SaaS evaluations.

The scope of evaluation depends on the specific tool, business process change, and the data that is processed and stored. Therefore, evaluations can be separated into two scopes:

Limited scope evaluations: Applications that fit into this scope don't affect business processes for the whole organization or departments. These apps are only used by one or a few employees (e.g. social media scheduling app,  educational app, etc.)

These evaluations require IT and InfoSec teams to be involved but without the finance team or department leaders since the price tag and risk are usually lower. Below are some basic SaaS information requirements that make the process much easier when an employee requests a new limited scope SaaS evaluation:

• Name of application
• Business purpose
• Contact for approval
• Impact of leak
• Impact of alter/deletion
• Impact when not available
• Due date

Full scope evaluations: Applications that fit into this scope do affect business processes and are used across the organization or within departments (e.g. HR systems, ERPs, CRMs, content management systems, etc.)

During the full scope evaluation stage, it is vital to involve the IT, finance, and information security teams together with end-users (e.g. for marketing applications it can be CMO, director, or managers). They should receive all the relevant information to ensure a swift Software-as-a-Service evaluation.

Each evaluation should follow a risk-based approach to ensure compliance with IT contractual requirements and industry best practices.

The more information your teams receive, the easier and faster the evaluation can be done. Note that the amount of effort and information should align with the criticality and price point of the software product.

In the following, we are detailing 3 critical SaaS selection steps.

1. Involve other stakeholders

As you can imagine, procuring SaaS single-handedly without consulting other stakeholders like InfoSec and legal teams, procurement or IT is not the best idea. After all, unknown SaaS exposes companies to a whole new set of risks that need to be addressed before it’s too late in the game. Always ask yourself who the end-user is and how they will benefit from the new software application.

Whether it’s through surveys, brainstorming sessions, or one-on-one feedback – understanding other stakeholders, their pain points, and the rules they need to comply with, helps speed up the software application evaluation process and the development of a sound evaluation framework. In order to facilitate future changes, negotiations, and administration of the SaaS application, it is also vital to establish the owner of the SaaS product.

2. Determine the purpose of the new SaaS

Needless to say, knowing the business purpose of a new SaaS application is important. After all, understanding the “why” will make it a lot easier to tackle the “how” and thus, eliminate all irrelevant options. Let the following questions guide you through this stage:

• Why do we need this SaaS? / Which business need will it address?
• Who will lead the SaaS evaluation?
• What is the proposed timeline?
• What approvals are needed for the software application evaluation?

As previously mentioned, there might be a good chance that your organization is already using a similar app that could be modified to meet new business needs. Next, learn how to create a software evaluation matrix.

3. Create a SaaS evaluation matrix

In order to be able to properly evaluate different SaaS products and their vendors, it’s always helpful to create an evaluation matrix that contains all the SaaS evaluation criteria that matter to the health and prosperity of the organization. Each SaaS option that is short-listed for procurement should receive scores based on its:

1. Essential features
2. Security
3. Service and support
4. Costs

Create a scorecard for each vendor that rates the application on each criteria and assign weights to each one.

It’s important to note that categories from 1 to 4 contain sub-categories as well. For security, you could add the items SOC 2 compliance, GDPR compliance, Single Sign-On Integration, and Multi-factor authentication.

Rate your SaaS vendor for these features and assign the urgency/importance to each criteria as one aspect might be more important than another one. This will help you get the best vendor for your needs.

Below is a formula and a first look at the evaluation template that you can download at the bottom of this page. You can use this formula to calculate a weighted score for each vendor you're evaluating:

Vendor's Grade x Urgency = Weighted score (Vendor Assessment)

Next, we’ll teach you how to establish the SaaS evaluation criteria that belong in your software evaluation matrix.

Identify essential SaaS application features

When it comes to software, there are some features that are nice to have and others that you probably shouldn’t budge on. In order to find out whether the evaluated software meets your SaaS evaluation criteria in terms of functionality, you should clearly define all must-have features beforehand with the application end-users. These can vary from company to company, so make sure to keep in mind individual workflows and what kind of tools teams are currently using to perform a certain task.

Example: If you are looking for new e-commerce software, then a checkout system, a shopping cart module, and a nice gallery layout for your products are must-have features. Instead of getting too caught up in what is best for your teams, don’t forget to ask yourself what the end-user needs and which features would create the best user experience. This is where survey results from the stage “Involve other stakeholders” come in handy. It’s okay to spend a good amount of time on this exercise as it lays the foundation for the following steps.

Risk & security assessment

SaaS is convenient and scalable but it also removes physical security barriers that protect your data when it's stored on-premise. So, it doesn’t matter how many great features a SaaS application has, if it’s not compliant with security certifications or doesn’t meet the regulatory standards of your company, it puts your organization’s health at risk. That’s why the security of SaaS solutions should be the most important aspect of your evaluation and why working with InfoSec is a must. Before you add security criteria to your software evaluation matrix, you can ask yourself the following questions:

• Where is the data being stored?
• Which security measures prevent common cyber-attacks?
• Does the vendor meet the security standards of your company?
• Is the vendor compliant with SOC2, HIPAA, or GDPR?
• Is the vendor using open source code/practices open-source code hygiene?
• Has the vendor changed certifications in the past?

You should only move forward with procurement once you’ve performed a proper SaaS risk assessment and receive the approval of your internal security teams.

Evaluate vendor's service and relationship

When you acquire a new SaaS solution you are also getting into a new business relationship with the respective vendor. Ideally, you want to build a lasting relationship that will benefit you for many years to come. Here are some questions you can ask yourself when evaluating a vendor in terms of service and availability:

• How established and reliable is the vendor based on existing customer reviews?
• What kind of deployment options does the vendor offer besides SaaS?
• What does the customer support look like and is it included in the price or paid?
• Does the vendor offer additional (free) training for more complex tools?
• How and when does the vendor perform software maintenance
• Does the vendor offer automated monthly reporting?

This will help you in determining whether the SaaS vendor can offer you and your business the support and value you are looking for right now and in the future.

Calculate the total cost of ownership of SaaS

Your cost section in the software evaluation matrix looks at all the different factors that make up the SaaS total cost of ownership or SaaS TCO. This includes the following:

• Project initiation costs: Whenever a legacy system is replaced, you need to include the retiring cost and the cost of keeping it running until it can be replaced.
• Initial setup costs: This captures the financial effort needed to set up the new tool and also covers expenses like new hardware and software acquisitions as well as the implementation itself.
• Ongoing costs: This addresses the cost of maintaining and operating the SaaS application year-upon-year, including subscription costs, technical support, and training.
• Projected costs: Generate a projection for the next 5 years based on the setup costs and the ongoing costs.
• License and renewal: This refers to the cost incurred through adding licenses or software renewal.

Conclusion

Once you’ve settled on an application after it passed your SaaS evaluation meeting all important criteria, it’s time to negotiate the SaaS contract, so you are getting the maximum value. As the market for cloud-based solutions is quite competitive, vendors are often willing to offer you more favorable conditions than their competitors.

After procurement and onboarding, make sure to perform annual reviews and assessments to keep an eye on pricing changes as well as contract obligations and to check whether compliance and security demands are met. You might even find that you need fewer seats or licenses and can re-negotiate the conditions before a software renewal takes place.