Cloud Interact Community

 

 

Metrics Categories

• Accountability
• Agility
• Assurance
• Financial

    ⇨  Performance

• Security and Privacy
• Usability

 

The GSA Cloud Products Team continues its tour of Scott Feuless’ Cloud Service Evaluation Handbook. This feature will cover the performance metrics of cloud computing, allowing you to answer questions such as:

 

• How do you rank your different functional needs when choosing a cloud service provider (CSP)?
• How do you measure the functionality you are acquiring against the relative ease or difficulty you’ll have in integrating it? 
• How important is interoperability between your systems and services and your new CSP?
• How important is service response time to your agency’s cloud need?

Functionality 

 

Functionality should  be your first priority when it comes to your cloud acquisition. It is easy to get caught up with a cloud service provider’s bells and whistles. 

 

However, if your acquisition doesn’t accomplish its core mission and make things easier for your end user, you’ve wasted your time. 

 

Your chosen CSP’s features and functions are only as useful to you the agency as the customer says they are.

 

To establish a functionality metric:

 

1. List everything you need in detail. 
2. Categorize each item by priority. Feuless suggests labelling them as must have, useful, or future potential. 
3. Create a matrix of CSPs to reveal how their offerings compare to your needs.

 

Consult the specific CSP’s prospective users when making these determinations. This means inviting infrastructure management teams for Infrastructure as a Service (IaaS), developers for Platform as a Service (PaaS), and end users of Software as a Service (SaaS) to evaluation sessions. (Feuless notes that customers seeking SaaS solutions generally rate this metric higher in priority, as each CSP usually comes with its own application.)

 

Feuless ranks all of his metrics on a 0 -10 scale. For example, if a CSP does not offer any must-have functions, it will receive a 0. If a CSP offers all must have and useful functions and at least one future potential function, then it would receive a 10. 

Interoperability

 

Interoperability is “the ease and effectiveness with which the functionality of the [cloud] service can be integrated with other systems and services.” 

 

Ideally, separate IT systems should work together as seamlessly as possible. The interoperability metric encompasses how well cloud solutions communicate with both existing and future systems and services. It is especially important for the following considerations:

 

• Higher levels of interoperability translate to lower service migration costs. 
• If you’re exploring opportunities in a community cloud, you can use shared services. If you’re looking at hybrid cloud solutions, they’ll inherently rely on interoperability to communicate processes and data among your systems, CSPs, and existing IT infrastructure.  
• Should the worst happen and an entire service or service location become unusable, you’ll want to have set up disaster recovery backup via a different service and provider. 

 

In any of these scenarios, you’ll want to be able to easily adjust your configuration to enable communication. This metric contains three submetrics: inter-service communications, functional compatibility, and management orchestration.

Inter-service communications

 

Inter-service communications is “the ability to trigger events in and exchange information with other systems and services.” Service providers use diverse formats, protocols, and specifications to make inter-service communication possible. They use established standards such as:

 

• APIs and API protocols
• Communication protocols like HTTP, SMTP, and IPSec
• Data formats like XML, JSON, and SAML

 

The variety of formats and protocols CSPs use require extra scrutiny, since both your systems must integrate and communicate with as little friction as possible. 

 

Inter-service communications is the most critical of the three elements of interoperability. The following two won’t be worth much if the systems can’t talk to each other. 

 

So how do we weigh inter-service communications? As always, its importance depends on your agency’s priorities. Feuless suggests that technical staff and programming teams developing cloud applications enter the conversation early on for this reason. The group should conduct a careful analysis of their current IT portfolio: 

 

1. List all the systems and services with which a potential solution will need to communicate. 
2. Determine how each item will or could communicate with the new service. 
3. Decide which items must communicate with the new service now and which may wait.
4. Identify the standard protocols, formats, and other communication methods your agency supports today and apply what you learned from Step 2 to predict what will be needed in the future. Add these future needs to your requirements.
5. Research the information from each evaluated service to see which formats, protocols, and specifications it supports. 
6. Create a matrix.

 

This procedure resembles Application Rationalization: identify your needs, inventory the apps you have, assess the business value and technical fit these apps will have with the potential CSP. Then score each CSP based on how well it meets your requirements.  

 

Once you’ve compiled the information from each CSP, assign it a score between 0-10 based on how many services and systems it can communicate with now and how many it is expected to be able to communicate with in the future. More formats and protocols equates to more interoperability. More interoperability is always better than less.

 

Functional compatibility

 

Inter-service communications determine whether systems can communicate; functional compatibility describes the degree to which such systems are positioned to perform together. 

 

A major benefit of interoperability is achieving the most functionality by leveraging multiple systems or CSPs.  Some places where you can apply the functional compatibility metric include: 

 

• Relative levels of security between systems
• Relative compliance needs between systems
• Relative elasticity between systems
• Resource utilization between systems
• Developer knowledge and training in multiple systems
• Relative schedule and levels of support between systems

 

To score functional compatibility, use the 6-Step method we introduced in the inter-service communications section. List the systems and services that will need to integrate, then identify the functional compatibility requirements of each to achieve interoperability. Then assign scores based on what dimensions matter to you most.

 

Management Orchestration

 

The last submetric, management orchestration, is the ability to take the various financial and operations monitoring tools of different systems and services, and integrate them into an easy to understand dashboard . 

 

When management tools and platforms are perfectly orchestrated, all users need is a single interface to effectively manage the environment. For example, having an interoperable financial interface means that though you may have multiple providers furnishing several different services, you still only receive one itemized bill per billing period. 

 

When you as a cloud service customer have integrated views into your operations and financials, quality of service goes up and admin costs go down. 

 

As with the previous two submetrics, list the systems and services that you want to work together. Include those that are relevant to monitoring your system’s health and performance, operations, resource consumption, and billing. 

 

Next, identify how much orchestration you want. See if you’ll need a third party tool to get the dashboard views you desire. 

 

Last, repeat the rationalization steps from above and assign scores based on what dimensions matter to you most. 

 

Cloud system administrators will usually have the right knowledge to guide you through this process, but thoroughly do your research on each CSP as to their management orchestration capabilities.

 

Service Response Time

 

The third main performance metric is service response time, which is “the expected time that the service will take to respond to an input after completing one or more actions corresponding to that input.” 

 

There are two types of response time:

 

• Average response time: the average time a service needs to do a specific action over a known interval of time.
• Threshold response time: a preset amount of time set by a customer or vendor fixed to measures the percentage of actions taking longer than it.  

 

To clarify, these “actions" usually describe either transaction action types or application action types. 

 

Transaction response times are central. They refer to data manipulations in databases from an application’s request to completion. Transactions vary widely depending on a request’s complexity, and depend on the timing workload of the rest of the infrastructure, so Feuless logically recommends taking the average time it takes all actions to complete over the course of a chosen time interval.  

 

Application response times are end-to-end (from pushing buttons to seeing the result on a screen), measured internally from the application, and only relevant to SaaS services. They include time from network connections between end users and the application infrastructure (unlike transaction times). You’ll need special software to measure application response time.

 

Typically, a CSP customer chooses one of these as a way to measure average response times, but using both can be useful also. Average response time is better measured when all transactions have similar levels of importance. Threshold is generally more useful when your agency is more interested in cutting down on as many long response times as possible, as opposed to overall service responsiveness. 

 

Your decision  depends on what your agency is more interested in seeing and how your service-level agreement (SLA) targets with your CSP are phrased. Use at least six months of data to evaluate this metric, whether sourced from your own monitoring and logging tools, or relayed from another user of the CSP. 

 

Follow these steps to evaluate this metric:

 

1. Identify the actions that will have response times critical to achieving your mission and meeting your business goals.
2. Give a maximum acceptable response time to each of the actions listed in Step 1. (For threshold response time, assign a desired response time threshold for each as well as a maximum percentage of actions that you would tolerate exceeding your threshold time.)
3. If this is an existing service that is already on your network, gather the average response times (or the % of actions with response times above the threshold) for each action in step 1 for the past 6 months. 
4. If it is not an existing service, either do a benchmark test as part of the CSP’s evaluation or gather historical performance data from other customers of the CSP and average their response times (or the percentage of actions with response times above the threshold) for each action. If neither of those options are available, pinpoint SLA targets that the CSP should meet for each of your critical actions.
5. Ensure that the CSP has the right staff and technology to meet their promised SLA targets. 
6. Form a matrix.
7. Make your choice. 

 

Service response times will have varying importance to customers, but it will probably be most important to your agency if you combine your high volumes of data and many transactions, and also emphasize  customer service.

Conclusion

 

More functionality, higher levels of interoperability, and better response times are all objectively good things. While it’s tempting to spring for the shiniest CSP that promises the best performance, you need to decide on what metrics matter most to meeting your organizational goals. Thoroughly analyze and evaluate these metrics for any potential CSP, and you are much more likely to make the best choice for a cloud contract.