Metrics Categories
⇨ Performance
The GSA Cloud Products Team continues its tour of Scott Feuless’ Cloud Service Evaluation Handbook. This feature will cover the performance metrics of cloud computing, allowing you to answer questions such as:
Functionality should be your first priority when it comes to your cloud acquisition. It is easy to get caught up with a cloud service provider’s bells and whistles.
However, if your acquisition doesn’t accomplish its core mission and make things easier for your end user, you’ve wasted your time.
Your chosen CSP’s features and functions are only as useful to you the agency as the customer says they are.
To establish a functionality metric:
Consult the specific CSP’s prospective users when making these determinations. This means inviting infrastructure management teams for Infrastructure as a Service (IaaS), developers for Platform as a Service (PaaS), and end users of Software as a Service (SaaS) to evaluation sessions. (Feuless notes that customers seeking SaaS solutions generally rate this metric higher in priority, as each CSP usually comes with its own application.)
Feuless ranks all of his metrics on a 0 -10 scale. For example, if a CSP does not offer any must-have functions, it will receive a 0. If a CSP offers all must have and useful functions and at least one future potential function, then it would receive a 10.
Interoperability is “the ease and effectiveness with which the functionality of the [cloud] service can be integrated with other systems and services.”
Ideally, separate IT systems should work together as seamlessly as possible. The interoperability metric encompasses how well cloud solutions communicate with both existing and future systems and services. It is especially important for the following considerations:
In any of these scenarios, you’ll want to be able to easily adjust your configuration to enable communication. This metric contains three submetrics: inter-service communications, functional compatibility, and management orchestration.
Inter-service communications is “the ability to trigger events in and exchange information with other systems and services.” Service providers use diverse formats, protocols, and specifications to make inter-service communication possible. They use established standards such as:
The variety of formats and protocols CSPs use require extra scrutiny, since both your systems must integrate and communicate with as little friction as possible.
Inter-service communications is the most critical of the three elements of interoperability. The following two won’t be worth much if the systems can’t talk to each other.
So how do we weigh inter-service communications? As always, its importance depends on your agency’s priorities. Feuless suggests that technical staff and programming teams developing cloud applications enter the conversation early on for this reason. The group should conduct a careful analysis of their current IT portfolio:
This procedure resembles Application Rationalization: identify your needs, inventory the apps you have, assess the business value and technical fit these apps will have with the potential CSP. Then score each CSP based on how well it meets your requirements.
Once you’ve compiled the information from each CSP, assign it a score between 0-10 based on how many services and systems it can communicate with now and how many it is expected to be able to communicate with in the future. More formats and protocols equates to more interoperability. More interoperability is always better than less.
Inter-service communications determine whether systems can communicate; functional compatibility describes the degree to which such systems are positioned to perform together.
A major benefit of interoperability is achieving the most functionality by leveraging multiple systems or CSPs. Some places where you can apply the functional compatibility metric include:
To score functional compatibility, use the 6-Step method we introduced in the inter-service communications section. List the systems and services that will need to integrate, then identify the functional compatibility requirements of each to achieve interoperability. Then assign scores based on what dimensions matter to you most.
The last submetric, management orchestration, is the ability to take the various financial and operations monitoring tools of different systems and services, and integrate them into an easy to understand dashboard .
When management tools and platforms are perfectly orchestrated, all users need is a single interface to effectively manage the environment. For example, having an interoperable financial interface means that though you may have multiple providers furnishing several different services, you still only receive one itemized bill per billing period.
When you as a cloud service customer have integrated views into your operations and financials, quality of service goes up and admin costs go down.
As with the previous two submetrics, list the systems and services that you want to work together. Include those that are relevant to monitoring your system’s health and performance, operations, resource consumption, and billing.
Next, identify how much orchestration you want. See if you’ll need a third party tool to get the dashboard views you desire.
Last, repeat the rationalization steps from above and assign scores based on what dimensions matter to you most.
Cloud system administrators will usually have the right knowledge to guide you through this process, but thoroughly do your research on each CSP as to their management orchestration capabilities.
The third main performance metric is service response time, which is “the expected time that the service will take to respond to an input after completing one or more actions corresponding to that input.”
There are two types of response time:
To clarify, these “actions" usually describe either transaction action types or application action types.
Transaction response times are central. They refer to data manipulations in databases from an application’s request to completion. Transactions vary widely depending on a request’s complexity, and depend on the timing workload of the rest of the infrastructure, so Feuless logically recommends taking the average time it takes all actions to complete over the course of a chosen time interval.
Application response times are end-to-end (from pushing buttons to seeing the result on a screen), measured internally from the application, and only relevant to SaaS services. They include time from network connections between end users and the application infrastructure (unlike transaction times). You’ll need special software to measure application response time.
Typically, a CSP customer chooses one of these as a way to measure average response times, but using both can be useful also. Average response time is better measured when all transactions have similar levels of importance. Threshold is generally more useful when your agency is more interested in cutting down on as many long response times as possible, as opposed to overall service responsiveness.
Your decision depends on what your agency is more interested in seeing and how your service-level agreement (SLA) targets with your CSP are phrased. Use at least six months of data to evaluate this metric, whether sourced from your own monitoring and logging tools, or relayed from another user of the CSP.
Follow these steps to evaluate this metric:
Service response times will have varying importance to customers, but it will probably be most important to your agency if you combine your high volumes of data and many transactions, and also emphasize customer service.
More functionality, higher levels of interoperability, and better response times are all objectively good things. While it’s tempting to spring for the shiniest CSP that promises the best performance, you need to decide on what metrics matter most to meeting your organizational goals. Thoroughly analyze and evaluate these metrics for any potential CSP, and you are much more likely to make the best choice for a cloud contract.
Views: 1350