Measuring Outputs Expanded

I talked at a high level in my Measure Outputs post about the types of measures that are useful. That post stated broadly that input measures do little to help improve, control or predict performance. In this post, I will expand on that concept to cover in more depth what types of measures focus on output and help drive process improvement.

Perhaps the best known framework for creating measures is the balanced score card. This sets up categories of metrics that support an organizations strategies and vision. From the Balanced Score Card Institute, this framework is shown to the right. I find it useful to apply it to business processes as well as overall strategy. Even though processes are just one component of the model, they ultimately support part of a business’ strategy. A framework helps to ensure we considering measures that contribute to the overall strategies and goals the process supports.

Once a framework is chosen, the next thing to consider is the customers and stakeholders that the process serves and what they value. This identification is important to ensure we know the objectives that the process is to support. If the stakeholders or customers tell us they want high performance, personalized, niche process (a BMW) the measures will be different than if they want a quality, efficient, repeatable process (a Toyota.)

The next question is what measures are valuable. I continue to see the same types of measures used across process improvement projects. For each category of the framework, there are common recurring measures that are useful:

Process: There are three key measures that are critical to almost all processes: Productivity, Quality and Cycle Time.

Productivity: Always measured as Output / Input. I once saw a company that measured productivity as the % of time spent on coding. I get the thought, but it is only half the equation and, frankly, the less important half.

Output: Products, features, backlog or other work units should have standard “story point” or relative size estimates assigned. This normalizes the output across different work effort or output value. It also helps to facilitate planning so that resources can be assigned to work based on size and with pricing so that appropriate prices are assigned per output unit.

Input: Measures the capacity of available resources used to create the outputs. Be comprehensive and don’t give resources credit for working off task. If you are paying for the resources, they should count against output even if not always focused directly on it. Also, you should be able to factor out overtime from this component of the measure as it is not sustainable capacity.

In Agile development, Velocity or BurnDown is a Productivity measure. You may say it only shows output, but since capacity is held steady by a dedicated team, only the output need be measured.

Quality: Repeat after me… “Never use defects found as a quality measure.” You want to find defects. The way to find them is to ensure quality is built-in through proactive testing.  The output measure for quality is the testing done to support acceptance.

Tests planned, run and passed is a proactive measure that attempts to ensure that the quality of a process is planned and built into the delivery. The testing process defines acceptance criteria and should be built into construction of deliverables and happen at various stages with the earliest coming during early requirements and build. The only rules here are from test driven development: the test should only test one criteria, it should only fail for that criteria, and it should be the only test that fails for that criteria.

OK, one exception to the mantra: Measure defects found by the customer. Track these with a vengeance and apply process and measures to them to ensure they are adequately corrected and customers stay satisfied.

Cycle Time: Cycle time is the total time from the beginning to the end of your process, as defined by you and your customer. Cycle time includes process time and time spent waiting to take the next action. It is a measure of time from request to delivery or concept to cash. It includes not just how fast development happens, but the time it takes to capture, prioritize, design, develop, test and deploy functionality and all the time in between. Ensure your cycle time measure is expansive or you will sub-optimize the overall process. Also, realize that not all requests are created equally. Ensure those that have the most value get priority treatment. The features in your backlog that are low priority may deserve to grow old on the backlog and eventually drop off.  Or perhaps there is value in tracking cycle time for bug fixes vs enhancements vs new features.

As long as your prioritization process is fast and exiting the backlog requires deployment, the average age your backlog is an easy proxy for cycle time.

My First Law of Development is about maximizing productivity while minimizing cycle time if you’d like a deeper read.

Some organizations use On-time Delivery as a proxy for cycle time. While this is an OK measure for customer satisfaction, it does little to report the health of a process. If important to your customers, use it as a customer measure, but also use cycle time as a process measure.

Customer: Often, quality is used as a proxy for customer measures. This can have the unintended effect of missing other critical voice of customer measures. The best way to ensure the customer is satisfied, is to ask them. Simple surveys or votes are perhaps the best way to capture this measure. Ask customers about what is important to them. There are many online survey tools that will allow you to ask relevant questions from your customers and track trends over time. Or, use a net promptor score by simply asking “How likely are you to recommend our product / service.”

Mike Griffiths has an eloquent sponsor/ customer tracking approach that asks for a  vote of confidence or satisfaction. In aggregate, this provides a customer-focused index over time. I also like the emoticons in his customer chart.

Learning: Learning should measure the capabilities of the team members. A matrix of who can use what tools and develop what products or product components with a level of expertise assigned is a good starting point for tracking this measure.
% coverage becomes a good aggregation of this measure. Be sure to include current skills as well as upcoming skills and both soft and hard skills in the matrix.
(From The Lean Six Sigma Pocket Toolbook: A Quick Reference Guide to 100 Tools for Improving Quality and Speed, Mike George, et al)

Financial: As high-level financial measures go, I have a pre-disposition to ROIC. It balances profitability with the investment costs required to generate it – and is a topic for another post. Most processes will not require this comprehensive of a measure, so simple cost tracking usually suffices. This can usually be further simplified to time tracking since development is largely made up of salary costs. Make sure you at least consider equipment and other capital and marketing costs before you settle in on time tracking as your financial measure. Normalizing financial tracking to earned value or planned spend helps to show tolerances of spend on a project (and display productivity with the financial measures.)

PS: I have intentionally tried to make this post very visual. The key to making measures useful is to make them easily understood and highly visible and public.