Counting What Matters

Think before you count.

Published: Tuesday, July 10, 2007 - 22:00

What data are the best to gather? What processes should you be tracking? What are your metrics telling you?

In observing various organizations’ attempts to fulfill ISO 9001 requirements around subclause 8.4—“Analysis of data,”–I’ve noticed a recurring problem. Companies allow the requirements of the standard—ISO 9001 or any comparable quality management system (QMS) model—to drive the activity. They put together what appears to be a reasonable list of what to monitor and then proceed to amass enormous amounts of data.

There are some perennial heavy hitters, such as on-time delivery, customer returns, and scrap. There’s no denying that in the typical organization these are always fairly good indicators. Are they the best indicators for your organization?

Gathering data you don’t care about expends precious resources, clogs the information network, and diffuses focus from important issues. In short, it’s a waste of time. Looking at the wrong data also risks ignoring real issues that are at the root of serious problems, while you fiddle and diddle with pretty charts you think will make an auditor happy. Over the long haul, it can erode commitment to consistent use of monitoring and analytical methodologies that really can benefit your organization.

Analysis of data should be driven by one question: “What is it that you need to know?”

The answer to that question is as diverse as the myriad organizations that populate the globe. What you need to know depends on who you are. That is, it depends on the nature of your organization, including such factors as products, market, size, and complexity of processes.

For example, a company for which I did some consulting work several years ago calculated their production yield by determining how much power was used per unit. The time it took to manufacture a part was negligible and the base cost of raw materials per unit was a fraction of a penny. The most significant cost associated with producing a part was the cost of energy—electricity and fuel. The prorated utilities costs expended whenever the production machinery was engaged was an accurate indicator of resource expenditure. When divided by total salable pieces produced per hour, it provided the best metric of production efficiency.

Calculating the lost time per unit or the scrapped raw material may have been a legitimate metric, but it wasn’t reflective of the biggest drain on the organization’s resources. In this scenario, focusing on scrap would have been nonsensical. It would have also perpetuated the notion that QMS requirements are a pencil-pushing exercise. To address inefficiencies, the organization is better served investigating the root cause of machine failures that result in nonsaleable units.

I’ve put together a chart below, with organizations on the left and appropriate metrics on the right. The challenge is to match the column on the right to the best choice for most critical performance indicator in the other column.

Medical lab	Zero defects
Software design firm	Incident-free days
Medical device manufacturer	Successfully resolved customer issues
Nuclear plant	On-time delivery of test results to medical facilities
Call center	Units sold
Mass producer of widgets	Projects completed on target

While an argument can be made that several metrics will matter to any organization, it’s fair to conclude that zero defects is a much larger concern in the medical device industry than it is in the widget business. In the former, the risk of even one bad unit, especially if it’s an implantable device, is a dead customer.

In the latter example, zero defects could put the widget maker out of business. They may have determined an acceptable quality level (AQL) of 1-percent defective parts. The cost to get closer to zero would erode the already tight margins found in many commodity industries.

Similarly, software developers are in a fast-paced industry, where it’s almost impossible to put out a product that is 100-percent bug-free. If they tried, the competition would beat them to the marketplace every time. The best they can do is release a product that has no major glitches and only insignificant hiccups that may not ever be detected by the casual user.

Finally, a metric like “parts per billion” is laughable if you produce only two units—such as aircraft carriers—per year.

It’s also important to consider that what organizations need to know may change over time. If there’s a focused project to decrease turnaround time for quoting new jobs, the need to actively monitor and analyze data may disappear once the process stabilizes.

So, deciding what to measure and how to measure it requires as much diligence as the actual activity, if the information you’re going to derive is to be meaningful and beneficial. Think before you count.

As to the matching exercise, these are my results. Yours may differ—you get to decide.

Medical lab	On-time delivery of test results to medical facilities
Software design firm	Projects completed on target
Medical device manufacturer	Zero defects
Nuclear plant	Incident-free days
Call center	Successfully resolved customer issues
Mass producer of widgets	Units sold

About The Author