Data Analysis—10 Key Questions and Reasons

Published: Friday, July 10, 2009 - 11:00

It is widely known among quality and process improvement practitioners that the lack of a clearly defined scope or charter is perhaps the leading cause for projects not getting started or completed on time and within budget. What are other causes? From my experience, the No. 2 cause for restarting process improvement projects is poor data. Without verifying the integrity of the data, project results can be meaningless.

In Donald J. Wheeler’s “Probability Models Don’t Generate Your Data” from his column “Thinking About Data Analysis” (Quality Digest, March 2009), Wheeler stresses, that the primary question of data analysis is, and always has been, “Are these data reasonably homogeneous, or do they contain evidence of a lack of homogeneity?”

The advice by Wheeler, a respected quality professional, is of course, statistically sound advice. This article expands on this central question by offering 10 additional key questions that can help to provide an even more complete story when analyzing data. The table in figure 1 summaries these 10 questions with an interpretation for each.

Below are more details for each of the 10 questions and reasons supporting these questions.

Question 1: What is the message?
With the preponderance of data generated by company databases and the internet, it is critical to get past the presentation and to the facts as quickly as possible. Consider that sources tend to personalize information to make it more interesting, but not everyone relates to the same things. Your perception of information can depend on whether it is presented as positive (half-full) or negative (half-empty). Flipping the statements looking for alternative ways to state them might change your perception. For example, if you see a small number of customers being affected by a product defect, remember this means a large number of customers are not affected, and vice versa. When the facts seem confusing, keep in mind that you might have been given false or incomplete information or you may have misunderstood the information given.

Question 2: Is the source reliable?
As quality professionals, remember that information comes from many sources, good and bad. Think about the information’s quality. All sources within businesses (product development, marketing, sales, operations, support, finance, information technology, and legal departments) have a motivation for providing information. Try to identify the source and its funding so that you can consider any possible biases. The fact that a source or its source of money may benefit from the information does not necessarily mean that the information is false. Product and service information can be based on untested claims, anecdotes, case reports, surveys, and statistical studies. Statistical studies, which take samples and apply the results to the whole population, often provide the best clues. Nonetheless, many studies are needed to be confident about an answer or decision. In figure 2, the table presents some factors that might help you judge the information:

Question 3: How strong is the evidence overall?
It is vital to understand how this information fits in with other evidence. Some sources generally strive to provide unbiased coverage, while others may be intentionally biased. Consider how many sides of the story you hear and whether your source tells you about all of the possibilities, and the weight of the evidence. Remember that extensive coverage of a story can be misleading if it does not reflect the amount of evidence that supports the claim. In particular, the results of early studies can turn out to be right or wrong after time. It is not uncommon for managers to mistakenly reject results that later proved true, and accept results that later proved false.

Question 4: Does the information matter?
Reports generated by independent consultants, your competition, or even your own organization should be carefully evaluated before you respond. For example, just because information appears in the media (e.g., JD Power Customer Satisfaction Ratings, advertisements by competitors) does not mean that it directly affects your organization. The result is that you might be led to worry about small risks that appear big and to ignore big risks that appear to be small. The bottom line is to determine whether the information changes your thinking and leads you to respond.

Question 5: What do the numbers mean?
This is probably the question on which most quality professionals tend to focus—and rightly so. To embellish the point, understanding the importance of a risk requires that you understand the numbers. To avoid confusion when presenting data, put the numbers in a format that you and your audience can understand. (Remember that you can also write 1 in 100 as 1%; 10 in 10,000; ten thousand out of a million; 0.01; or 1 ´ 10-2.)

Quality professionals report their findings as expected values within a range. The breadth of the range shows how confident they are about the results. When only one number is reported, it is probably pulled out of the range and it does not inform you about the researcher’s confidence in the result. In such cases, it is vital to understand whether the number reflects the worst case, best case, or base case.

Question 6: How does this risk compare to others?
For quality practitioners one important skill for comparing risks is making sure that comparisons all involve the chances of the same outcome. In other words, put the risk into context. For example, the following numbers that compare U.S. deaths per year per 10 million peopler:

• 200,000   from heart disease (people over 64)
• 6,000       from lung cancer
• 3,000       from accidents
• 1,000       from homicides
• 400          from accidental poisoning
• 20            from train accidents
• 2              from lightening

Since numbers about risk can be presented in many forms (like the chances of dying from a cause over a lifetime, during a year, or during an event), make sure you compare similar forms. Consider that reporting different parts of a range for different risks (the best case for one vs. worst case for another) can be very misleading. Finally, in making comparisons, other factors may be important to you. For example, consider the extent to which you:

• Think the risk is new.
• Choose the risk.
• Can control, manage, or prevent harm?
• Gain things you want by accepting the risk.
• Fear the risk.
• Feel anxious from lack of knowledge

These factors might mislead you sometimes. For example, an unfamiliar chemical like dihydrogen monoxide might sound threatening, even though it is simply another name for water. You must decide what an acceptable risk is and make decisions based on your professional judgment.

Question 7: What actions can be taken to reduce risk?
As quality professionals, it is important to identify the ways you can mitigate the risk to improve your situation. Be creative. For the risks that are new to you, take the time to think about them before forming an opinion. Keep in mind that just because someone you know picks one action does not mean that the same action will be right for you.

Question 8: What are the trade-offs?
Every decision involves trade-offs. Make sure you can live with the trade-offs associated with different actions. Taking action can also lead to trade-offs of other important resources, particularly time and money. For example, beefing up audit inspections can lead to improvements in services or products, but may affect operations (i.e., diverting resources from day-to-day production activities). Ignoring potential trade-offs when considering an action to reduce or eliminate a risk might ultimately put you or your organization at greater risk.

Question 9: What else do I need to know?
Remember that statistical information is always uncertain even if it is not reported that way. Think about what information is missing and how you would use more information if you had it. In other words, focus on identifying the information that would help you make a better decision. Keep in mind that if you rely on the tantalizing bullet points in an executive management PowerPoint deck as the basis for managing your business unit, you are likely to overlook the well-established (and consequently, not usually mentioned) strategies for improving your business.

Question 10: Where can I get more information?
For most industries and businesses, there is a wealth of information to tap. Find the information that you need to make a better decision. Below are a few sources:

• Manufacturers
• Suppliers and vendors
• Customers (internal and external)
• Front-line workers
• Competitors
• Internet
• Library
• Research organizations (i.e., Gardner Research)
• Government agencies (i.e., U.S. Department of Transportation, U.S. Department of Agriculture)
• Consumer rating groups (i.e., Consumer Reports, J.D. Power and Associates)
• Your original source

In summary, following all or even some of these 10 questions early in the process can help you make a better, more informed decision as a quality professional.

For more information on team training, check out Quality Digest’s Knowledge Guide, “Eight Steps to Team Problem Solving.”

About The Author

Peter J. Sherman

Peter J. Sherman is a managing partner of Riverwood Associates, a lean Six Sigma certification training and consulting firm based in Atlanta. Sherman brings more than 20 years experience in designing and implementing process improvement programs. Sherman is a certified lean Six Sigma Master Black Belt, an ASQ-certified quality engineer, and an APICS-certified supply chain professional. He holds a master’s degree in engineering from MIT and an MBA from Georgia State University.