Featured Product
This Week in Quality Digest Live
Six Sigma Features
Douglas C. Fair
Part 3 of our series on SPC in a digital era
Scott A. Hindle
Part 2 of our series on SPC in a digital era
Donald J. Wheeler
Part 2: By trying to do better, we can make things worse
Douglas C. Fair
Introducing our series on SPC in a digital era
Donald J. Wheeler
Part 1: Process-hyphen-control illustrated

More Features

Six Sigma News
How to use Minitab statistical functions to improve business processes
Sept. 28–29, 2022, at the MassMutual Center in Springfield, MA
Elsmar Cove is a leading forum for quality and standards compliance
Is the future of quality management actually business management?
Too often process enhancements occur in silos where there is little positive impact on the big picture
Collect measurements, visual defect information, simple Go/No-Go situations from any online device
Good quality is adding an average of 11 percent to organizations’ revenue growth
Floor symbols and decals create a SMART floor environment, adding visual organization to any environment
A guide for practitioners and managers

More News

Peter J. Sherman

Six Sigma

Data Analysis—10 Key Questions and Reasons

Published: Friday, July 10, 2009 - 11:00

It is widely known among quality and process improvement practitioners that the lack of a clearly defined scope or charter is perhaps the leading cause for projects not getting started or completed on time and within budget. What are other causes? From my experience, the No. 2 cause for restarting process improvement projects is poor data. Without verifying the integrity of the data, project results can be meaningless.

In Donald J. Wheeler’s “Probability Models Don’t Generate Your Data” from his column “Thinking About Data Analysis” (Quality Digest, March 2009), Wheeler stresses, that the primary question of data analysis is, and always has been, “Are these data reasonably homogeneous, or do they contain evidence of a lack of homogeneity?”

The advice by Wheeler, a respected quality professional, is of course, statistically sound advice. This article expands on this central question by offering 10 additional key questions that can help to provide an even more complete story when analyzing data. The table in figure 1 summaries these 10 questions with an interpretation for each.

Figure 1: Key Questions and Interpretations of Data Analysis





What is the message?

Get past the presentation to the facts


Is the source reliable?

Think about the information’s quality.


How strong is the evidence overall?

Understand how this information fits with other evidence.


Does the information matter?

Determine whether the information changes your thinking and leads you to respond.


What do the numbers mean?

Remember that understanding the importance of risk requires that you understand the numbers.


How does the risk compare to others?

Put the risk into context.


What actions can be taken to reduce risk?

Identify the ways you can mitigate the risk to improve your situation.


What are the trade-offs?

Make sure you can live with the trade-offs associated with different actions.


What else do I need to know?

Focus on identifying the information that would help you make a better decision.


Where can I get more information?

Find the information you need to make a better decision.

Below are more details for each of the 10 questions and reasons supporting these questions.

Question 1: What is the message?
With the preponderance of data generated by company databases and the internet, it is critical to get past the presentation and to the facts as quickly as possible. Consider that sources tend to personalize information to make it more interesting, but not everyone relates to the same things. Your perception of information can depend on whether it is presented as positive (half-full) or negative (half-empty). Flipping the statements looking for alternative ways to state them might change your perception. For example, if you see a small number of customers being affected by a product defect, remember this means a large number of customers are not affected, and vice versa. When the facts seem confusing, keep in mind that you might have been given false or incomplete information or you may have misunderstood the information given.

Question 2: Is the source reliable?
As quality professionals, remember that information comes from many sources, good and bad. Think about the information’s quality. All sources within businesses (product development, marketing, sales, operations, support, finance, information technology, and legal departments) have a motivation for providing information. Try to identify the source and its funding so that you can consider any possible biases. The fact that a source or its source of money may benefit from the information does not necessarily mean that the information is false. Product and service information can be based on untested claims, anecdotes, case reports, surveys, and statistical studies. Statistical studies, which take samples and apply the results to the whole population, often provide the best clues. Nonetheless, many studies are needed to be confident about an answer or decision. In figure 2, the table presents some factors that might help you judge the information:

Figure 2: Factors of Information Source Reliability

Less Reliable (less certain)

More Reliable (more certain)

One or a few observations

Many observations

Anecdote or case report

Scientific or statistical study


Published and peer-reviewed

Not repeated

Reproduced results

Results not related to hypothesis

Results about tested hypothesis

No limitations mentioned

Limitations discussed

Not compared to previous results

Relationship to previous studies discussed

Question 3: How strong is the evidence overall?
It is vital to understand how this information fits in with other evidence. Some sources generally strive to provide unbiased coverage, while others may be intentionally biased. Consider how many sides of the story you hear and whether your source tells you about all of the possibilities, and the weight of the evidence. Remember that extensive coverage of a story can be misleading if it does not reflect the amount of evidence that supports the claim. In particular, the results of early studies can turn out to be right or wrong after time. It is not uncommon for managers to mistakenly reject results that later proved true, and accept results that later proved false.

Question 4: Does the information matter?
Reports generated by independent consultants, your competition, or even your own organization should be carefully evaluated before you respond. For example, just because information appears in the media (e.g., JD Power Customer Satisfaction Ratings, advertisements by competitors) does not mean that it directly affects your organization. The result is that you might be led to worry about small risks that appear big and to ignore big risks that appear to be small. The bottom line is to determine whether the information changes your thinking and leads you to respond.

Question 5: What do the numbers mean?
This is probably the question on which most quality professionals tend to focus—and rightly so. To embellish the point, understanding the importance of a risk requires that you understand the numbers. To avoid confusion when presenting data, put the numbers in a format that you and your audience can understand. (Remember that you can also write 1 in 100 as 1%; 10 in 10,000; ten thousand out of a million; 0.01; or 1 ´ 10-2.)

Quality professionals report their findings as expected values within a range. The breadth of the range shows how confident they are about the results. When only one number is reported, it is probably pulled out of the range and it does not inform you about the researcher’s confidence in the result. In such cases, it is vital to understand whether the number reflects the worst case, best case, or base case.

Question 6: How does this risk compare to others?
For quality practitioners one important skill for comparing risks is making sure that comparisons all involve the chances of the same outcome. In other words, put the risk into context. For example, the following numbers that compare U.S. deaths per year per 10 million peopler:

  • 200,000   from heart disease (people over 64)
  • 6,000       from lung cancer
  • 3,000       from accidents
  • 1,000       from homicides
  • 400          from accidental poisoning
  • 20            from train accidents
  • 2              from lightening

Since numbers about risk can be presented in many forms (like the chances of dying from a cause over a lifetime, during a year, or during an event), make sure you compare similar forms. Consider that reporting different parts of a range for different risks (the best case for one vs. worst case for another) can be very misleading. Finally, in making comparisons, other factors may be important to you. For example, consider the extent to which you:

  • Think the risk is new.
  • Choose the risk.
  • Can control, manage, or prevent harm?
  • Gain things you want by accepting the risk.
  • Fear the risk.
  • Feel anxious from lack of knowledge

These factors might mislead you sometimes. For example, an unfamiliar chemical like dihydrogen monoxide might sound threatening, even though it is simply another name for water. You must decide what an acceptable risk is and make decisions based on your professional judgment.

Question 7: What actions can be taken to reduce risk?
As quality professionals, it is important to identify the ways you can mitigate the risk to improve your situation. Be creative. For the risks that are new to you, take the time to think about them before forming an opinion. Keep in mind that just because someone you know picks one action does not mean that the same action will be right for you.

Question 8: What are the trade-offs?
Every decision involves trade-offs. Make sure you can live with the trade-offs associated with different actions. Taking action can also lead to trade-offs of other important resources, particularly time and money. For example, beefing up audit inspections can lead to improvements in services or products, but may affect operations (i.e., diverting resources from day-to-day production activities). Ignoring potential trade-offs when considering an action to reduce or eliminate a risk might ultimately put you or your organization at greater risk.

Question 9: What else do I need to know?
Remember that statistical information is always uncertain even if it is not reported that way. Think about what information is missing and how you would use more information if you had it. In other words, focus on identifying the information that would help you make a better decision. Keep in mind that if you rely on the tantalizing bullet points in an executive management PowerPoint deck as the basis for managing your business unit, you are likely to overlook the well-established (and consequently, not usually mentioned) strategies for improving your business.

Question 10: Where can I get more information?
For most industries and businesses, there is a wealth of information to tap. Find the information that you need to make a better decision. Below are a few sources:

  • Manufacturers
  • Suppliers and vendors
  • Customers (internal and external)
  • Front-line workers
  • Competitors
  • Internet
  • Library
  • Research organizations (i.e., Gardner Research)
  • Government agencies (i.e., U.S. Department of Transportation, U.S. Department of Agriculture)
  • Consumer rating groups (i.e., Consumer Reports, J.D. Power and Associates)
  • Your original source

In summary, following all or even some of these 10 questions early in the process can help you make a better, more informed decision as a quality professional.

For more information on team training, check out Quality Digest’s Knowledge Guide, “Eight Steps to Team Problem Solving.”


About The Author

Peter J. Sherman’s picture

Peter J. Sherman

Peter J. Sherman is a managing partner of Riverwood Associates, a lean Six Sigma certification training and consulting firm based in Atlanta. Sherman brings more than 20 years experience in designing and implementing process improvement programs. Sherman is a certified lean Six Sigma Master Black Belt, an ASQ-certified quality engineer, and an APICS-certified supply chain professional. He holds a master’s degree in engineering from MIT and an MBA from Georgia State University.