How Acceptance Sampling Works

What can you say about the sampled lot?

Published: Monday, April 4, 2022 - 12:03

Acceptance sampling uses the observed properties of a sample drawn from a lot or batch to make a decision about whether to accept or reject that lot or batch. Although the textbooks are full of complex descriptions of various acceptance sampling plans, there are some very important aspects of acceptance sampling that are not included in the textbooks.

Extrapolation

In the interest of simplicity, the product that has been measured will be referred to as the “sample,” while the product that hasn’t been measured will be referred to as the “lot.” Every time we attempt to use a sample to characterize the quality of a lot, we will be making an extrapolation from the product that has been measured to the product that hasn’t been measured.

So how can we extrapolate from the sample to the lot? Extrapolation inevitably requires an assumption that the sample is representative of the lot. When this assumption is appropriate, the extrapolation will make sense. When this assumption is wrong, the extrapolation will also be wrong. No matter what computations we may use, the validity of our statements about the lot will always come back to the question of representation. The ability of any sample to represent a lot will be affected by the homogeneity of the lot itself and the manner in which the sample is selected from the lot.

Let’s begin with the case of random samples drawn from uniform lots. Given a lot having uniform quality, and given a random sample obtained from this lot, we can use probability theory to develop a mathematical treatment of the uncertainty of the extrapolation from the sample to the lot.

Estimates of lot quality

Needless to say, the twin assumptions of uniform lot quality and a random sample remove a host of problems. A random sample may be broadly defined as one that is obtained in such a way that every item in the lot has the same chance of being included in the sample. When the assumptions of uniform lot quality and a random sample are satisfied, the following results will hold.

Whether we are measuring a property, testing for functionality, or counting blemished items, a sample of n items will contain some fixed number of nonconforming items, Υ, and the sample fraction nonconforming will be:

This value is known as the “binomial point estimate” for the lot fraction nonconforming. For hundreds of years it has been known to be the best single-number estimate of the lot fraction nonconforming. However, we need more than a single-number estimate. We also need an interval estimate to incorporate the uncertainty of our extrapolation from the sample to the lot.

The best modern practice here is to use a 95-percent Agresti-Coull interval estimate. This interval estimate will be centered on the ratio.

And our 95-percent Agresti-Coull interval estimate may be found using the formula:

This Agresti-Coull estimate is good all the way down to Υ = 0 and all the way up to Υ = n. A table of 95-percent Agresti-Coull interval estimates for various values of Υ and n is given in Figure 1.

Figure 1: 95-percent Agresti-Coull interval estimates for percent nonconforming in lot

Say, for example, that a random sample of n = 175 parts has been selected from a lot containing 6,250 parts, and when measured we find that Υ = 3 of these parts are nonconforming. Then, our best point estimate for the lot fraction nonconforming would be (3/175) = 1.7 percent, while from Figure 1 we find our 95-percent Agresti-Coull interval estimate for the lot fraction nonconforming to be 0.4 percent to 5.2 percent. This means that we can be reasonably certain that the lot fraction nonconforming is no more than 5.2 percent, and also that it’s probably greater than 0.4 percent. With these data we can’t really narrow it down more than this.

Figure 2: What we know about the lot when Υ = 3 and n = 175

If knowing that there is more than 0.4-percent nonconforming in this lot is bad enough to justify rejecting the lot, then reject this lot.

On the other hand, if knowing that there is less than 5.2-percent nonconforming in this lot is good enough to justify accepting the lot, then accept this lot.

But if 0.4-percent nonconforming isn’t bad enough to justify rejecting the lot, or if 5.2-percent nonconforming isn’t good enough to justify accepting the lot, then you don’t have sufficient evidence to take either action.

Figure 3 illustrates this truth about acceptance sampling. Think about this very carefully.

Figure 3: The truth about acceptance sampling

In addition to forcing a decision based on inadequate information, acceptance sampling also implicitly assumes that there is some nonzero level of nonconforming product that is acceptable. Thus, acceptance sampling always aims at less than perfection.

Rather than using an interval estimate to characterize what the sample reveals about the lot being sampled, acceptance sampling plans are commonly characterized by quantities such as the acceptable quality level (AQL), the lot tolerance percent defective (LTPD), or the average outgoing quality limit (AOQL). All of these numbers describe long-term properties of the product stream, and as such they apply to the warehouse as a whole.

The quality in the warehouse

A company making hardwood flooring used the following procedure as its quality assurance plan. After fabricating 6 x 6 parquet tiles, the tiles were glued together into 12 x 12 squares and boxed 25 squares to a box.

Figure 4: Pallets of parquet flooring

As each pallet of 250 boxes was ready to move to the ware­house, the quality auditor would select seven boxes (175 squares) for inspection based on a Dodge-Romig 5-percent average outgoing quality limit (AOQL) plan. If 15 or more defective squares were found in these seven boxes, the pallet would be rejected and subjected to 100-percent inspection. The pallet would be torn down, the remaining 243 boxes would be opened, all of the squares would be inspected, defective squares would be replaced with good squares, and then all would be reboxed and placed on the pallet.

If the number of defective squares was 14 or fewer, the pallet would be accepted. Any defectives found in the sample would be replaced with good squares, the seven boxes would be closed up and placed on the pallet, and the pal­let would be moved to the warehouse.

Thus, the warehouse was filled with pallets of parquet floor tiles. Some pallets would contain material that had been subjected to 100-percent screening inspection. Other pallets would contain material substantially the same as it came from the production line. The warehouse was a checkerboard of pallets having two different levels of defective product.

To illustrate this point, assume that 2.5 percent of the 6 x 6 tiles are defective. When four of these tiles are glued into 12 x 12 squares, we’ll end up with 9.6 percent defective squares because:

Fraction of defective squares = [ 1 – (1 – 0.025)⁴ ] = 0.0963

This means that the seven-box samples will average about 16.8 defective squares per sample. As a result, the AOQL plan described above will reject and screen about 71 percent of the pallets. The other 29 percent of the pallets will be accepted and shipped to the warehouse. In this scenario, the warehouse will end up containing about 97 percent good squares overall, but while 71 percent of the pallets will be virtually perfect, the remaining 29 percent will have an average of 9.5 percent defective squares after the defectives in the seven sampled boxes have been replaced.

Figure 5: Pallets in the warehouse

At this point, the quality manager wanted to know if he could assure the president of the company that there was at least 95-percent good product in the ware­house. Because he was using a 5-percent AOQL plan, the answer to his question was yes. In the scenario above, the warehouse actually had 97-percent good stuff. However, his question was the wrong question, because the customers didn’t buy the warehouse: They bought this product by the box.

A distributor gets a pallet of floor tiles and sells a few boxes at a time to installers. When the distributor gets a screened pallet, the installers find the material to be satisfactory, and everyone builds up certain expectations regarding this product. When the distributor gets an unscreened pallet, the installers find more defective squares than they had expected. As a result, they may not have enough material to finish their job; they have to get more material. In the meantime, the contractor and the homeowner are unhappy about the delay. Thus, the customers are unhappy, the installer is unhappy, and the distributor is unhappy about all the complaints and the returned material he has to handle. Yet the president of the flooring company and his quality manager sleep soundly each night, comforted by the knowledge that the warehouse contains at least 95-percent good stuff.

Hit-or-miss inspection based on acceptance sampling will often make things worse by creating an inconsistent product stream.

What can be said about the pallets?

Consider what the 95-percent Agresti-Coull interval estimates can tell us about the pallets above. All of the pallets that were accepted had 14 or fewer defective squares in their 175-square sample. For those pallets with 14 defectives, the upper 95-percent Agresti-Coull interval estimate is 13.1 percent. So this 5-percent AOQL plan will accept some pallets having up to 13.1 percent nonconforming.

Those pallets that were rejected and screened all had at least 15 defectives in their 175-square sample. For those pallets with 15 defectives, the lower 95-percent Agresti-Coull interval estimate is 5.2 percent. Those pallets that were rejected are likely to have had at least 5.2 percent defective squares before screening.

So this 5-percent AOQL plan will reject some of the pallets having more than 5.2-percent nonconforming while accepting others having up to 13-percent nonconforming.

Figure 6: Actions taken based on the 5-percent AOQL plan for flooring

So, which is it? Do you want to accept those lots with less than 13.1 percent defective, or do you want to reject those lots with more than 5.2 percent defective? This 5-percent AOQL plan lets you alternate between doing both. Here, our actions depend more on the luck of the draw than anything else. Acceptance sampling is just another way to play roulette!

Random samples and convenience samples

The computations behind all acceptance sampling plans assume that you have a random sample from a uniform lot. A random sample is defined as one where each of the items in a lot has the same chance of being included in the sample. This assumption is the basis for the extrapolation from the sample to the lot. However, in practice, the samples are rarely random. Given the 250 boxes of parquet squares as shown in Figure 7, where would you select your sample of seven boxes? In industrial practice, the samples are almost always drawn from the end of the roll, the top of the basket, and the outer layer of boxes on the pallet.

Figure 7: Where a pallet of boxes gets sampled

Given the complexities of breaking down a pallet, opening every box, inspecting the 6,075 remaining squares, and repacking them all, how long do you think it will be before the workers begin to load “specially selected” boxes on the outer corners of the pallets? By putting 16 good boxes on the corners, they can greatly affect the outcome of the acceptance sampling plan. When this happens, the percentage of rejected pallets will drop and the quality in the warehouse will drift back toward 9.5-percent defective throughout, in spite of using a 5-percent AOQL plan.

Figure 8: Pallets in the warehouse after workers place “specially selected boxes” on corners

In addition to the problem of using a convenience sample, there is the fact that the inspector selects boxes rather than individual squares. Although the 95-percent Agresti-Coull interval estimates may characterize the uncertainties associated with drawing a random sample from a uniform lot, they can’t begin to characterize the nonsampling errors associated with using convenience samples or selecting boxes rather than individual squares. Because these nonsampling errors are likely to increase the uncertainties associated with the extrapolation from the sample to the lot, we may be worse off than the interval estimate might lead us to believe. We certainly are unlikely to ever be better off. Thus, with convenience samples the interval estimates become “best-case scenarios.”

Two assumptions of acceptance sampling

Acceptance sampling implicitly makes two contradictory assumptions about your product stream. The first of these is that the product quality is uniform throughout each lot. If a lot is internally uniform, then the extrapolation from the sample to the lot will be reasonable. But how can this assumption be justified in the absence of data? Without a process behavior chart that shows a predictable product stream, any assumption about within-lot uniformity is simply wishful thinking.

The second assumption of acceptance sampling is that the lot quality is highly variable from lot to lot. If this weren’t so, why would we need to accept some lots and reject others? Acceptance sampling plans can’t reliably separate good lots from bad lots unless the lot qualities are substantially different. In the example given here, when each lot is internally uniform, the 5-percent AOQL plan will reliably sort lots with more than 13 percent defective from those with less than 5 percent defective. But, as shown back in Figure 6, it will do a poor job with uniform lots of intermediate quality.

Figure 9: When the 5-percent AOQL plan for flooring would work well

So where will acceptance sampling work? You will have to have some reason to believe that each lot is internally uniform, and yet also believe that the lot quality is substantially different from lot to lot.

But if your process is being operated predictably, then the lots are likely to be internally uniform and also uniform from lot to lot. Any use of acceptance sampling here will arbitrarily reject and accept lots having essentially the same quality level.

On the other hand, when your process is being operated unpredictably, process changes are unlikely to occur only between lots. Each lot may be nonuniform, which will undermine the extrapolation from your convenience sample to the lot as a whole. Any use of acceptance sampling here will fail to separate the good stuff from the bad stuff because each lot is likely to have some of both.

So when you use acceptance sampling, you must assume that the product quality is highly variable from lot to lot, but, at the same time, that it is very uniform within each lot.

If the product quality is changing from lot to lot, isn’t it likely that it’s also changing within each lot? And if this is the case, then how likely is it that our convenience samples will properly characterize each lot?

If the product quality is uniform from lot to lot, then why do we need to accept some batches and reject others?

The only situation where acceptance sampling schemes will work with convenience samples is one where the lots are known to be internally uniform, but where the lot quality is highly variable from lot to lot. This is the case in Figure 5. Thus, the condition assumed by the use of acceptance sampling is often the condition created by the use of acceptance sampling.

Inspection vs. acceptance sampling

Hit-or-miss inspection based on acceptance sampling will not minimize the cost of any operation. The role of inspection is to improve the economics of production. Inspection upstream will reduce subsequent costs by reducing the waste when the defectives are found downstream. Three times in my brief conversation with the quality manager of the flooring company, he commented on how painful it was to throw away the three good 6 x 6 tiles every time they found a square with one defective tile. Consider the impact of the 5-percent AOQL plan they were using. With 2.5 percent defective 6 x 6 tiles, they have 9.6 percent defective squares. Their plan would result in screening 71 percent of the pallets. With replacement of defective squares, this means they will have to produce material for 114.3 pallets to actually get 100 pallets in the warehouse.

Contrast this with what they could do with 100-percent inspection for the 6 x 6 tiles prior to gluing them into squares. Say this inspection is 90 percent effective and cuts the fraction of defective tiles from 2.5 percent to 0.25 percent. This would result in 1 percent of the 12 x 12 squares being defective. Boxing these and sending them straight to the warehouse without any further inspection would result in 99 percent good stuff on each pallet, and 99 percent good stuff throughout the warehouse as well. To get 100 pallets in the warehouse would require the production of 6 x 6 tiles sufficient for 102.3 pallets. Given the complexities of screening 71 percent of the pallets, this 100-percent inspection upstream would be cheaper to implement than the acceptance sampling plan, would result in higher process yields, and would produce a consistent stream of higher-quality product.

Summary

The whole thrust of acceptance sampling is to strike a compromise with imperfection. It begins with the idea that there is some nonzero level of nonconforming product that is acceptable. Next, it uses hit-or-miss inspection to give the appearance of having scraped the burnt toast. And by focusing on the quality in the warehouse, it distracts people from thinking about the quality of the lot in hand.

Thus, by using acceptance sampling, we can give the appearance of having done something about quality without having to actually perform a 100-percent inspection. By accepting or rejecting each lot, regardless of whether we have sufficient evidence to do so, we pretend to know things that we don’t know. When this ignorance catches up with us, we simply blame it on a bad sample and go on as before. Thus, acceptance sampling is nothing more than a Band-Aid on the problem of nonconforming product.

Using interval estimates will allow you to approximate what you actually know about the lot in hand. These interval estimates may only be a best-case scenario, but they at least quantify some of the uncertainty in your extrapolation from the sample to the lot.

Finally, to use an acceptance sampling plan, you have to simultaneously believe two contradictory things about successive lots: Each lot is uniform, but successive lots can be very different in quality. This means that about the only time acceptance sampling might be appropriate is when you have to clean up the mess left when someone upstream used acceptance sampling as their quality assurance plan.

When it’s a matter of rectifying defects, the only economic levels of inspection are all or nothing. Hit-or-miss inspection based on acceptance sampling will not minimize the cost of any operation. In fact, it can actually make things worse.

About The Author

Donald J. Wheeler

Dr. Wheeler is a fellow of both the American Statistical Association and the American Society for Quality who has taught more than 1,000 seminars in 17 countries on six continents. He welcomes your questions; you can contact him at djwheeler@spcpress.com.