DOE Is Primarily a Research Tool, Not a Production Tool

It’s overhyped and virtually of no benefit in production. The essential production tool is the control chart.

Published: Monday, October 28, 2019 - 11:03

You’ve set aside Sunday afternoon to bake some cookies, but you discover you have run out of eggs. Your partner in marital bliss has gone out and taken the car. You call a couple of mates, and they tell you to try bananas, vegetable oil, or applesauce as egg substitutes. You decide to have some fun and mix it up on a few batches to see what’s best. Woohoo, you’re on the road to design of experiments (DOE).

Transpose the above to a business situation, and you have a food technologist in a research laboratory. He has plenty of eggs, but he wants to cut the cost of using them. He doesn’t have any mates to call, but he has thousands of options such as mangoes, guava, and mashed potatoes instead of the banana; olive oil, peanut oil, sesame oil, and many others. He needs some way of measuring things. DOE gives a structured way to do experiments with a great number of known variables, by adjusting the many known factors in groups, with the minimum number of trials. It gives some clever sums to tell him which is best, with what interactions.

DOE: A research tool

There’s plenty of DOE examples. Everyone has heard of stainless steel. Anyone who has bought it has heard of 18/8, 304, and 316. These give exact proportions of iron, nickel and chromium, and molybdenum, which a research metallurgist 100 years ago found were best for certain properties. He probably used DOE.

There’s nothing new or magical about DOE. It dates back at least 270 years, when it was used for research on scurvy. Almost every one of the three million new engineers and scientists each year learn DOE as part of their university courses. Few ever use it, with the main exception of those with Ph.D.s heading for research.

The words “design of experiments” offer an important clue. DOE is a way to carry out experiments. It’s a research tool. However, in recent years hawkers of certificates have claimed DOE as a production nirvana. Why haven’t millions of engineers already been using their university education and applying DOE everywhere? How can a certificate supposedly provide what a university education cannot? What is the reality?

Like most engineers, I learned DOE at a university. I have worked in research, development, and in production. In research, I worked on two major projects at different times, with a co-worker, in a research laboratory. Both processes were reworks of discarded old methods.

Research labs

The first project was a new process for extraction of CO₂ from flue gas. It used a solid rather than the more common liquid absorber. Now, you can DOE flow rates and temperatures till you are blue in the face, but the problem is the low contact area of a solid adsorbent. A bit of thinking was required. My innovation was a new type of substrate and a new way of loading it, to dramatically increase surface area. Once the idea happened, the magic happened. Setting up testing was then trivial.

The second project was a new process for clarification of raw-sugar washings. Again, you could DOE compositions, temperatures, and flow rates till you were blue in the face, but the clarification floc clogged the filters. The floc worked great, but you couldn’t get rid of the stuff. In this case, the key was examining the floc under a microscope. Again, another bit of thinking. It revealed that some microscopic crystals were also being filtered along with the floc. I reasoned that if the crystals could be grown, they might act as a lattice, to act as a filter aid. Again, the idea allowed the magic to happen. Again, testing was then routine.

I have closely examined other research projects such as the progress of construction of an ore sorter, where individual rocks are heated, and the radiation spectrum analyzed to determine whether to keep or eject. Again, DOE played an insignificant role. The key was thinking and clever ideas.

However, there are situations, such as creating new cookies or new forms of steel, where the process is simply about the best combination of temperatures and compositions of components. In such cases, research is done on as small a scale as is practical in a research facility, and it is just a matter of DOE.

Production

Once the research is finished, products roll into production. The rule is look but don’t touch. I cannot imagine any factory manager in his right mind allowing some fellow clutching a certificate, to start experimenting on his process. The aim is to keep product rolling out the door. If it stops, money is lost, and the factory manager has to answer for it. The process should already be understood from the research phase.

While experimental tools are for research, the key tool in production is the observational tool, the control chart. As Donald Wheeler points out “Observational studies are like studying lions in the wild. Experimental studies are like studying lions in a zoo.”

In the research laboratory, the major variables are known and may be changed in experiments. However, this is not the case in production. There are myriad unknown factors that may affect a product. The way to find them is by observation. This is the role of the control chart.

Production experiments

In one production job, I worked as shift foreman in a large factory making plaster, wallboard, and cornice. On one occasion, the cornice factory started making reject. I was told to stay away. For six weeks, three shifts a day, the plant swarmed with head office engineers, while a warehouse filled with reject. There was no chance of rework. The stuff barely even looked like cornice.

The cornice setting belt had a dozen shaped rollers down the line that forced the wet cornice to the correct shape in the setting belt. The engineers spent weeks changing the positions of the rollers and changing roller pressures. This might have been great DOE potential! Dozens of variables and potentially hundreds of experiments! However, they could have DOE’d till they were blue in the face. It would have had zero benefit.

Suddenly, the problem vanished of its own accord.

Two months later, at around 2 a.m., I was on shift, and suddenly the cornice problem raised its ugly face again. It was just me and a bunch of Greeks and Turks, with whom I had become very close. I respected them, and they all respected and looked up to me. It was essential to have their help, trust, and support. My first step was to find the cause. I checked back through the records and noticed that a new batch of PLB board had a much higher porosity. I reasoned it might be stiffer. I could imagine how this might create the strange, flattened, defective shape.

The next step was to carry out simple experiments. I was on my own, so I could do what I pleased. The boys had my back. The scoring wheels didn’t have adjustments, so I had the operator find a heavy bolt and we did our own. Sparks flew. My aim was to research the effect of every possible thing that we could change. No math. No statistics. No clever experimental structure. We’d make a change and I’d yell or run down to the end of the line to see what effect it had. One change at a time. We kept it simple. Just common sense, thinking, and dedicated operators.

In four hours, I showed that production was an exercise in origami. By adjusting the relative depths and separation of the scoring wheels, I could control every parameter.

I also showed that the forming rollers, which the engineers spent six weeks adjusting, did absolutely nothing. Too much pressure on a roller, and the wet plaster was forced out, causing a plant shutdown. I removed every roller.

DOE was not a solution for anything. The key was again clear thinking, not tools.

The aftermath

At 9 a.m. when staff arrived, they were shocked to see the changes I had made. Much screaming ensued. All the rollers were promptly put back into place, despite my protestations. I explained that the forming rollers were more superstition than science and contributed greatly to plant stoppages. Superstition ruled.

My changes actually produced a slightly different shape on the hidden side, that made the cornice much stronger when stacked in storage. This was also ignored.

I had also developed new operating procedures. There was no reward, no praise, only admonishment. You don’t carry out experiments on the process, I was told. You don’t touch the process. If something goes wrong, you call staff. Things went back to normal. Pompous staff with noses in the air, trying to avoid eye contact with operators, continued to do their walks (long before the term gemba existed). Staff never paused to ask about my handwritten description of new control procedures, sticky taped to the former machine. Nor did they question the bolt hanging on the string next to my instructions. The operators understood. They had watched everything. It was passed on from one operator to the next in languages I didn’t understand.

In four hours we had performed simple experiments that should have been carried out in a research facility, before the factory was built. It should not have been needed in production. If research had been done properly, the former station would have been designed very differently with adjustable scoring wheels.

Process improvement

You have optimized your process, perhaps with DOE in the research labs or pilot plant, and you roll it out to production. Your workers are well-trained in quality improvement, and they work with management to reduce special causes. Operating processes, predictably, are by far the most important steps.

The next step is to reduce common-cause variation, or variation inherent in the system. Again, this is the role of research. This might also apply where cheaper raw materials are to be investigated. If improvements are found, they are fed into production carefully, one at a time. There must be minimum risk of disrupting production and minimum risk of waste production.

In some circumstances even a pilot plant does not mimic the process sufficiently well. An example is a glass furnace in insulation manufacture. A full DOE trial, however, is usually far too risky. It could easily disrupt production. Single variable changes may be made carefully, to observe their effect on the process. One-factor-at-a-time experimentation may be less efficient than simultaneous multivariable changes, but it is far less disruptive and far less likely to produce waste. The one-factor-at-a-time method does not detect potential variable interactions, but these are usually of secondary importance. Experimentation is conservative if it’s essential in production.

In some situations, where research is needed on a large scale rather than in the laboratory, experimentation in the factory may be possible if production is not continuous. In sugar mills, for example, the factory is shut down during an off season, allowing experimental trials to be carried out with minimal risk.

A major factor is whether production is batch or continuous. The loss of a single item or batch may be of far less consequence than disruption to continuous production. Regardless, DOE is not a magic button and is no substitute for insight and thinking.

In all these situations, experimentation is best carried out by experienced engineers or researchers, not certificate holders. It comes down to common sense. We would never experiment with the production version of our software, and we’d certainly never let a neophyte mess with it. We continually do research offline and feed in any improvements gradually, to production. We always aim for happy customers, without risk.

EVOP

Statistician George Box suggested not only making simultaneous changes to multiple variables via DOE on a production process, but also to do so continuously, in what he called EVOP, evolutionary operation. In situations where a process is being operated predictably, that is, in control; where cost of reject is low; where plant shutdowns are brief and painless; where startups are quick, easy, and produce minimal waste; where operations are batch-wise; where there are large returns from small improvements; where operators are enlightened; and where management is unafraid of complexity and is prepared to boldly go forth without fear of disruption or split infinitives—this may be appropriate.

Note that Box’s EVOP is different than the EVolutionary OPtimizer hill-climbing control system.

Summary

DOE is plain toast with the crust cut off, for Ph.D.s in research labs. Almost every engineer and scientist has learned DOE, but most have never used it. For nonengineers or nonscientists, DOE certification is a total waste of time and money. DOE is greatly overhyped and of virtually no benefit in production. The essential production tool is the observational tool, the control chart, not DOE. The real key to process improvement is not tools, but being able to think logically, often under extreme pressure, and having the support of the operators.

Thanks to Donald Wheeler, N. El-Thaher, C. Norman, R. James, and Scott Hindle for their contributions.

About The Author

Anthony D. Burns

Anthony Burns, Ph.D., has a bachelor of engineering and a doctorate in chemical engineering from the University of New South Wales in Sydney, Australia. He has 36 years of experience and his company, MicroMultimedia Pty. Ltd., is responsible for the development of the e-learning quality product Q-Skills and its support tools.