by Davis Balestracci
The current design of experiments (DOE) renaissance seems to favor factorial designs and/or orthogonal arrays as a panacea. In my 25 years as a statistician, my clients have always found much more value in obtaining a process "road map" by generating the inherent response surface in a situation. It's hardly an advanced technique, but it leads to much more effective optimization and process control.
The following are typical curricula for DOE in Six Sigma programs. The main topics for many basic DOE courses include:
• Full and fractional factorial designs
• Screening designs
• Residual analysis and normal probability plots
• Hypothesis testing and analysis of variance
Main topics for many advanced DOE courses include:
• Taguchi signaltonoise ratio
• Taguchi approach to experimental design
• Responsesurface designs (e.g., central composite and BoxBehnken)
• Hill climbing
• Mixture designs
No doubt these are all very interesting, but people don't need statistics; they need to solve their problems. A working knowledge of a good, basic experimental design software package containing easytouse, appropriate regression diagnostics is far more valuable and negates the need to teach several of the above topics.
This article will use a relatively common scenario to demonstrate sound experimental design philosophy and its flexibility as well as a "takeaway" I consider a breadandbutter design tool for any quality practitioner. It will also show you, to paraphrase Yogi Berra, that "90 percent of statistics is half planning."
Let's consider a production process where the desired product immediately decomposes into a pesky, tarry byproduct. The process is currently averaging approximately 15 percent tar, and each achievable percent reduction in this tar is equivalent to more than $1 million in annual savings.
Process history has shown three variables to be crucial in controlling the process. The variables and chosen regions for experimentation are temperature, 5565° C; copper sulfate (CuSO_{4}), 2631 percent; and excess nitrite (NO_{2}), 012 percent. Any combination of the three variables within these ranges would represent a safe and economical operating condition. The current operating condition is in the midpoint of these ranges.
For the purposes of experimentation only, the equipment is capable of operating in the following ranges if necessary: temperature, 5070° C; CuSO_{4}, 2035 percent; and excess NO_{2}, 020 percent.
Suppose you had a budget of 25 experiments.
• Where should the process operate (i.e., where should the three variables be set)?
• What percent tar would you expect?
• What's your best estimate of the process variation (i.e., tar ± x percent)?
This is the scenario I use to introduce my experimental design seminars. I form groups of three to four people to simulate experimental conditions... and get as many different answers for the optimum as there are groups in the room, and just as many strategies (and number of experiments run) for reaching their conclusions.
General observations:
• Most try holding one to two of the variables constant while varying the third and then optimizing around their best result.
• Each experiment seems to be run based only on the previous result.
• Some look at me smugly and do a 2^{3} factorial design (and, generally, get the worst answers in the room).
• Some run more than the allotted 25 experiments.
• Some go outside of the established variable ranges.
• Some look at me like I'm nuts when I tell them:
Repeating a condition uses up one of their experiments.
They should repeat their alleged optimum (and it counts as an experiment…).
• I'm accused of horrible things when the repeated condition gets a different answer (sometimes differing by as much as 1114 percent). I simply ask, "If you run a process at the same conditions on two different days, do you get the same results?"
Because of the above observations:
• I'm sometimes told the "process is out of control," so there's no use experimenting.
• Most estimates of process variability are low.
• Groups have no idea how to present results in a way that would "sell" them to management.
• The suggested optimal NO_{2}  settings are all over the range of 012 (and, by the way, that variable is modeled to have no effect).
It's not as simple as declaring, "We want to run an experiment." The presence of natural process variation clouds the issue, so we must first consider this question: Is the objective to optimize or establish effects?
Given any situation, there's an inherent response surfacei.e., the "truth"that one is trying to model and map. Any experimental result represents this true response, clouded, unfortunately, by the commoncause variation in the process.
The true surface (called a "contour plot") of the tar scenario is shown in figure 1. (The NO_{2}  variable has zero effect.) My simulator generates this true number along with a random, normally distributed error that has a standard deviation of four. Note that any combination of temperature and copper sulfate concentration allows one to read the corresponding resulting tar formation from the contours, and that tar is minimized at 65° C and approximately 28.8 percent CuSO_{4}.
Figure 1: True Tar Response Surface
It can be generated either directly in the case of a high state of process knowledge (i.e., "optimize"), or established sequentially for a low state of process knowledge once critical variables and their important ranges are found (i.e., "establish effects").
From calculus, any function can be modeled as an infinite power series. Response surface DOE's objective is to model the current situation by a power series truncated after the quadratic termsa quadratic "French curve." In the current case of our three variables:
Y=B_{0} + B_{1}x1 + B_{2}x2 + B_{3}x3 + B_{12}x_{1}x_{2} + B_{13}x_{1}x_{3} + B_{23}x_{2}x_{3} + B_{11}x_{1}^{2} + B_{22}x_{2}^{2} + B_{33}x_{3}^{2}
So, what design(s) gives one the best estimates of these "B" coefficients? It's like building a tabletop: Where does one place the legs? I often see people, in essence, running large numbers of experiments along the "diagonal" of the table, when four wellplaced experiments at the corners would actually be more efficient.
The first two designs that follow consider the "optimize" objective and represent a damnthetorpedoes approach. You just want to optimize your process and get on with life.
The third design demonstrates the "establish effects" objective and its inherent sequential approach. This is a low "state of the knowledge" strategy where time is needed to identify critical variables and map the interesting design subspace.
If this were a relatively mature process where these three variables had evolved as key process controllers, and if DOE was a new concept to the organization and a quick success was needed, I'd use the design shown in figure 2. It's known as a threevariable BoxBehnken design with 15 runs. It's shown in its coded (i.e., geometric) form on the left and actual variable settings on the right (1 represents the low setting of the variable, and +1 represents the high setting). It's structured to run all possible 2x2 factorial designs for the three possible independent variable pairings (i.e., x1 and x2, x1 and x3, x2 and x3) while holding the other variable at its midlevel.
These 15 experiments should be run randomly. Note that the data point with each variable at its midpoint (i.e., center point) is run three separate times. These replications, called a "lack of fit" test, are key to checking whether the region chosen is too large to be approximated by a quadratic equation.
Lack of fit is only one of four diagnostics of the generated quadratic equation. It's beyond this article's scope to teach the appropriate regression analysis and its diagnostics. However, briefly, the residuals (i.e., actual values minus the predicted values) contain all of the information needed to diagnose the model fit.
So, besides the above lackoffit test:
• A residualsvs.predicted values plot, which should look like a shotgun scatter, might indicate the need for a transformation of the response (e.g., viscosity).
• A normal plot of the residuals tests the implicit assumption of normality for least squares regression.
• A run chart of the residuals in the order in which the experiments were run checks whether an outside influence crept in during the experiment.
For my money, this should be a "breadandbutter" design for all quality practitioners in a high "state of knowledge" situation needing some fast answers. Think about it: You could generate a response surface by the efficient use of only 15 experiments.
Note that, in contrast to the BoxBehnken, it's usually not possible to optimize a process using, in this case, a 2^{3} factorial experiment. A factorial experiment can only get the linear terms and interaction terms of the underlying equation, not the quadratic terms. Optimizing via the eight experimental runs from the factorial design can only lead to one of the cube's cornerswhich in this case would be incorrect. As will be shown, should a more sequential strategy (to establish effects) be chosen, adding center points (i.e., all variables at their midpoints) to a standard factorial design allows one to test whether further experimentation is indeed even needed for optimization, which would build on the current design. (The ability to optimize solely with a factorial design tends to be the exception.)
What's another possibility for our situation? To use the statistician's favorite answer, "It depends." Another design many of you have heard about is the central composite design. In this situation, it would be set up as in the table in figure 3.
It's a factorial design augmented by an additional block of experiments (called "star points"), where each individual variable is taken to two further extremes while the other two are held at their midpoints. The subsequent response surface will be most valid for variables in their [1, 1] coded ranges.
You'll also see that it takes 20 experimental runs in this case and creates the potential problem of dealing with five levels for each variable. It has happened often enough in my experience to mention that, occasionally, this additional starpoint block of experiments causes problems with the quadratic approximation because the region is extended beyond its reasonable limits, as indicated by the lackoffit test.
It depends. How well do you know your process and, once again, is your objective to establish effects or optimize?
For example, maybe the objective of initial experimentation was to see whether all three variables were necessary before proceeding with generating the response surface. The naive approach would be to run an unreplicated 2^{3} factorial design.
Now, you'll notice that the central composite design does just that; however, one has committed up front to running the entire design and must therefore plot the resulting equation regardless of effects.
Excuse me while I digress to two crucial topics:
It's not as simple as running a 2^{3} factorial (because you have three variables) and "picking the winner." In this case, running eight experiments could detect only a variable's effect if it made a difference of approximately 11 percent tar when set at its upper range, as opposed to being set at its lower range (dependent on the process standard deviation of ±4). Repeating the entire experiment16 runs plus center pointsnow allows you to detect a difference of 8 percent.
If you're going to run 16 experiments (plus a few center points), did you know that you could add up to five more variables (i.e., eight variables total) to see whether their effects are significant? (For more on screening designs, see R. M. Moen, T. W. Nolan and L. P. Provost, Quality Improvement Through Planned Experimentation, Second Edition, McGrawHill, 1999.) This strategy also allows one to make important decisions regarding noncontinuous variables (i.e., catalyst A vs. catalyst B, with no center point possible) before continuing to the map.
Then there are "interactions," which are the rule rather than the exception. From the tar response surface, note the difference in the effect of temperature at the low level of CuSO_{4}, where tar can vary from 12 percent at 65° C to more than 50 percent at 55° C, vs. the high level of CuSO_{4}, where the percent tar is relatively constant around 10 percent, regardless of the temperature. Because of the concept of interactions, ignoring other variables puts you at the mercy of their current settings. In other words, if you decide to experiment in the future on those variables currently held constant, it could possibly (and would most probably) invalidate the results of your current experiment.
Rules of thumb:
• Sixteen factorial experiments are an efficient number to run.
• Be inclusive with the independent variables, especially when they amount to eight or fewer.
We were asking the question, "Are three variables necessary?"
Scenario 1 : Suppose one did repeat a 2^{3} factorial design (with ranges of 5565° C for temperature, 2630 percent for CuSO_{4} and 012 percent for NO_{2} ) and discovered that the NO_{2}  variable was indeed not significant. How would one proceed?
You'd run the starpoint block of experiments as seen in figure 4 (in random order), probably setting NO_{2}  at zero. Because of the geometry of the situation (i.e., repeated factorial cube), the starpoint settings are now ±2 instead of ±1.633. (Note that you'll have now run 27 experiments total.)
These data would build on your previous data, all of which would be used in the analysis.
Scenario 2: Suppose that NO_{2}  was significant? Because the initial objective was to see whether it was indeed necessary, the lower level of NO_{2}  has already been set to zero, unlike some experiments that anticipate zero being the low starpoint setting. In this case, it would now be impossible to have the coded "1.633" setting for the lower star point because negative NO_{2}  is impossible. Experience has shown that one should use its lower limit (i.e., 0) as the lowest starpoint setting and repeat that condition for the "tabletop" stability needed in the equation generation and/or analysis. This replicate would be independently randomized into the design, as shown in figure 5.
What seemed like a relatively straightforward scenario turned out to have a lot of potential approaches for its solution. However, good experimentation relies on the process of asking the appropriate questions to define clear objectives beyond the typical, "We need to run an experiment."
Davis Balestracci is a member of the American Society for Quality and past chair of its statistics division. Visit his Web site at www.dbharmony.com.
