Interpreting Data in Context

In memory of Al Phadt, Ph.D.

This article is a reprint of a paper Al and I presented several years ago. It illustrates how the interpretation and visual display of data in their context can facilitate discovery. Al’s integrated approach is a classic example not only for clinical practitioners but also for everyone who needs to turn data into knowledge.

This is an example of how process behavior charts were used to (1) evaluate outcomes; and (2) assist in making clinical decisions in the treatment of severe, potentially life-threatening, self-injurious behavior (viz., self-inflicted wounds to the body caused by head-banging and biting the wrists and fingers). The treatment of Peter, a 25-year-old man with autistic disorder who functions in the severe range of intellectual disability and has been blind since birth, is described from two points of view. First, from the perspective of Dr. Al Pfadt, the behavioral psychologist who constructed and analyzed the charts shown here; then from the perspective of Peter’s parents. The following material was provided by Pfadt and his colleagues at the New York State Institute for Basic Research in Developmental Disabilities and is used with the permission of Peter’s parents.

Rather than being a story of a dramatic cure, this is the story of incremental improvement, based on the careful collection and analysis of data to discover relationships between presumed causes and observable outcomes.

Peter’s self-injurious behavior first emerged during his infancy and persisted throughout his life. He has had periods of improved functioning, and he has suffered disappointing relapses. In a previous, out-of-state, residential placement, a variety of aversive behavioral treatment procedures were used, up to and including electric shocks. While these procedures were intended as a form of punishment whenever he engaged in self-injury, they were all unsuccessful in suppressing Peter’s dangerous behaviors. Because of this long history of failures, Peter was transferred to the Institute for Basic Research on March 4.

Because of his self-injurious behavior, Peter was routinely kept in restraints consisting of devices resembling handcuffs that tethered his wrists to a belt worn around his waist He also wore a helmet to prevent injury to his head. When they first attempted to remove these restraints, Peter became extremely agitated and attempted to injure himself so violently that it was necessary to restore them immediately to protect him from harm. This suggested that the devices functioned as a form of self-restraint that is frequently encountered among individuals who engage in self-destructive behavior. This conceptualization of Peter’s behavior also provided the basis for the behavioral component of his treatment plan. Since the restraints appeared to function as a form of self-control, Pfadt decided not to prohibit their use. Rather, he decided to teach Peter to ask for restraints in a more socially acceptable manner, and then to gradually attempt to develop his tolerance for having restraints removed periodically for short intervals.

The measure used was the number of minutes per shift that Peter spent out of his restraints. This value could range from a low of 0 minutes to a maximum of 480 minutes. This measure was chosen as an outcome measure for three reasons. First, it reflected Peter’s clinical status at the time of admission and provided a focus for his treatment plan, as will be explained later. Second, this measure was objective and was readily obtained by the use of a simple timing device which could be set and reset by staff when restraints were either removed or replaced. Third, without requiring any additional paperwork, it satisfied a state requirement—mechanical restraints are prohibited by law unless there is documentation that they are removed every two hours and a safety check performed. These measurements, therefore, provided both a useful indicator of clinical status and a way to comply with a regulation.

The initial success of this strategy was shown by the running record in figure 1. Within the first four days after Peter was admitted to the unit, he was spending most of his time out of restraints. By the fourth day, the staff was able to remove restraints and keep them off for five consecutive work shifts, although there was a “relapse” when he began to attend his day program.

The start of the day program is an understandably stressful experience for any blind per­son who must learn to navigate his way around a new environment, even more so for an individual with autistic disorder, who may be agitated by changes in routine and exposure to strangers.

Figure 1: Peter’s time out of restraints per shift, March 4 to March 15

Figure 1 shows that with the start of the day program, Peter once again was spending his days in restraints. He was, however, spending his nights free of restraints. As he was pressed to perform routines that were spelled out in his treatment plan, this tendency became more pronounced, until, on the 15th, he spent a large portion of the night in restraints. (The routines involved in this part of the program were activities that were intended to increase his functional independence, such as teaching him to brush his teeth. In the past it had appeared that Peter used his self-injurious behavior as a means of ending the frustration attached to learning new routines and independent functioning.)

But how can we be sure that the changes seen in figure 1 aren’t just routine variation? The only way to be sure is to filter out the routine variation. Using the data for March 4 through March 31 results in natural process limits of 145 minutes to 340 minutes, as seen in figure 2. Only six of the 82 values fall within these limits. This is why we’re justified in interpreting the running record at face value: Peter is indeed cycling from good periods to bad periods, and these cycles are made manifest on the running record.

Figure 2: X chart for time out of restraints, March 4 to March 31

Initially, good periods were more common during the night shift, as Peter learned to sleep through the night without the need for restraints. The severity of Peter’s behavioral outbursts had necessitated the assignment of two staff members to protect him from harm on each work shift, and this was accomplished within the unit by means of a rotational assignment of overtime duty that brought virtually the entire roster in the building into contact with him during the course of this period. Some staff worked with Peter more effectively than others, and Pfadt learned to identify their presence (or absence) by carefully inspecting changes in the running record. Other “unexpected” changes also became predictable over time, such as the effect that parental visits had on Peter’s behavior, particularly when they didn’t occur on weekends as he had come to expect.

As shown in figure 2, a placebo treatment phase was initiated on March 18 in anticipation of the introduction of a new medication (Drug B) in combination with the medication Peter was taking when he entered the treatment unit (Drug A).

This placebo was introduced to minimize the possibility that an expectation of improvement might confound the interpretation of results due to the planned introduction of this new (and still somewhat experimental) medication at a later date. The staff were aware that Peter was taking a placebo in addition to his prescribed medication (this was not a double-blind placebo controlled study). To their surprise, both subjectively and objectively, as seen in figure 2, Peter seemed to improve following the use of one full tablet of the placebo.

Given the sheer volume of points on the shift-by-shift running record, they decided to start a summary graph consisting of the weekly averages. The average of the 21 shift-by-shift values for each week would be plotted as one point on this new chart. This provided a higher-level summary and allowed everyone to see the big picture more clearly than was possible with the shift-by-shift values. This weekly summary for the first 12 weeks is shown in figure 3, where the improvement on the placebo is shown in weeks three, four, and five.

Because of this documented improvement while Peter was on the placebo, the staff members insisted that the placebo was helping and initially opposed the plans to have it dis­continued. However, according to the protocol, they did discontinue the placebo at the end of week five. The running record clearly indicates that Peter’s improved level of functioning persisted even though the placebo had been discontinued. An explanation of this placebo ef­fect came when they discovered that these supposedly inert tablets contained significant traces of aspirin!

Figure 3: Weekly average time out of restraints per shift

Before introducing a new medication (Drug B), the dosage of the original medication was increased during weeks nine through 12 to determine if it was possible to help Peter control his self-injurious behaviors using only a single medication. Increasing the dosage of Drug A didn’t seem to do much.

These four weeks with the increased levels of Drug A were considered to be the baseline for evaluating the effect of Drug B. In order to obtain limits to use for this evaluation, these four weekly averages were used to obtain three moving ranges, and limits for an XmR chart were obtained in the usual manner. This baseline chart is shown in figure 4.

Figure 4: Baseline XmR chart for evaluating Drug B

Four values and three moving ranges are certainly a minimal amount of data to use in computing limits. Such limits will, of necessity, be soft. However, these soft limits were sufficient to detect the changes that followed. First, since the objective is to determine when a change has occurred, we don’t have to have perfect limits. Having Peter wait for six months while they collected 25 to 30 weeks worth of data prior to starting Drug B would have been unconscionable.

Second, this is an example of using a high-level summary (the weekly averages) in a chart for individual values. Other types of process behavior charts exist. One of these is the average and range chart. Although it might seem natural to use an average chart for these weekly averages, it’s inappropriate in this case. In part, this is because the individual values have a restricted range of 0 to 480 minutes, and many of these values are at the extremes of this restricted range. As a result the usual relationships used by the average and range chart no longer apply.

The new medication (Drug B) was introduced during week 13. The time out of restraints for each shift was still plotted (as in figure 1), and every week the weekly average was plotted on the X chart shown in figure 5.

Figure 5: X chart for weekly averages used to evaluate Drug B

The chart in figure 5 was continually updated and inspected during this drug trial, leading them to the conclusion that increasing the dosage of Drug B, up to the limit the treatment protocol allowed (100 mg per day), didn’t result in improved functioning. In fact, the higher doses (i.e., 75 mg and 100 mg) of Drug B were associated with a regression in clinical status. Not only is this seen in the points below the lower limit in figure 5, but it was borne out by the subjective impressions of staff and other objective indicators. Peter was clearly not doing well on Drug B—he inflicted some injuries to his face that required medical interventions (sutures and use of antibiotics to treat an infected wound). The reversal of this regression in clinical status that accompanied the decreased dosages of Drug B confirmed this overall conclusion.

The introduction of Drug C (administered in combination with his original medication) during week 28 (on September 15) produced a dramatic improvement in clinical status. This improvement was shown objectively by the weekly averages in figure 6 and was confirmed by the reports of the staff working with Peter. From Week 33, the use of restraints was virtually eliminated, except for one instance that coincided with an ear infection and another that occurred with an episode of cellulitis. Treatment with an antibiotic seems to have resulted in a decrease in agitation and also avoided a large scale-relapse, such as was reported in the past when Peter had recurrent ear infections.

Figure 6: X chart for weekly average time out of restraints

Peter’s parents, who had struggled with the challenge of managing his self-injurious behaviors for nearly 23 years, wrote the following about this whole decision-making process.

“The statistical process of measuring our son’s behavior has enabled those who have interpreted the graphs to have a clearer picture of what underlying factors influence the exhibited behaviors. The precise recording and interpretation of the daily processes have pinpointed the incidents that our son is unable to verbalize or relate to his caregivers. Through the years it has been difficult to identify and understand those factors that may have contributed to his complex behavior, which is manifested by severe battering of his face and biting of his hands."

“Many behavior-altering programs have been tried with varying responses in improved or worsening behavior. It has always been difficult, in a retrospective approach, to try to sort out those treatment techniques that were effective. This sorting was usually left to subjective analysis and interpretive impressions by observers and involved staff who interacted with our son in his daily management and treatment regimen. It appears that with the mode of precise recording and analysis using the continuous graph formats that were used with our son in this program, one could relate his specific behavior at a given point in time to any factor in his daily life that would affect his behavior pattern. This has permitted us to correlate changes in behavior and response to treatment as a total interaction of all the factors, e.g., starting, changing, or terminating medication; parental visits as related to program days and nonprogram days; and different interactions with specific members of staff. A major triggering factor had always been the undetectable onset of illness, e.g., earache, sore throat, or viral infection, which would now be evident on analysis of his behavior chart."

“We feel that this is a far more appropriate approach toward the trials of medication in attempts to control the self-injurious behavior. In all the years of our son’s behaviors, since the age of three, we have never felt that a treatment program and staff were as totally involved with him as has been the case during the past year.”

In summary, the charts provided objective criteria for making data-based clinical decisions, which resulted in improved outcomes for the patient. Compared with the marked lack of success that characterized prior courses of treatment, the results are nothing less than spectacular. The visual display of the available data provided a clear and unequivocal way of understanding and evaluating both the status of the client and the apparent effectiveness of the treatment. The charts are the voice of the process, and this voice is an important part of any treatment strategy.

Summary

Process behavior charts provide an objective way of separating routine variation (probable noise) from exceptional variation (potential signals). Once you and your co-workers have a way to separate potential signals from probable noise, you’ll find it much easier to agree on what constitutes a signal and, therefore, what deserves attention. At the same time, you and everyone else will spend less time chasing noise, dreaming up explanations of routine variation, and generally spinning your wheels.

Thus, the real secret of the effective use of process behavior charts is in the follow-through. It’s how you use the charts to discover what your process can do, or can be made to do. It’s how you interact with the chart to understand the behavior of your process. It’s how you communicate your process knowledge to others in a way that they can comprehend quickly and easily.

A process behavior chart is an operational definition of continual improvement. It answers the second and third questions needed to make progress.

Figure 7: A process behavior chart is an operational definition of improvement

In short, the secret of the effective use of process behavior charts is the way of thinking that goes with them. And you only get this through sound teaching and practice. Where this way of thinking is practiced, productivity increases, quality improves, and the organization’s competitive position is strengthened.

About The Authors