How not to improve a process

One of the things that Six Sigma is all about is data driven decision making.  This seems like a good thing to do intuitively, but I think people skip it anyway because getting the data is difficult a lot of time.  Now, your experience probably won’t be as disastrous as the one I’m about to describe to illustrate my point, but it could get there…

Fortunately, my example has nothing to do with a real process I was working on.  I was in black belt training and we were using the Statapult to learn about various statistical techniques.  I am, by nature, super competitive so I wanted to win the competition that was set up between the various teams.  That meant putting the most balls on target and then presenting out your findings (your learning experience) to the teams at the end.

For those of you not familiar with the Statapult, I’ll provide some background.  Essentially, it’s a very small catapult that fires wiffle and pingpong balls.  It has a bunch of adjustments on it.  Here’s a picture of it:

A statapult

You have many adjustments you can make to it.  There’s the tension pin, the stop pin, the pullback angle, the rubber band connection point on the arm and the cup that holds the ball on the arm.  Two things need to be tackled with the statapult experiment.

  1. You have to figure out how the various adjustments affect the distance the ball flies.  After all, the goal is to launch the ball onto some target a random distance away.
  2. You have to figure out how to get the variation out of the system so that you can do #1.  This is critical.  There is a lot of noise in an uncontrolled statapult.  The rubber band is strong enough to move the entire statapult when released even if a couple hefty folks are holding it down.  And then there’s other things, like the tension pin which rotates in it’s slot as you pull back the rubber band.

In fact, it’s that very tension pin that taught me more about the statapult than anything else.  Here’s the important message that I want to demonstrate to you: do not improve a process based on your assumptions because you can do bad, bad things without realizing it.

After getting a day to play with the statapult, we decided (again, being very competitive) that we needed some serious improvements to control variation.  I went to the hardware store on the way home and bought:

  • 4 2 1/2 inch lag bolts
  • 16 washers
  • 4 wing nuts
  • A roll of foam insulating tape
  • 2 metal plates
  • A piece of 1/4″ plywood for a base (I had this in my garage)

When we came back in, I built a base to control our statapult’s movement.  We drilled holes in the plywood and pushed the bolts (with washers to prevent rip-out) through the board.  We then placed the statapult between the bolts and used the metal plates, more washers and wing nuts to hold the statapult to the base.  I attached the insulating foam to the underside of the metal plates to a) protect the statapult from damage (they’re insanely expensive) and b) to provide shock absorption. 

Finally we looked at what else we could do to take noise out of the system.  Well surely that spinning tension pin was a bad thing.  Every time we pulled back the arm it would rotate and the rubber band was clearly slipping over it.  This had to be a bad thing, so we clamped down the tension pin so it wouldn’t rotate.

By the time we were done with this, we were learning about Design of Experiments (DOEs).  We were doing a 2k DOE, which is pretty straightforward and we expected that when we were done that we’d have the best results out of the entire class.  After all, we had all the noise under control. 

The reality was that we had one of the worst results and we didn’t understand why.  The Master Black Belt teaching the class took a look at our data and concluded it was because of the firing angles we had chosen.  For a DOE to be effective it helps to have at least one continuous variable to adjust.  All the other adjustments on the statapult are discrete, so it is important to use a wide range of firing angles to get good fine control over the firing distance. Because of where we were in the classroom, if we used a wide range of options on the firing angle we’d launch the ball right into the wall.  This meant we couldn’t get a measurement on the shot and therefore wouldn’t be able to compute the results.  To compensate we used a firing angle of between 135 and 140 degrees for our experiments and it seemed like that might have not been enough variety in ranges.

Our MBB was kind enough to stay with us and do the experiment over with a wider range of firing angles (and a bit more room to fire) and see what we got.  The results still sucked.  It wasn’t the firing angle, but we discovered something very interesting.  When you do a DOE, you randomize the order in which you adjust all the parameters.  You do this to avoid introducing bias into the experiment.  You also take two measurements at each set of parameters.  So, if you were using settings A, B and C in the first shot, you might do 15 more shots with various settings before coming back around and doing settings A, B and C again.  Because it was getting late our MBB said to skip that randomizing because each time you do that you have to reset the whole device and it takes a while.  So, we were doing shot 1 and shot 2 at settings A, B, and C.  And we found out something crazy, the two shots weren’t producing anywhere near the same result each time.  If you do the same settings on the statapult and you’ve gotten rid of all the noise in the system it should produce the same result (or at least darn close to it).

That’s when we had an epiphany.  Locking down the tension pin rotation was a bad thing!  Yes, indeed the rubber band did move over the rotating pin but it was a good thing, not a bad thing.  By allowing the pin to rotate it equalized the tension on the rubber band on both sides of the pin.  Without it, sometimes the rubber band would be really tight and sometimes really loose and result in unpredictable shots.

Alas, it was now quite late and our MBB was unable to stay for round number 3.  One of my teammates and I did the entire experiment again with just the two of us (it’s much easier with 4 or more people).  And finally, we got a good DOE and a regression equation with a 95.6% R-Sqr (adjusted).

In the end, we didn’t win the competition.  It turns out there’s one more noise factor in the system we didn’t count on – having everyone else in class watch you during competition makes you really nervous.  We did come in second place, primarily because all our failures let us tell one of the best learning experiences about the statapult.

Getting back to my message for you of “do not improve a process based on your assumptions because you can do bad, bad things without realizing it” you can see how making bold assumptions about the statapult process resulted in us making the system work worse than it did if we just hadn’t played with it at all.  This is what you should take away about real-life process improvement.  It is not enough to think you know what’s right for the process, if you don’t understand how the process works, then don’t play with it.  Or do play with it, but have the fortitude to admit you screwed it up and now know enough to not do that again.

4 thoughts on “How not to improve a process”

  1. Our SOP’s are available for anyone to brush up on factors surrounding noise reduction on the NCMR Catapult. Avail yourself of them for a more rewarding, and possibly winning experience. Great piece.

  2. Not sure what to make of the statement: “for a DOE to be effective it must have at least one continuous variable to adjust.” I suppose this is a way of saying you’d like to have an infinite range of levels to select from. At the end of the day though you are still going to have to set levels to test even in the real non-Statapult world and even in the world of all truly continuous variables. There have been many successful design optimizations with all quantum and discrete variables. Your adjustments in these cases are typically coarse, not fine.

  3. Yes true, I should have been clearer and said for the “DOE for the statapult to be effective it must have at least one continuous variable to adjust.” Of course, which the point is, if you want fine control, you’re going to need an adjustment which allows for it. By having such a narrow range, although still possible to select 135.1, 135.2, and even finer increments thereof, we limited ourselves to basically coarse adjustments. 5 degrees of arm angle just wasn’t going to cover the ‘dead zones’ in between the distance changes made by the coarser variables.

    Thanks for the comment! I’m always looking to improve the readability of my entries and will make updates!

Leave a Reply

Your email address will not be published. Required fields are marked *