Agile Methodology & the Marshmallow Challenge

I was recently thinking about this TED talk by Tom Wujec on team problem solving. He talks about the Marshmallow Challenge, where teams of four have to build the tallest free standing structure out of 20 sticks of spaghetti, 1 yard of tape and a marshmallow, with the marshmallow on top. What he has found is that kindergarteners actually do better than MBA graduates. He suggests that one of the reasons is that the MBA grads tend to try to find the single right plan and then focus all their effort on finding that plan, and when they get around to implementing it, and finally placing the marshmallow on top, it fails and they have run out of time. When I watched this talk it immediately made me think of the Waterfall Model for software development (see also Big Design Up Front). 

This then got me thinking about how I personally tend to approach problems, which is usually more like the kindergartener teams that Wujec talks about. They immediately start trying out different things until they find a workable approach. I do the same thing when I approach a new programming problem. I think about different ways to attack it, then I pick one that I think might work and jump in. I then iterate on that solution as needed until I have something that seems to be doing what I want, and if needed do some refactoring to reach the final product.   A good non-programming example of this approach came up this weekend. We were packing up from a camping trip and I was loading our fabric car top carrier with some of our gear. I had packed it up before we left, so I basically knew how much stuff I would fit if I was able to pack it exactly the same. My wife was giving me things to put into it and she indicated her doubts about how much would fit. It really irked me (to the point of me getting kind of nasty with her) because I was in the process trying different things and try to put things in so that they would fit, and it seemed like she was predicting that what I was doing was just not going to be possible. I would much rather try out some different approaches to solving the problem than decide beforehand that a thing can't be done.

It occurs to me that this whole iterative / agile approach is really just solving a problem with a kind of learn as you go philosophy. In software, it is rare that you have sufficient knowledge of all aspects of the technology and domain to do a 100% complete specification before you implement something (waterfall model). Instead, it is better to learn and gain knowledge about the domain and technology as you go by jumping in and writing some code. Looking at it as learning process also allows you to decide where planning does make sense (when you do have knowledge about certain problems and solutions). In the Marshmallow Challenge, the competitors who did the best were the architects and engineers and I think this is because they had some existing knowledge and training that allowed them to quickly select or narrow down the possible solutions. So, when we are architecting software or creating marshmallow structures, one of the critical first steps is recognizing what we know and more importantly what we don't know.