Structured Brainstorming and Pre-mortems

When I was working in the mid-1990's, I developed a technique that I called Structured Brainstorming based on the Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes . It began by writing an open-ended question on a flip chart. Then each member of the team would spend some quiet time recording their answers to the question on their individual notepads. Then I would go around the team, one member at a time, and ask them for their first idea. I would write it on the flip chart, in exactly the words that they told me, and assign a number to the idea. No one was allowed to comment on the ideas that were presented at this time. I would continue to go around the team one at a time until all of their ideas are recorded on the flip chart paper. Then, as a team, we would group the ideas into categories. We named the categories and I labelled them as letters of the alphabet. Then we would prioritize the categories using an anonymous ranking process in which I used a Borda Count methodology. The whole session usually took about 45 minutes. Then I would go away with the flip charts and write up a summary of the ideas, according to the prioritized categories.

When I was running a workshop, I had to train the facilitators of the teams on the technique. I would run a Structured Brainstorming session with the question: What could go wrong with this workshop? Most of the time, this exercise was very refreshing. The facilitators got their worries recorded and evaluated. They were often relieved that their worries were being taken seriously. This led them to conduct a successful workshop were they avoided the potential problems.

Daniel Kahneman, who wrote Thinking, Fast and Slow distrusts project managers' ability to predict future program costs because they often have in inside view. Kahneman's distrust for the intuition of project managers is criticized by Gary Klein who wrote Sources of Power: How People Make Decisions and is a proponent of the naturalistic decision making processes of experts.

Klein has a novel approach that he recommends for project managers which he calls a Performing a Project Premortem . In a pre-mortem, the project team imagines they have completed the project but it has failed to deliver on time and within budget. They imagine all the ways the project could go astray.

I thought this was similar to my Structured Brainstorming method of training facilitators for workshops.

The Inside View and the Outside View

Daniel Kahneman, in Thinking, Fast and Slow , tells a story about his experience in a group of people who were writing a book. They had worked a few months and written a couple of chapters.

So Daniel decided to run a little experiment. He asked the members of the group how long they thought it would take to finish the book. To ensure that he got the best estimate, he asked all of the members to estimate the length of time independently.

He then collected the estimates and found that the average length of time that they thought it would take was 1.5 years and the values ranged from 1 to 2 years.

Then he realized that one of the members of the group was an experienced publisher of such books. So Daniel asked him what he knew of the experiences of other groups like theirs. He thought carefully and said that groups like them usually took 8 to 10 years to finish a book like theirs and as many as 40% never finished. Daniel felt that this expert was providing an outside view of the project.

The group did what most people do with such information. They ignored it and persevered. They felt that their situation was different from the experiences of other groups because they had an inside view of their capabilities.

It eventually took 7 years to finish the book.

The problem, Kahneman suggests, is that people are unable to go from general results to specific cases. Therefore, they can seldom utilize the potential benefits of an outside view.

An Example of Optimism Bias

When I worked on Cost Estimation and Performance Measurement in Canadian Defence: A Principal-Agent-Monitor Perspective , I wanted to learn why political leaders continue to put up with poor cost estimation from project managers in the bureaucracy. I read about optimism bias among project managers who under-estimate costs and over-estimate benefits of their projects.

Daniel Kahneman, in his book Thinking, Fast and Slow , mentions a similar concept that he calls the planning fallacy.

Kahneman also describes an astounding example of poor cost estimation. The case is the construction of the Scottish Parliament Building. In the table below, I show the how the cost estimates changed over time (this is described fully here).

Date	Cost Estimate
24/01/97	10-40m pounds
06/07/98	50-55m pounds
17/06/99	109m pounds
05/04/00	195m pounds
01/11/01	241m pounds
01/10/02	295m pounds
01/07/03	373.9m pounds

The Scottish Parliament Building opened in October, 2005 and a final accounting of all the costs was completed in February 2007 at 414.4m pounds.

Something that Kahneman thinks would help to avoid this kind of under-estimation of costs is an outside view.  I will discuss this concept in my next few blog posts.

Deja Vu All Over Again

In my early years in the Center for Operational Research and Analysis while I was working on life cycle costing, the CF-18 program was under scrutiny from the Office of the Auditor General for cost overruns and creative accounting.  I discuss this extensively in Cost Estimation and Performance Measurement in Canadian Defence: A Principal-Agent-Monitor Perspective .

I didn't have direct personal involvement in this program but I remember at the time my director at logistics analysis and my future director at air operational research took a trip to the PRICE corporation headquarters.  PRICE was a leading company in the field of parametric costing.  They collected data on the cost of military systems and then used statistics regression to help estimate the cost of future military systems.  The visit by these two directors was intended to determine if the PRICE software and data base could help with the cost estimation of major programs, like the CF-18, during the Canadian Forces rearmament program of the 1980's.

These two directors came back with different impressions of the value of the PRICE software and data base.  My director at logistics analysis was in favour of acquiring the PRICE software and data base for the Canadian Forces.  My future director at air operational research thought the PRICE data base was not applicable to the Canadian Forces situation and their regression technique was not accurate.

The Canadian Forces didn't acquire the PRICE software and data base or use the services of the PRICE consultants.

Twenty years later in the early 2000's, I was doing a study of costing for the maritime forces.  I had an opportunity to visit the PRICE headquarters and I took a course on their software and how to use their data base.

I came home and wrote a trip report which described how the PRICE software could be used in the shipbuilding program that was starting in Canada at that time.  I recommended that the Canadian Forces acquire the services of the PRICE consultants to help with the costing of future military acquisitions for the Canadian Forces.  We didn't and instead continued to estimate the costs of our military acquisition programs poorly.

The Office of the Auditor General has recently criticized the Canadian Forces for underestimating the cost of the F-35 which was chosen to replace the CF-18.

It is deja vu all over again.

The Real Impact of Operational Research

The Centre for Operational Research and Analysis, where I worked for 33 years, likes to refer to their work as decision support.  However, I heard many times, while I worked there, the complaint that "if only the decision makers would follow our advice, things would be so much better".

I am reminded of a paper by Donella Meadows called Places to Intervene in a System.  This paper lists 12 leverage points in increasing level of importance.  Meadows is specifically referring to Systems Thinking but the lowest and highest leverage points are relevant to the discussion here.

The lowest leverage point is numbers.  Most operational research studies produce numbers.  Many operational researchers are physical scientists or mathematicians. Therefore, they are predisposed to look at problems numerically.  However, Meadows suggests that numbers rarely change behaviour and have little long-term impact.

The highest leverage point is paradigm shift.  This involves changing decision makers' mental models and world-view.  Although this can be a difficult and long process of hard work, it can be very worthwhile.

One paradigm that can be traced back to operational research is the idea of constrained optimization.  Most of the work in operational research is either a methodology or application of constrained optimization.  Constrained optimization is the first subject that is taught in a university operational research program.

The concepts behind constrained optimization have become a major paradigm in decision making today.  All leaders and managers implicitly know what a constraint is and what it means to maximize or minimize an objective.  They also know that if they can overcome a constraint, they can achieve a breakthrough to a new level of value or benefit.

Military leaders may wish to minimize the number of casualties subject to the constraint that they win the battle or the war.  Or they may wish to maximize the performance of a new weapon system subject to a budget constraint.

However, operational researchers are not the only people who realize that paradigms matter.  One of the paradigms from microeconomics may help operational researchers understand why many decision makers fail to accept their globally optimized results.

This is the concept of Pareto Improvement, in which, a change in allocation of resources will only be made if the result improves one person's situation and does not adversely affect anyone else.  For example, a naval admiral would be unwilling to give up a naval capability so that the money saved could be spent on an air force capability, even if the reallocation led to a greater amount overall capability for the Canadian Forces as a whole.

Another appreciation of this paradigm can be related to an individual decision maker and is referred to by Richard Thaler as the endowment effect.  In this case, a decision maker will value a capability they already have, more than a new capability that they might acquire by giving up the existing capability.

Thus, we have a case of the competing paradigms of global optimization and Pareto Improvement.  This could well be resolved by operational researchers taking more consideration of the current environment as it might constrain in their optimization calculations.

Introduction

Hello, My name is Ivan Taylor.  This blog is intended to share my research with friends and associates.  I have a BSc in Mathematics, an MSc in Information and Systems Science and PhD in Public Policy.  I spent 33 years working in the Canada's Defence Research and Development Agency's Centre for Operational Research and Analysis.  I am an active member of the System Dynamics Society and am an editor of the Systems Thinking World Journal.

I have written a number of conference papers on Life Cycle Cost Analysis, Expert Systems, and System Dynamics.  I have written a book based on my PhD thesis called Cost Estimation and Performance Measurement in Canadian Defence: A Principal-Agent-Monitor Perspective .  I have also written over 200 internal operational research papers in Defence Research and Development Canada.

I live in Ottawa, Ontario, Canada and Avon Park, Florida, USA with my wife Barb.  I have three grown daughters Julia, Laura and Sarah.  I enjoy men's fastpitch softball in Ontario and play senior's slowpitch softball in Florida.  I have another blog which is intended to share my research on men's fastpitch softball.