Modelling and Simulation as Thought Experiments

In my last post, I talked about the potential use of linear regression in cost estimation.  Linear regression is a simple type of model.  I spent much of my career building and using simulation models and mathematical models to predict system behaviour.  These models, that were sometimes complicated, were simplifications of the real-world that could be solved using a computer.

I recently read Jim Manzi's new book, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society. In it, he discusses the potential value of randomization to divide subjects into test and control groups during experiments. 

Manzi begins the book with a summary of the history of experimentation. He notes that the success of the physical sciences is directly related to the fact that the problems in those fields have low causal density. 

Manzi suggests that the reason that social science has made relatively little progress is that predicting human behaviour involves high causal density and is holistically integrated.

Manzi says that randomized field trials, that have proven useful in clinical trials, are being applied successfully by modern businesses. He suggests that they should be applied more widely in social science and public policy. He says that randomized field trials are the only scientific way to determine if the findings from social science research are valid and the proposed public policies will have the desired effect.

In Manzi's opinion, theory and experimentation are a continuous cycle of knowledge development. However, they are quite separate activities.  Theories can be developed in any manner one may wish.  However, experimentation involves a rigorous method that includes test and control groups and the ability to conduct replications.

By this reasoning, modelling and simulation can be considered an extensive form of theory development.  

For the predictions from models and simulations to be verified, one would need to conduct randomized field trials in the real-world.

Belief in Modelling and Simulation

Recently I read Nate Silver's book The Signal and the Noise: Why So Many Predictions Fail — but Some Don't. Nate Silver develops models and uses them to make predictions. He suggests that if the developer of the model thinks his model is good, he or she should be willing to bet on the predictions that it makes.

I also read Daniel Kahneman's book Thinking, Fast and Slow. Kahneman discusses Philip Tetlock's book Expert Political Judgment: How Good Is It? How Can We Know? which suggests that the best experts in making political estimates and forecasts are no more accurate than fairly simple mathematical models of their estimative processes. This is yet another confirmation of what Robyn Dawes termed "the robust beauty of improper linear models." The inability of human experts to out-perform models based on their expertise has been demonstrated in over one hundred fields of expertise over fifty years of research; one of the most robust findings in social science.

In an earlier post, I mentioned the company PRICE Systems Inc who use linear regression to estimate the acquisition cost of military equipment. In a paper by Francois Melesse and David Rose on behalf of the Armed Forces Comptroller called "The Mother of All Guesses", the authors suggest that linear regression can be used not only to estimate the cost of a new military equipment but also to estimate a confidence interval around the cost estimate.

Here is an example of how this can be done based on a sample of 13 observations.

Cost $Million	X1	X2	X3	X4
52.7	55	9	20	13
73.5	49	15	27	9
61.4	44	11	15	15
32.6	43	7	8	6
28.9	38	7	11	1
47.4	38	8	14	4
40.5	37	5	10	14
21.4	28	4	6	4
15.4	26	2	4	4
37.5	24	6	6	6
57.1	21	5	6	4
21.1	19	3	3	4
20.0	10	1	2	4

Using the Microsoft Excel linear regression application, they found the following statistics.

 

Regression Statistics
Multiple R	0.91694416
R Square	0.84078659
Adjusted R Square	0.76117989
Standard Error	8.89208319
Observations	13
ANOVA
	df	SS	MS	F	Significance F
Regression	4	3340.43608	835.109	10.56176	0.00280361
Residual	8	632.553147	79.06914
Total	12	3972.98923
	Coefficients	Standard Error	t Stat	P-value	Lower 95%	Upper 95%	Lower 95.0%	Upper 95.0%
Intercept	21.2250048	8.29795744	2.557859	0.033759	2.08988065	40.36013	2.089881	40.36013
X Variable 1	-0.72742589	0.40687546	-1.78783	0.111608	-1.6656824	0.210831	-1.66568	0.210831
X Variable 2	4.22796199	1.91636066	2.206245	0.058422	-0.1911736	8.647098	-0.19117	8.647098
X Variable 3	0.70018905	1.13353292	0.617705	0.553942	-1.9137425	3.314121	-1.91374	3.314121
X Variable 4	1.18724	0.70677474	1.6798	0.131508	-0.4425855	2.817065	-0.44259	2.817065

So the regression equation is

Cost = 21.225 - 0.727(X1) + 4.228(X2) + 0.700(X3) + 1.187(X4) 

Our new equipment has the following values for X1, X2, X3 and X4.

X1 = 33, X2 = 5, X3 = 8, X4 = 1

So the predicted cost is

Cost = 21.225 - 0.727(33) + 4.228(5) + 0.700(8) +1.187(1)= 25.15

That is, $25.15 million.

The standard error is $8.89 million.

So based on the normal distribution, 90% of the time the true value will be 1.645 standard deviations away from the estimate.  Thus, the 90% confidence interval on the estimate is

[25.15 – 1.645(8.89), 25.15 + 1.645(8.89)] 

= [$10.52 million, $39.78 million].

I would suggest that this estimate based on a linear regression model of past cost data would be a better prediction of the final cost than the expert opinion of the project manager who is subject to optimism bias.  You can bet on it.

Mental Simulation, Intuition and Insight

In an earlier post, I mentioned Gary Klein, who studies naturalistic decision making or intution.

In his book Sources of Power: How People Make Decisions , he explains how he started by interviewing fire-fighters. He found that when fire-fighters make life and death decisions, they don't consider alternatives and evaluate them according to criteria to determine the best option as suggested by the process recommended by many Operations Researchers for rational decision-making .

Instead, fire-fighters run mental simulations to try to predict the outcome of one course of action after another until they find one that they believe will work.

Without this understanding, it would appear that they are using intuition to make decisions. However, Klein believes that this type of mental simulation only works after many years of experience.

Also in an earlier post, I mentioned that the ultimate goal of Operations Research is the creation of a paradigm shift. Another word for paradigm shift that applies to individuals is “insight” or the “Aha” effect.

In this article, Klein explains how a friend of his gained insight with the help of an associate.  The associate ran the friend through a mental simulation.  Through the mental simulation, his friend could see the fallacy in his thinking and discover how to change is mindset.

A tool of Operations Research is computer simulation. Computer simulations can take many months to build. They also can be difficult to interpret and explain. Because of these latter issues, much of the time computer simulations do not have an impact commensurate with the effort to build them.

If we follow Gary Klein's advice, we should use mental simulation to explain the findings from our computer simulations. This might help us have more impact in changing paradigms.

Statistical Contingency Cost Estimation

Brent Flyvberg , who wrote Megaprojects and Risk: An Anatomy of Ambition, describes a technique for conducting statistical contingency cost estimation, called reference class forecasting.  This is another form of outside-view that could be useful for improving cost estimation processes.

I will provide an example of how it could help a defence capital program.

First, one needs to collect a representative set of data on the capital cost overruns from the past.  That is, collect data on the original cost estimate for the program and the final actual cost of the program.

Below is simulated cost overrun data for 50 capital programs.

Case #	Overrun
1	100%
2	110%
3	70%
4	140%
5	60%
6	90%
7	90%
8	110%
9	80%
10	90%
11	60%
12	100%
13	50%
14	110%
15	100%
16	110%
17	120%
18	110%
19	140%
20	130%
21	90%
22	100%
23	80%
24	80%
25	60%
26	70%
27	100%
28	80%
29	90%
30	110%
31	110%
32	90%
33	90%
34	100%
35	80%
36	120%
37	90%
38	80%
39	100%
40	80%
41	80%
42	110%
43	110%
44	60%
45	90%
46	140%
47	40%
48	110%
49	120%
50	110%

Then I can find the cumulative probability distribution function from this data.  I need to sort the data from lowest to highest and calculate the appropriate the percentile value for each value.

Below is a table showing the cumulative probability results for this sample.

Overrun	Percentile
40%	2%
50%	4%
60%	6%
60%	8%
60%	10%
60%	12%
70%	14%
70%	16%
80%	18%
80%	20%
80%	22%
80%	24%
80%	25%
80%	27%
80%	29%
80%	31%
90%	33%
90%	35%
90%	37%
90%	39%
90%	41%
90%	43%
90%	45%
90%	47%
90%	49%
100%	51%
100%	53%
100%	55%
100%	57%
100%	59%
100%	61%
100%	63%
110%	65%
110%	67%
110%	69%
110%	71%
110%	73%
110%	75%
110%	76%
110%	78%
110%	80%
110%	82%
110%	84%
120%	86%
120%	88%
120%	90%
130%	92%
140%	94%
140%	96%
140%	98%

Then I produce a smooth curve of the cost overrun versus the percentile.  See the table and graph below.

Overrun	Percentile
40%	2%
50%	4%
60%	9%
70%	15%
80%	25%
90%	41%
100%	57%
110%	75%
120%	88%
130%	92%

 

 

From this graph, I can use the cumulative percentage value on the vertical axis to lookup a cost overrun value on the horizontal axis.  In this way, I can estimate the probability of an actual cost overrun being less than or equal to a particular overrun value.  For example, 25% of the time the actual cost overrun will be less than or equal to 80% of the initial estimate, 50% of the time the actual cost overrun will be less than or equal 100%, and 90% of the time the actual cost overrun will be less than or equal to 125%.

An easier way to interrupt these results is by using the inverse of this function which I found by linear interpolation.  In this case, I can provide a confidence level that a particular contingency cost will cover the expected cost overrun.  See table and graph below for the inverse function.

Confidence Level	Contingency Cost
10%	62%
20%	75%
30%	83%
40%	89%
50%	96%
60%	102%
70%	107%
80%	114%
90%	125%

Thus, using this graph, if I wanted to be 90% confident that I covered the expected cost overrun, I would need to have a contingency cost of 125%.  A contingency cost of 60% would only provide 10% confidence of covering the expected cost overrun.