BwN Building Blocks  BwN Toolbox  Pilots and cases  BwN Knowledge 

In the Netherlands, according to the OEIguideline an uncertainty analysis is mandatory in every socioeconomic costbenefit analysis (SCBA). For other countries having an SCBA guideline, the same holds true. The purpose of such an analysis is to determine the influence of uncertain assumptions on the balance (net present value) and ranking of the alternatives. In many SCBA's, the uncertainty analysis is executed in a rather informal way. With the tool presented here, a more formal probabilistic sensitivity analysis can be carried out, based on Monte Carlo simulation. The advantage of this method is that it will give insight in the cumulative effect of multiple uncertainties, including possible interactions between them. The cumulative effect is important for valuation of nature, because in nature valuation many entries in the balance sheet have rather large uncertainty margins. The formal sensitivity analysis yields information on which effects contribute most to total uncertainty. This insight can help decision makers in focusing efforts on issues producing the highest uncertainty. 
General Tool Description
Uncertainties in socioeconomic costbenefit analysis
A socioeconomic costbenefit analysis (SCBA) is used to assess the welfare impacts of large infrastructural projects. The welfare effect of a project is usually expressed in terms of net present value (in euro’s, dollars, or other currency).
In practice, an SCBA consists of the following eight steps. Here, we focus on step 7: sensitivity analysis. Information about the other steps can be found in text books on costbenefit analysis (see the references of this tool description):
 Problem analysis
 Describing project alternatives with respect to a baseline
 Cost calculation of every alternative
 Identification and assessment of physical impacts for every alternative
 Identification, quantification and monetarisation of welfare impacts for every alternative
 Discounting and making up the balance
 Sensitivity analysis
 Conclusions and recommendations
Usually, the estimation of uncertainties in the effects is done in a rather simple way by changing the basic assumptions of one effect at a time. The drawback of this socalled sensitivity analysis is that uncertainties are only investigated in the vicinity of the ‘pivot’ point in parameter space (i.e. the standard setting around which each input is varied) and interactions and cumulative effects of uncertainties are poorly understood. A method to explore the entire parameter space and to include interactions and cumulative effects of uncertainties in a more formal way is Monte Carlo simulation.
Monte Carlo simulation
Monte Carlo simulation is a method based on repeated random sampling of inputs to a deterministic model or calculation procedure. For each simulation all input variables are randomly drawn from predefined probability density functions. For each set of inputs a deterministic model run or calculation is made. The outputs of a large number of such simulations are analysed statistically, to yield probability density functions of the output variables.
Usage skills
This tool can be used by SCBAspecialists with statistical skills or an SCBAspecialist and a statistician working together.
Relevance for Building with Nature
The Building with Nature strategy aims at including nature value in project design and using natural processes to human benefit. However, it is a strategy with more uncertainties than a traditional design, partly because the natural processes are sometimes poorly understood, partly because they include inherent uncertainties. Building with Nature would therefore benefit from a tool that estimates cumulative uncertainties in a socioeconomic costbenefit analysis, thus enabling to formally incorporate uncertainties in important investment decisions.
This tool can be used in any investment project where an SCBA is made and where uncertainties play an important role. The SCBA is usually applied in an early stage (e.g. a feasibility study) of infrastructure developments requiring major investments.
How to Use
Phased plan process
Performing a Monte Carlo simulation on an existing model or calculation procedure requires the following steps:
 Define the uncertainties in the input variables (probability distribution and parameters);
 Generate a number of sets of input variables randomly drawn from the predefined probability distributions;
 Perform a simulation (model run, calculation) with each of the datasets;
 Send the outputs of each simulation to a dataset;
 Statistically analyse all outputs in the dataset.
Requirements
The reliability of the results improves with the number of simulations performed. In the example presented in the practical application 10.000 simulations were done. Monte Carlo simulation can be performed with a number of mathematical software packages. Here MatLab was used in combination with an Excel spreadsheet for the SCBAcomputations. Other software packages have not been tested when developing this tool description.
Advice and recommendations
SCBA’s are to be made by trained specialists, as they are susceptible to mistakes. Although most people understand the concept of costs and benefits, carrying out a thorough SCBA for a large project is not a simple task. SCBA in combination with Monte Carlo simulation makes the analysis even more complex, because it also requires statistical expertise.
Practical Applications
In this section we present a casestudy in which we have applied Monte Carlo simulation in combination with a socioeconomic costbenefit analysis to new developments around Lake Grevelingen in the Netherlands (Witteveen+Bos 2011).
Case: socioeconomic costbenefit analysis lake Grevelingen
Lake Grevelingen is a former estuary, part of the Rhine/Meuse/Scheldt complex in the southwest of the Netherlands. Since the closure of the estuary by the Grevelingendam (1965) at the landward side and the Brouwersdam (1971) at the seaward side, and the construction of a saltwater inlet sluice (1978) ,it is the largest saltwater lake in Europe. In the meantime, Lake Grevelingen has developed into a valuable natural and recreational area of national and international significance. The lake is used for swimming, windsurfing, canoeing, diving, boating and fishing. Commercial fishing of lobster, eel and oyster is also an important function. However, the Brouwersdam has several sideeffects, such as the ecological degradation of the lake. In order to counter these sideeffects, the national government has started a feasibility study with the following objectives:
 Flood protection related to climate change;
 Improving water quality;
 Tidal Power Generation (renewable energy and innovation);
 Improvement of the regional economic structure (tourism, aquaculture);
 Restoration of tidal nature.
In the feasibility study 5 alternatives to reach these objectives were investigated. Here, we use the first alternative to show the applicability of Monte Carlo Simulation. This alternative consists of the following measures (see figure):
 The tide in Lake Grevelingen is reintroduced by making a connection to the North Sea by means of an inlet through the Brouwersdam.
 In this inlet a tidal power station is built to produce renewable energy.
 Lake Grevelingen is designed as a storage area for excess water from the rivers Rhine and Meuse. To enable this, an opening in the Grevelingendam between lake Grevelingen and lake VolkerakZoommeer has to be made. In this alternative the opening in the Grevelingendam is designed as a bridge. As a result, the VolkerakZoommeer will become a salt water lake as well.
 A lock is built in the Brouwersdam in order to create a connection between lake Grevelingen and the North Sea for recreational vessels.
Results of the socioeconomic costbenefit analysis
Table 1 presents the results of de socioeconomic costbenefit analysis, a study completed in 2011 (Witteveen+Bos, 2011). Column 1 presents the results of the original alternative described above. The results show that alternative 1 has a negative net present value. In the sensitivity analysis it was investigated whether it is possible to reduce the costs and to augment the benefits so as to obtain a positive balance. This was done via trial and error by changing the basic assumptions of the most important entries in the balance sheet:
 the investment and maintenance costs. Cost reduction was obtained by changing the design of the turbines, the shipping lock in the Brouwersdam and the design of the opening in the Grevelingendam. It became clear that basic design assumptions strongly affect the investment costs;
 energy yield. Energy prices fluctuate considerably over the years. An important question was how much energy prices should go up so as to obtain a positive balance (breakeven point). From this computation it became clear that energy prices should double from 0.06 euro/kWh to 0.12 euro/kWh in order to give alternative 1 a positive balance.
Column 2 presents the results under the assumption that all design optimizations are effectuated, without increasing energy prices. The investments are much lower in this alternative because the original alternative was optimized and some other choices were made (Witteveen+Bos, 2011). Column 2 shows that in this case alternative 1 can get a positive balance. However, this is only the case if all design optimizations will be incorporated. It is not clear how realistic this assumption is. Therefore a Monte Carlo simulation is carried out to quantify the probability of obtaining a positive balance.
Table 1. Results of the SCBA in million Euros
Costs 
original alternative 1 
optimized alternative 1 

investments 
602 
299 
operation and maintenance 
100 
54 
total costs 
702 
353 
Benefits 


energy yield 
226 
139 
recreation 
7 
7 
water sports 
2 
2 
shellfish fishery (mussels) 
91 
91 
flood control (safety) 
188 
188 
reduction of CO2emissions 
34 
16 
employment 
31 
31 
knowledge and innovation 
10 
10 
total benefits 
589 
484 
net present value (benefits  costs) 
113 
131 
ratio (benefits/costs) 
0,84 
1,37 
Monte Carlo simulation
For a Monte Carlo simulation it is necessary to have the probability distributions of the values used in the calculations for the SCBA. To limit the research effort we keep the smaller benefits fixed. Moreover, we assume that all probability distributions are normal. We only collect the parameters of the normal probability distributions for the costs and large benefits:
 Investment and maintenance costs;
 Flood protection benefits;
 Energy yield of the tidal power station;
 Profit of shellfish (mussels) fisheries.
Investment and maintenance costs
In general the height of the project costs depends on many factors. For the casestudy of Lake Grevelingen the total costs of the measures vary considerably between 702 and 353 million euro's, depending mainly on the design of the structures. The maintenance costs are calculated as a fixed percentage of the realisation costs.
Flood protection benefits
The benefit of water storage on Lake Grevelingen is that investments for increasing dike height along the lower reaches of the debouching rivers can be cancelled or postponed. Here two types of uncertainty are considered:
 the moment when investments are necessary;
 uncertainty in cost estimates.
Uncertainty about the moment when flood protection measures are necessary depends mainly on uncertainty in expected future water level statistics in the rivers. The meteorological scenario applied shows an uncertainty in the water level forecasts of approximately 0.2 meters (DHV/HKV 2010). On average it takes approximately 30 years for a rise in water level of 0.2 meters. This is the combined effect of sea level rise and increase of river floods. This means that the moment at which the investment has to be made has an uncertainty range of 30 years (see table 2).
The cost of increasing the height of dikes varies between EUR 5 million and EUR 10 million per km. In the MonteCarlo simulation EUR 5 million and EUR 10 million are used as the 5% and 95% probability of exceedance, with a mean value of EUR 7,5 million (see table 2).
Energy yield of the tidal power station
Here we consider two types of uncertainty:
 uncertainty in energy production;
 uncertainty in energy prices.
A hydraulic model was used to determine the energy production of the tidal power plant (Royal Haskoning 2010). This study gives some information about the bandwidth in energy production (see table 2).
For the MC simulation, a justifiable assumption is 0,2% growth per year for the 5% probability of exceedance level, 4.5% growth per year for the 95% probability of exceedance level and an average of 1.5% per year (see table 2).
Profit of shellfish (mussels) fisheries
Bringing back the tide in Lake Grevelingen creates opportunities for mussel farming. Here three types of uncertainty related to mussels are considered:
• uncertainty in mussel production;
• uncertainty in mussel prices per kg.
In the cost benefit analysis it is assumed that the Grevelingen will produce 10,000 tons of mussels, taking production numbers of the Oosterschelde as a reference. Based on further analysis of mussel production (Deltares, 2010) 7,500 tons was taken as the mean value in the MC simulation; the above 10,000 tons as the 95% probability of exceedance level and 5,000 tons as the 5% probability of exceedance level (see table 2).
There is a wide variation in the price of mussels. After analysis of various sources, in the MC simulation a 5% probability of exceedance level of EUR 0.25 per kilogram was assumed and a 95% probability of exceedance level of EUR 1.90 per kilogram, with an average of EUR 1.10 per kilogram (see first part or table 2).
The type of probability distribution can differ for each uncertainty, but in this case no reliable information was available to select distributions other than the normal distribution.
Uncertainty 
Type of distribution 
Mean value 
5% probability of exceedance 
95% probability of exceedance 

Investment year for dikes with height surplus from 0,24 m to 0, 5 m 
Normal 
2050 
2035 
2065 
Investment year for dikes with height surplus from 0,5 m to 0,74 m 
Normal 
2070 
2055 
2085 
Costsavings increasing dikes (million euro/km dike) 
Normal 
7,5 
5 
10 
Energy production (GWh per year) 
Normal 
193 
183 
203 
Energy price increase (% per year) 
Skewed 
1,5% 
0,2% 
4,5% 
Mussel production (million kilogram of mussels) 
Normal 
7,5 
5 
10 
Mussel price (euro per kg, price level of 2011) 
Normal 
1,10 
0,25 
1,90 
Mussel price increase (% per year) 
Skewed 
7% 
0% 
25% 
Results of the MC simulation
For alternative 1 an MC simulation consisting of 10.000 runs was performed. For each run input variables were randomly drawn from the probability distributions of table 2. The figure on the right shows the MC simulation results for the total costs, the total benefits and the balance. The expected net present value is EUR 248 million and the standard deviation EUR 337 million. The probability of a positive balance is still 23%.
As can be seen in the figure, the benefits vary more than the costs. The factors contributing to the benefits are added and therefore the uncertainties of these contributions are directly translated into the balance (the same holds true for the costs). To reduce the uncertainties, it is in this case advisable to do more research on the benefits. If you want to know which factor contributes most to the total uncertainty, an extra MC can be performed. This extra analysis has not been done in this practical application.
The case: lessons learned
Effort and time
First conclusion is that performing a costbenefit analysis with an MC simulation is not too complicated, but it is time consuming. The main difficulty is finding useful information for establishing the probability distributions. For different types of benefits one can rely on available databases which include standard values for calculating the physical effect and the impacts on welfare (including prices). In most cases they include the mean values only. Data providing information on type and parameters of the most suitable probability distribution are often lacking.
Reducing uncertainties
The MC simulation yields useful information on which items in the balance sheet contribute most to the total uncertainty in the net present value. This information can be used to guide further research so as to reduce uncertainty.
More comprehensive information in the decisionmaking process?
An MC simulation should result in more comprehensive information about uncertainties and thus add to the decisionmaking process. It is true that an MC simulation leads to more information (see figure 1): the figures show additional information about the bandwidth of costs, benefits and the balance between them. Based on the MCsimulation it is also possible to give a prediction of the probability of a positive balance between benefits and costs. The question is whether this ‘new’ information is useful and workable to the decision maker. Reporting that there is a 23% chance of a positive balance can be very confusing to decision makers, especially if he thinks this is a completely random process.
An advice to a decision maker should therefore give insight into the nature of the uncertainties in the study results presented. Decision makers, on the other hand, should be well aware of possible financial setbacks or breaks. Hence it can be concluded that MC simulation does have something to add to the decision making process.
References
CBS 2011 (a), website: Statline, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)
CBS 2011 (b), Statistisch Jaarboek 2011, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)
CPB, MNP, RPB (2006), Welvaart en leefomgeving  een scenariostudie voor Nederland in 2040. (In Dutch)
Deltares (2010). memo van J.van Gils: Expert judgement naar draagkracht van de Grevelingen voor schelpdierkweek (In Dutch)
DHV/HKV (2010), gevoeligheidsanalyse waterberging zuidwestelijke delta, versie 2.0. Rijkswaterstaat. (In Dutch)
Natuur en Recreatieschap Grevelingen (2010), MIRTverkenning Grevelingen  Notitie Reikwijdte en Detailniveau, August 2010. (In Dutch)
Royal Haskoning, (2010). Tidal power station in de Brouwersdam. Variantenstudie. Definitief rapport, in opdracht van Groenservice ZuidHolland. (In Dutch)
Ruijgrok, E.C.M., A.J. Smale, R. Zijlstra, R. Abma, R.F.A. Berkers, A.A. Nemeth, N. Asselman, P.P. de Kluiver, R.S. de Groot, U. Kirchholtes, P.G. Todd, E. Buter, P.J.G.J. Hellegers, F. A. Rosenberg, (2006). Kentallen waardering Natuur, Water, Bodem en Landschap, Hulpmiddel bij MKBA, Ministerie van LNV, Den Haag. (In Dutch)
Witteveen+Bos (2011), MKBA verkenning Grevelingen, Deventer. (In Dutch)
Suggestions for further reading
University of Virginia (2006) PROJECT MANAGEMENT GUIDELINE, APPENDIX D – Cost/Benefit Analysis Guide. ITRM Guideline CPM 11001. Date: January 23, 2006.
Mishan, E. J. CostBenefit Analysis. 2nd ed. New York: Praeger Publishers, 1976. Classic theoretical development of costbenefit analysis with applications in the public and private sector.
Riggs, James L. Engineering Economics: 2nd ed. New York: McGrawHill, 1982. Thorough presentation of project evaluation techniques including costbenefit analysis in the public sector.
Treasury Board of Canada Secretariat (2007). Canadian CostBenefit Analysis Guide. Regulatory Proposals. Catalogue No. BT585/2007. ISBN 9780662050391. This document is available on the Treasury Board of Canada Secretariat website .
European Union (2008) Guide to Cost Benefit Analysis of Investment Projects. Structural Funds, Cohesion Fund and Instrument for PreAccession. July 2008. EUROPEAN COMMISSION. Directorate General Regional Policy.
Carel J.J. Eijgenraam, Carl C. Koopmans, Paul J.G. Tang, A.C.P. Verster, (2000) EVALUATIE VAN INFRASTRUCTUURPROJECTEN; LEIDRAAD VOOR KOSTENBATENANALYSE. Deel I: Hoofdrapport en Deel II Capita Selecta. Onderzoeksprogramma Economische Effecten Infrastructuur