Monte Carlo simulation and Social Cost Benefit Analysis


Type: Method

Project Phase: Initiation, Planning and Design

Purpose: Integrating cumulative uncertainties of nature values in a socio-economic cost benefit analysis

Requirements: Statistical skills, knowledge on cost-benefit analyses

Relevant Software: Excel, mathematical software packages (eg. MATLAB)


 

 

About

Monte Carlo simulation is a method based on repeated random sampling of inputs to a deterministic model or calculation procedure. With Monte Carlo simulation, cumulative uncertainties of nature values can be integrated in a socio-economic cost benefit analysis. In the Netherlands, an uncertainty analysis is mandatory in every socio-economic cost-benefit analysis (SCBA), according to the EMVI-guideline. This also applies to other countries that have SCBA guidelines. The purpose of such an analysis is to determine the influence of uncertain assumptions on the balance (net present value) and ranking of the alternatives. In many SCBA's, the uncertainty analysis is executed in a rather informal way. With the tool presented here, a more formal probabilistic sensitivity analysis can be carried out, based on the Monte Carlo simulation. The advantage of this method is that it provides insight in the cumulative effect of multiple uncertainties, including possible interactions between them. The cumulative effect is especially important for valuation of nature, because these values tend to have rather large uncertainty margins in the balance sheet. The formal sensitivity analysis yields information on which effects contribute most to total uncertainty. This insight can help decision makers in focusing efforts on issues producing the highest uncertainty.

 Read more

 

Uncertainties in socio-economic cost-benefit analysis

socio-economic cost-benefit analysis (SCBA) is used to assess the welfare impacts of large infrastructural projects. The welfare effect of a project is usually expressed in terms of net present value (in euro’s, dollars, or other currency). In practice, an SCBA consists of the following eight steps:

  1. Problem analysis
  2. Describing project alternatives with respect to a baseline
  3. Cost calculation of every alternative
  4. Identification and assessment of physical impacts for every alternative
  5. Identification, quantification and monetisation of welfare impacts for every alternative
  6. Discounting and making up the balance
  7. Sensitivity analysis
  8. Conclusions and recommendations

 

Here, we focus on step 7: sensitivity analysis. Information about the other steps can be found in text books on cost-benefit analysis (see the references of this tool description). Usually, the estimation of uncertainties of the effects is done in a rather simple way by changing the basic assumptions of one effect at a time. The drawback of this so-called sensitivity analysis is that uncertainties are only investigated in the vicinity of the ‘pivot’ point in parameter space (i.e. the standard setting around which each input is varied) and interactions and cumulative effects of uncertainties are poorly understood. The Monte Carlo simulation provides a method to explore the entire parameter space and to include interactions and cumulative effects of uncertainties in a more formal way.

 

Monte Carlo simulation

Monte Carlo simulation is a method based on repeated random sampling of inputs to a deterministic model or calculation procedure. For each simulation all input variables are randomly drawn from predefined probability density functions. For each set of inputs, a deterministic model run or calculation is made. The outputs of a large number of such simulations are analysed statistically, to yield probability density functions of the output variables.


Relevance for Building with Nature

The Building with Nature strategy aims at including nature value in project design and using natural processes to human benefit. However, it is a strategy with more uncertainties than a traditional design, partly because the natural processes are sometimes poorly understood, partly because they include inherent uncertainties. Building with Nature would therefore benefit from a tool that estimates cumulative uncertainties in a socio-economic cost-benefit analysis, thus enabling to formally incorporate uncertainties in important investment decisions. This tool can be used in any investment project where an SCBA is made and where uncertainties play an important role. The SCBA is usually applied in an early stage (e.g. a feasibility study) of infrastructure developments requiring major investments.

 

How to Use

SCBA’s are to be made by trained specialists, as they are susceptible to mistakes. Although most people understand the concept of costs and benefits, carrying out a thorough SCBA for a large project is not a simple task. SCBA in combination with Monte Carlo simulation makes the analysis even more complex, because it also requires statistical expertise. This tool may be used by SCBA-specialists with statistical skills or an SCBA-specialist and a statistician working together.

 Read more

 

Phased plan process

Performing a Monte Carlo simulation on an existing model or calculation procedure requires the following steps:

  1. Define the uncertainties in the input variables (probability distribution and parameters);
  2. Generate a number of sets of input variables randomly drawn from the predefined probability distributions;
  3. Perform a simulation (model run, calculation) with each of the datasets;
  4. Send the outputs of each simulation to a dataset;
  5. Statistically analyse all outputs in the dataset.

Requirements

The reliability of the results improves with the number of simulations performed. In the example presented in the practical application 10.000 simulations were done. Monte Carlo simulations can be performed with a number of mathematical software packages. Here MatLab was used in combination with an Excel spreadsheet for the SCBA-computations. Other software packages have not been tested when developing this tool description.

 

Practical Applications

In this section we present a case-study in which we have applied a Monte Carlo simulation in combination with a socio-economic cost-benefit analysis to new developments around Lake Grevelingen in the Netherlands (Witteveen+Bos 2011).

 Read more

 

Case: socio-economic cost-benefit analysis Lake Grevelingen

Lake Grevelingen is a former estuary, part of the Rhine/Meuse/Scheldt complex in the southwest of the Netherlands. Since the closure of the estuary by the Grevelingendam (1965) at the landward side and the Brouwersdam (1971) at the seaward side, and the construction of a saltwater inlet sluice (1978) ,it is the largest saltwater lake in Europe. In the meantime, Lake Grevelingen has developed into a valuable natural and recreational area of national and international significance. The lake is used for swimming, windsurfing, canoeing, diving, boating and fishing. Commercial fishing of lobster, eel and oyster is also an important function. However, the Brouwersdam has several side-effects, such as the ecological degradation of the lake. In order to counter these side-effects, the national government has started a feasibility study with the following objectives:

  • Flood protection related to climate change;
  • Improving water quality;
  • Tidal Power Generation (renewable energy and innovation);
  • Improvement of the regional economic structure (tourism, aquaculture);
  • Restoration of tidal nature.

In the feasibility study 5 alternatives to reach these objectives were investigated. Here, we use the first alternative to show the applicability of the Monte Carlo Simulation. This alternative consists of the following measures (see figure):

 

  1. The tide in Lake Grevelingen is reintroduced by making a connection to the North Sea by means of an inlet through the Brouwersdam.
  2. In this inlet a tidal power station is built to produce renewable energy.
  3. Lake Grevelingen is designed as a storage area for excess water from the rivers Rhine and Meuse. To enable this, an opening in the Grevelingendam between Lake Grevelingen and lake Volkerak-Zoommeer has to be made. In this alternative the opening in the Grevelingendam is designed as a bridge. As a result, the Volkerak-Zoommeer will become a salt water lake as well.
  4. A lock is built in the Brouwersdam in order to create a connection between lake Grevelingen and the North Sea for recreational vessels.

Results of the socio-economic cost-benefit analysis

The results of the socio-economic cost-benefit analysis are presented in the table below, a study completed in 2011 (Witteveen+Bos, 2011). Column 1 presents the results of the original alternative described above. The results show that alternative 1 has a negative net present value. In the sensitivity analysis it was investigated whether it is possible to reduce the costs and to augment the benefits so as to obtain a positive balance. This was done via trial and error by changing the basic assumptions of the most important entries in the balance sheet:

  • The investment and maintenance costs
  • Energy yield

 

Cost reduction was obtained by changing the design of the turbines, the shipping lock in the Brouwersdam and the design of the opening in the Grevelingendam. It became clear that basic design assumptions strongly affect the investment costs.

 

Energy prices fluctuate considerably over the years. An important question was how much energy prices should go up so as to obtain a positive balance (break-even point). From this computation it became clear that energy prices should double from 0.06 euro/kWh to 0.12 euro/kWh in order to give alternative 1 a positive balance. Column 2 presents the results under the assumption that all design optimisations are effectuated, without increasing energy prices. The investments are much lower in this alternative because the original alternative was optimised and some other choices were made (Witteveen+Bos, 2011). Column 2 shows that in this case alternative 1 can get a positive balance. However, this is only the case if all design optimisations will be incorporated. It is not clear how realistic this assumption is. Therefore a Monte Carlo simulation is carried out to quantify the probability of obtaining a positive balance.

Table 1: Results of the SCBA in million Euros:

Costs

original alternative 1

optimized alternative 1

investments

602

299

operation and maintenance

100

54

total costs

702

353

Benefits

 

 

energy yield

226

139

recreation

7

7

water sports

2

2

shell-fish fishery (mussels)

91

91

flood control (safety)

188

188

reduction of CO2-emissions

34

16

employment

31

31

knowledge and innovation

10

10

total benefits

589

484

net present value (benefits - costs)

-113

131

ratio (benefits/costs)

0,84

1,37

 

Monte Carlo simulation

For a Monte Carlo simulation it is necessary to have the probability distributions of the values used in the calculations for the SCBA. To limit the research effort we keep the smaller benefits fixed. Moreover, we assume that all probability distributions are normal. We only collect the parameters of the normal probability distributions for the costs and large benefits:

  1. Investment and maintenance costs;
  2. Flood protection benefits;
  3. Energy yield of the tidal power station;
  4. Profit of shell-fish (mussels) fisheries.

Investment and maintenance costs

In general the height of the project costs depends on many factors. For the case-study of Lake Grevelingen the total costs of the measures vary considerably between 702 and 353 million euro's, depending mainly on the design of the structures. The maintenance costs are calculated as a fixed percentage of the realisation costs.

 

Flood protection benefits

The benefit of water storage on Lake Grevelingen is that investments for increasing dike height along the lower reaches of the debouching rivers can be cancelled or postponed. Here two types of uncertainty are considered:

  • the moment when investments are necessary;
  • uncertainty in cost estimates.

Uncertainty about the moment when flood protection measures are necessary depends mainly on uncertainty in expected future water level statistics in the rivers. The meteorological scenario applied shows an uncertainty in the water level forecasts of approximately 0.2 meters (DHV/HKV 2010). On average it takes approximately 30 years for a rise in water level of 0.2 meters. This is the combined effect of sea level rise and increase of river floods. This means that the moment at which the investment has to be made has an uncertainty range of 30 years (see table below).

 

The cost of increasing the height of dikes varies between EUR 5 million and EUR 10 million per km. In the Monte-Carlo simulation EUR 5 million and EUR 10 million are used as the 5% and 95% probability of exceedance, with a mean value of EUR 7,5 million (see table below).

Energy yield of the tidal power station

Here we consider two types of uncertainty:

  • uncertainty in energy production;
  • uncertainty in energy prices.

A hydraulic model was used to determine the energy production of the tidal power plant (Royal Haskoning 2010). This study gives some information about the bandwidth in energy production (see table below). 


For the MC simulation, a justifiable assumption is -0,2% growth per year for the 5% probability of exceedance level, 4.5% growth per year for the 95% probability of exceedance level and an average of 1.5% per year (see table below).

 

Profit of shell-fish (mussels) fisheries

Bringing back the tide in Lake Grevelingen creates opportunities for mussel farming. Here three types of uncertainty related to mussels are considered:
• uncertainty in mussel production;
• uncertainty in mussel prices per kg.


In the cost benefit analysis it is assumed that the Grevelingen will produce 10,000 tons of mussels, taking production numbers of the Oosterschelde as a reference. Based on further analysis of mussel production (Deltares, 2010), 7,500 tons was taken as the mean value in the MC simulation; the above 10,000 tons as the 95% probability level, and 5,000 tons as the 5% probability of exceedance level (see table below).

 

There is a wide variation in the price of mussels. After analysis of various sources, in the MC simulation a 5% probability of exceedance level of EUR 0.25 per kilogram was assumed and a 95% probability of exceedance level of EUR 1.90 per kilogram, with an average of EUR 1.10 per kilogram (see table below).The type of probability distribution can differ for each uncertainty, but in this case no reliable information was available to select distributions other than the normal distribution.

 

Table 2: Valuation of uncertainties with 5% and 95% thresholds, and their mean values:

Uncertainty

Type of distribution

Mean value

5% probability of exceedance

95% probability of exceedance

Investment year for dikes with height surplus from 0,24 m to 0, 5 m

Normal

2050

2035

2065

Investment year for dikes with height surplus from 0,5 m to 0,74 m

Normal

2070

2055

2085

Cost-savings increasing dikes (million euro/km dike)

Normal

7,5

5

10

Energy production (GWh per year)

Normal

193

183

203

Energy price increase (% per year)

Skewed

1,5%

-0,2%

4,5%

Mussel production (million kilogram of mussels)

Normal

7,5

5

10

Mussel price (euro per kg, price level of 2011)

Normal

1,10

0,25

1,90

Mussel price increase (% per year)

Skewed

7%

0%

25%

Results of the MC simulation

For alternative 1 an MC simulation consisting of 10.000 runs was performed. For each run, input variables were randomly drawn from the probability distributions of table 2. The figure on the right shows the MC simulation results for the total costs, the total benefits and the balance. The expected net present value is EUR -248 million and the standard deviation EUR 337 million. The probability of a positive balance is still 23%.

 

As can be seen in the figure, the benefits vary more than the costs. The factors contributing to the benefits are added and therefore the uncertainties of these contributions are directly translated into the balance (the same holds true for the costs). To reduce the uncertainties, it is in this case advisable to do more research on the benefits. If you want to know which factor contributes most to the total uncertainty, an extra MC can be performed. This extra analysis has not been done in this practical application.

 

Lessons learned

Effort and time: First conclusion is that performing a cost-benefit analysis with an MC simulation is not too complicated, but it is time consuming. The main difficulty is finding useful information for establishing the probability distributions. For different types of benefits, one can rely on available databases which include standard values for calculating the physical effect and the impacts on welfare (including prices). In most cases they include the mean values only. Data providing information on type and parameters of the most suitable probability distribution are often lacking.

 

Reducing uncertainties: The MC simulation yields useful information on which items in the balance sheet contribute most to the total uncertainty in the net present value. This information can be used to guide further research so as to reduce uncertainty.


Decision-making: An MC simulation should result in more comprehensive information about uncertainties and thus add to the decision-making process. It is true that an MC simulation leads to more information (see figure above): the graphs show additional information about the bandwidth of costs, benefits and the balance between them. Based on the MC-simulation it is also possible to give a prediction of the probability of a positive balance between benefits and costs. The question is whether this ‘new’ information is useful and workable to the decision maker. Reporting that there is a 23% chance of a positive balance can be very confusing to decision makers, especially if he thinks this is a completely random process. 


Any advice to a decision maker should therefore give insight into the nature of the uncertainties in the study results presented. Decision makers on the other hand, should be well aware of possible financial setbacks or breaks. In general, it can be concluded that MC simulations positively contribute to the decision making process.

 

References

 Read more

  • CBS 2011 (a), website: Statline, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)
  • CBS 2011 (b), Statistisch Jaarboek 2011, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)
  • CPB, MNP, RPB (2006), Welvaart en leefomgeving - een scenariostudie voor Nederland in 2040. (In Dutch)
  • Deltares (2010). memo van J.van Gils: Expert judgement naar draagkracht van de Grevelingen voor schelpdierkweek (In Dutch)
  • DHV/HKV (2010), gevoeligheidsanalyse waterberging zuidwestelijke delta, versie 2.0. Rijkswaterstaat. (In Dutch)
  • Natuur- en Recreatieschap Grevelingen (2010), MIRT-verkenning Grevelingen - Notitie Reikwijdte en Detailniveau, August 2010. (In Dutch)
  • Royal Haskoning, (2010). Tidal power station in de Brouwersdam. Variantenstudie. Definitief rapport, in opdracht van Groenservice Zuid-Holland. (In Dutch)
  • Ruijgrok, E.C.M., A.J. Smale, R. Zijlstra, R. Abma, R.F.A. Berkers, A.A. Nemeth, N. Asselman, P.P. de Kluiver, R.S. de Groot, U. Kirchholtes, P.G. Todd, E. Buter, P.J.G.J. Hellegers, F. A. Rosenberg, (2006). Kentallen waardering Natuur, Water, Bodem en Landschap, Hulpmiddel bij MKBA, Ministerie van LNV, Den Haag. (In Dutch)
  • Witteveen+Bos (2011), MKBA verkenning Grevelingen, Deventer. (In Dutch)

Suggestions for further reading

  • University of Virginia (2006) PROJECT MANAGEMENT GUIDELINE, APPENDIX D – Cost/Benefit Analysis Guide. ITRM Guideline CPM 110-01. Date: January 23, 2006.
  • Mishan, E. J. Cost-Benefit Analysis. 2nd ed. New York: Praeger Publishers, 1976. Classic theoretical development of cost-benefit analysis with applications in the public and private sector.
  • Riggs, James L. Engineering Economics: 2nd ed. New York: McGraw-Hill, 1982. Thorough presentation of project evaluation techniques including cost-benefit analysis in the public sector.
  • Treasury Board of Canada Secretariat (2007). Canadian Cost-Benefit Analysis Guide. Regulatory Proposals. Catalogue No. BT58-5/2007. ISBN 978-0-662-05039-1. This document is available on the Treasury Board of Canada Secretariat website .
  • European Union (2008) Guide to Cost Benefit Analysis of Investment Projects. Structural Funds, Cohesion Fund and Instrument for Pre-Accession. July 2008. EUROPEAN COMMISSION. Directorate General Regional Policy.
  • Carel J.J. Eijgenraam, Carl C. Koopmans, Paul J.G. Tang, A.C.P. Verster, (2000). EVALUATIE VAN INFRASTRUCTUURPROJECTEN; LEIDRAAD VOOR KOSTEN-BATENANALYSE. Deel I: Hoofdrapport en Deel II Capita Selecta. Onderzoeksprogramma Economische Effecten Infra-structuur.