Building with Nature BwN Guideline Environments Project phases Governance BwN Knowledge base
BwN Building Blocks BwN Toolbox Pilots and cases BwN Knowledge

Log in

In the Netherlands, according to the OEI-guideline an uncertainty analysis is mandatory in every socio-economic cost-benefit analysis (SCBA). For other countries having an SCBA guideline, the same holds true. The purpose of such an analysis is to determine the influence of uncertain assumptions on the balance (net present value) and ranking of the alternatives. In many SCBA's, the uncertainty analysis is executed in a rather informal way. With the tool presented here, a more formal probabilistic sensitivity analysis can be carried out, based on Monte Carlo simulation. The advantage of this method is that it will give insight in the cumulative effect of multiple uncertainties, including possible interactions between them. The cumulative effect is important for valuation of nature, because in nature valuation many entries in the balance sheet have rather large uncertainty margins. The formal sensitivity analysis yields information on which effects contribute most to total uncertainty. This insight can help decision makers in focusing efforts on issues producing the highest uncertainty.

    General Tool Description

    Uncertainties in socio-economic cost-benefit analysis

    socio-economic cost-benefit analysis (SCBA) is used to assess the welfare impacts of large infrastructural projects. The welfare effect of a project is usually expressed in terms of net present value (in euro’s, dollars, or other currency). 
    In practice, an SCBA consists of the following eight steps. Here, we focus on step 7: sensitivity analysis. Information about the other steps can be found in text books on cost-benefit analysis (see the references of this tool description):

    1. Problem analysis
    2. Describing project alternatives with respect to a baseline
    3. Cost calculation of every alternative
    4. Identification and assessment of physical impacts for every alternative
    5. Identification, quantification and monetarisation of welfare impacts for every alternative
    6. Discounting and making up the balance
    7. Sensitivity analysis
    8. Conclusions and recommendations

    Usually, the estimation of uncertainties in the effects is done in a rather simple way by changing the basic assumptions of one effect at a time. The drawback of this so-called sensitivity analysis is that uncertainties are only investigated in the vicinity of the ‘pivot’ point in parameter space (i.e. the standard setting around which each input is varied) and interactions and cumulative effects of uncertainties are poorly understood. A method to explore the entire parameter space and to include interactions and cumulative effects of uncertainties in a more formal way is Monte Carlo simulation.

    Monte Carlo simulation

    Monte Carlo simulation is a method based on repeated random sampling of inputs to a deterministic model or calculation procedure. For each simulation all input variables are randomly drawn from predefined probability density functions. For each set of inputs a deterministic model run or calculation is made. The outputs of a large number of such simulations are analysed statistically, to yield probability density functions of the output variables.

    Usage skills

    This tool can be used by SCBA-specialists with statistical skills or an SCBA-specialist and a statistician working together.

    Relevance for Building with Nature

    The Building with Nature strategy aims at including nature value in project design and using natural processes to human benefit. However, it is a strategy with more uncertainties than a traditional design, partly because the natural processes are sometimes poorly understood, partly because they include inherent uncertainties. Building with Nature would therefore benefit from a tool that estimates cumulative uncertainties in a socio-economic cost-benefit analysis, thus enabling to formally incorporate uncertainties in important investment decisions.

    This tool can be used in any investment project where an SCBA is made and where uncertainties play an important role. The SCBA is usually applied in an early stage (e.g. a feasibility study) of infrastructure developments requiring major investments.

    How to Use

    Phased plan process

    Performing a Monte Carlo simulation on an existing model or calculation procedure requires the following steps:

    1. Define the uncertainties in the input variables (probability distribution and parameters);
    2. Generate a number of sets of input variables randomly drawn from the predefined probability distributions;
    3. Perform a simulation (model run, calculation) with each of the datasets;
    4. Send the outputs of each simulation to a dataset;
    5. Statistically analyse all outputs in the dataset.

    Requirements

    The reliability of the results improves with the number of simulations performed. In the example presented in the practical application 10.000 simulations were done. Monte Carlo simulation can be performed with a number of mathematical software packages. Here MatLab was used in combination with an Excel spreadsheet for the SCBA-computations. Other software packages have not been tested when developing this tool description.

    Advice and recommendations

    SCBA’s are to be made by trained specialists, as they are susceptible to mistakes. Although most people understand the concept of costs and benefits, carrying out a thorough SCBA for a large project is not a simple task. SCBA in combination with Monte Carlo simulation makes the analysis even more complex, because it also requires statistical expertise.

    Practical Applications

    In this section we present a case-study in which we have applied Monte Carlo simulation in combination with a socio-economic cost-benefit analysis to new developments around Lake Grevelingen in the Netherlands (Witteveen+Bos 2011).

    Case: socio-economic cost-benefit analysis lake Grevelingen

    Lake Grevelingen is a former estuary, part of the Rhine/Meuse/Scheldt complex in the southwest of the Netherlands. Since the closure of the estuary by the Grevelingendam (1965) at the landward side and the Brouwersdam (1971) at the seaward side, and the construction of a saltwater inlet sluice (1978) ,it is the largest saltwater lake in Europe. In the meantime, Lake Grevelingen has developed into a valuable natural and recreational area of national and international significance. The lake is used for swimming, windsurfing, canoeing, diving, boating and fishing. Commercial fishing of lobster, eel and oyster is also an important function. However, the Brouwersdam has several side-effects, such as the ecological degradation of the lake. In order to counter these side-effects, the national government has started a feasibility study with the following objectives:

    • Flood protection related to climate change;
    • Improving water quality;
    • Tidal Power Generation (renewable energy and innovation);
    • Improvement of the regional economic structure (tourism, aquaculture);
    • Restoration of tidal nature.

    In the feasibility study 5 alternatives to reach these objectives were investigated. Here, we use the first alternative to show the applicability of Monte Carlo Simulation. This alternative consists of the following measures (see figure):

    1. The tide in Lake Grevelingen is reintroduced by making a connection to the North Sea by means of an inlet through the Brouwersdam.
    2. In this inlet a tidal power station is built to produce renewable energy.
    3. Lake Grevelingen is designed as a storage area for excess water from the rivers Rhine and Meuse. To enable this, an opening in the Grevelingendam between lake Grevelingen and lake Volkerak-Zoommeer has to be made. In this alternative the opening in the Grevelingendam is designed as a bridge. As a result, the Volkerak-Zoommeer will become a salt water lake as well.
    4. A lock is built in the Brouwersdam in order to create a connection between lake Grevelingen and the North Sea for recreational vessels.

    Results of the socio-economic cost-benefit analysis

    Table 1 presents the results of de socio-economic cost-benefit analysis, a study completed in 2011 (Witteveen+Bos, 2011). Column 1 presents the results of the original alternative described above. The results show that alternative 1 has a negative net present value. In the sensitivity analysis it was investigated whether it is possible to reduce the costs and to augment the benefits so as to obtain a positive balance. This was done via trial and error by changing the basic assumptions of the most important entries in the balance sheet:

    • the investment and maintenance costs. Cost reduction was obtained by changing the design of the turbines, the shipping lock in the Brouwersdam and the design of the opening in the Grevelingendam. It became clear that basic design assumptions strongly affect the investment costs;
    • energy yield. Energy prices fluctuate considerably over the years. An important question was how much energy prices should go up so as to obtain a positive balance (break-even point). From this computation it became clear that energy prices should double from 0.06 euro/kWh to 0.12 euro/kWh in order to give alternative 1 a positive balance.
      Column 2 presents the results under the assumption that all design optimizations are effectuated, without increasing energy prices. The investments are much lower in this alternative because the original alternative was optimized and some other choices were made (Witteveen+Bos, 2011). Column 2 shows that in this case alternative 1 can get a positive balance. However, this is only the case if all design optimizations will be incorporated. It is not clear how realistic this assumption is. Therefore a Monte Carlo simulation is carried out to quantify the probability of obtaining a positive balance.

    Table 1. Results of the SCBA in million Euros

    Costs

    original alternative 1

    optimized alternative 1

    investments

    602

    299

    operation and maintenance

    100

    54

    total costs

    702

    353

    Benefits

     

     

    energy yield

    226

    139

    recreation

    7

    7

    water sports

    2

    2

    shell-fish fishery (mussels)

    91

    91

    flood control (safety)

    188

    188

    reduction of CO2-emissions

    34

    16

    employment

    31

    31

    knowledge and innovation

    10

    10

    total benefits

    589

    484

    net present value (benefits - costs)

    -113

    131

    ratio (benefits/costs)

    0,84

    1,37

    Monte Carlo simulation

    For a Monte Carlo simulation it is necessary to have the probability distributions of the values used in the calculations for the SCBA. To limit the research effort we keep the smaller benefits fixed. Moreover, we assume that all probability distributions are normal. We only collect the parameters of the normal probability distributions for the costs and large benefits:

    1. Investment and maintenance costs;
    2. Flood protection benefits;
    3. Energy yield of the tidal power station;
    4. Profit of shell-fish (mussels) fisheries.

    Investment and maintenance costs

    In general the height of the project costs depends on many factors. For the case-study of Lake Grevelingen the total costs of the measures vary considerably between 702 and 353 million euro's, depending mainly on the design of the structures. The maintenance costs are calculated as a fixed percentage of the realisation costs.

    Flood protection benefits

    The benefit of water storage on Lake Grevelingen is that investments for increasing dike height along the lower reaches of the debouching rivers can be cancelled or postponed. Here two types of uncertainty are considered:

    • the moment when investments are necessary;
    • uncertainty in cost estimates.
      Uncertainty about the moment when flood protection measures are necessary depends mainly on uncertainty in expected future water level statistics in the rivers. The meteorological scenario applied shows an uncertainty in the water level forecasts of approximately 0.2 meters (DHV/HKV 2010). On average it takes approximately 30 years for a rise in water level of 0.2 meters. This is the combined effect of sea level rise and increase of river floods. This means that the moment at which the investment has to be made has an uncertainty range of 30 years (see table 2). 
      The cost of increasing the height of dikes varies between EUR 5 million and EUR 10 million per km. In the Monte-Carlo simulation EUR 5 million and EUR 10 million are used as the 5% and 95% probability of exceedance, with a mean value of EUR 7,5 million (see table 2).

    Energy yield of the tidal power station

    Here we consider two types of uncertainty:

    • uncertainty in energy production;
    • uncertainty in energy prices.

    A hydraulic model was used to determine the energy production of the tidal power plant (Royal Haskoning 2010). This study gives some information about the bandwidth in energy production (see table 2). 
    For the MC simulation, a justifiable assumption is -0,2% growth per year for the 5% probability of exceedance level, 4.5% growth per year for the 95% probability of exceedance level and an average of 1.5% per year (see table 2).

    Profit of shell-fish (mussels) fisheries

    Bringing back the tide in Lake Grevelingen creates opportunities for mussel farming. Here three types of uncertainty related to mussels are considered:
    • uncertainty in mussel production;
    • uncertainty in mussel prices per kg.
    In the cost benefit analysis it is assumed that the Grevelingen will produce 10,000 tons of mussels, taking production numbers of the Oosterschelde as a reference. Based on further analysis of mussel production (Deltares, 2010) 7,500 tons was taken as the mean value in the MC simulation; the above 10,000 tons as the 95% probability of exceedance level and 5,000 tons as the 5% probability of exceedance level (see table 2).

    There is a wide variation in the price of mussels. After analysis of various sources, in the MC simulation a 5% probability of exceedance level of EUR 0.25 per kilogram was assumed and a 95% probability of exceedance level of EUR 1.90 per kilogram, with an average of EUR 1.10 per kilogram (see first part or table 2).

    The type of probability distribution can differ for each uncertainty, but in this case no reliable information was available to select distributions other than the normal distribution.

    Uncertainty

    Type of distribution

    Mean value

    5% probability of exceedance

    95% probability of exceedance

    Investment year for dikes with height surplus from 0,24 m to 0, 5 m

    Normal

    2050

    2035

    2065

    Investment year for dikes with height surplus from 0,5 m to 0,74 m

    Normal

    2070

    2055

    2085

    Cost-savings increasing dikes (million euro/km dike)

    Normal

    7,5

    5

    10

    Energy production (GWh per year)

    Normal

    193

    183

    203

    Energy price increase (% per year)

    Skewed

    1,5%

    -0,2%

    4,5%

    Mussel production (million kilogram of mussels)

    Normal

    7,5

    5

    10

    Mussel price (euro per kg, price level of 2011)

    Normal

    1,10

    0,25

    1,90

    Mussel price increase (% per year)

    Skewed

    7%

    0%

    25%

    Results of the MC simulation

    For alternative 1 an MC simulation consisting of 10.000 runs was performed. For each run input variables were randomly drawn from the probability distributions of table 2. The figure on the right shows the MC simulation results for the total costs, the total benefits and the balance. The expected net present value is EUR -248 million and the standard deviation EUR 337 million. The probability of a positive balance is still 23%.

    As can be seen in the figure, the benefits vary more than the costs. The factors contributing to the benefits are added and therefore the uncertainties of these contributions are directly translated into the balance (the same holds true for the costs). To reduce the uncertainties, it is in this case advisable to do more research on the benefits. If you want to know which factor contributes most to the total uncertainty, an extra MC can be performed. This extra analysis has not been done in this practical application.

    The case: lessons learned

    Effort and time
    First conclusion is that performing a cost-benefit analysis with an MC simulation is not too complicated, but it is time consuming. The main difficulty is finding useful information for establishing the probability distributions. For different types of benefits one can rely on available databases which include standard values for calculating the physical effect and the impacts on welfare (including prices). In most cases they include the mean values only. Data providing information on type and parameters of the most suitable probability distribution are often lacking.

    Reducing uncertainties
    The MC simulation yields useful information on which items in the balance sheet contribute most to the total uncertainty in the net present value. This information can be used to guide further research so as to reduce uncertainty.

    More comprehensive information in the decision-making process?
    An MC simulation should result in more comprehensive information about uncertainties and thus add to the decision-making process. It is true that an MC simulation leads to more information (see figure 1): the figures show additional information about the bandwidth of costs, benefits and the balance between them. Based on the MC-simulation it is also possible to give a prediction of the probability of a positive balance between benefits and costs. The question is whether this ‘new’ information is useful and workable to the decision maker. Reporting that there is a 23% chance of a positive balance can be very confusing to decision makers, especially if he thinks this is a completely random process. 
    An advice to a decision maker should therefore give insight into the nature of the uncertainties in the study results presented. Decision makers, on the other hand, should be well aware of possible financial setbacks or breaks. Hence it can be concluded that MC simulation does have something to add to the decision making process.

    References

    CBS 2011 (a), website: Statline, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)

    CBS 2011 (b), Statistisch Jaarboek 2011, Centraal Bureau voor de statistiek, Den Haag. (In Dutch)

    CPB, MNP, RPB (2006), Welvaart en leefomgeving - een scenariostudie voor Nederland in 2040. (In Dutch)

    Deltares (2010). memo van J.van Gils: Expert judgement naar draagkracht van de Grevelingen voor schelpdierkweek (In Dutch)

    DHV/HKV (2010), gevoeligheidsanalyse waterberging zuidwestelijke delta, versie 2.0. Rijkswaterstaat. (In Dutch)

    Natuur- en Recreatieschap Grevelingen (2010), MIRT-verkenning Grevelingen - Notitie Reikwijdte en Detailniveau, August 2010. (In Dutch)

    Royal Haskoning, (2010). Tidal power station in de Brouwersdam. Variantenstudie. Definitief rapport, in opdracht van Groenservice Zuid-Holland. (In Dutch)

    Ruijgrok, E.C.M., A.J. Smale, R. Zijlstra, R. Abma, R.F.A. Berkers, A.A. Nemeth, N. Asselman, P.P. de Kluiver, R.S. de Groot, U. Kirchholtes, P.G. Todd, E. Buter, P.J.G.J. Hellegers, F. A. Rosenberg, (2006). Kentallen waardering Natuur, Water, Bodem en Landschap, Hulpmiddel bij MKBA, Ministerie van LNV, Den Haag. (In Dutch)

    Witteveen+Bos (2011), MKBA verkenning Grevelingen, Deventer. (In Dutch)

    Suggestions for further reading

    University of Virginia (2006) PROJECT MANAGEMENT GUIDELINE, APPENDIX D – Cost/Benefit Analysis Guide. ITRM Guideline CPM 110-01. Date: January 23, 2006.

    Mishan, E. J. Cost-Benefit Analysis. 2nd ed. New York: Praeger Publishers, 1976. Classic theoretical development of cost-benefit analysis with applications in the public and private sector.

    Riggs, James L. Engineering Economics: 2nd ed. New York: McGraw-Hill, 1982. Thorough presentation of project evaluation techniques including cost-benefit analysis in the public sector.

    Treasury Board of Canada Secretariat (2007). Canadian Cost-Benefit Analysis Guide. Regulatory Proposals. Catalogue No. BT58-5/2007. ISBN 978-0-662-05039-1. This document is available on the Treasury Board of Canada Secretariat website .

    European Union (2008) Guide to Cost Benefit Analysis of Investment Projects. Structural Funds, Cohesion Fund and Instrument for Pre-Accession. July 2008. EUROPEAN COMMISSION. Directorate General Regional Policy.

    Carel J.J. Eijgenraam, Carl C. Koopmans, Paul J.G. Tang, A.C.P. Verster, (2000) EVALUATIE VAN INFRASTRUCTUURPROJECTEN; LEIDRAAD VOOR KOSTEN-BATENANALYSE. Deel I: Hoofdrapport en Deel II Capita Selecta. Onderzoeksprogramma Economische Effecten Infra-structuur

    Back to Top

    PDF