John Libbey Eurotext

Science et changements planétaires / Sécheresse

Évaluer l’efficacité environnementale : directives et cadre d’évaluation pour un système embarqué d’aide au pilotage Volume 23, numéro 3, Juillet-Août-Septembre 2012


Auteur(s) : Maya Leroy1, Laurent Mermet2

1 AgroParisTech Équipe de recherche Montpellier Recherche en Management (EA MRM) 648, rue Jean-François Breton 34093 Montpellier cedex 5 France

2 AgroParisTech Centre de Recherche Sens, Éthique, Société (UMR CERSES) 19, avenue du Maine 75732 Paris cedex 15 France

Reprints: M. Leroy

Since the 1980s, the environmental field has seen an accumulation of commitments (policy goals, principles, international conventions) and of rules, plans, programs, and projects intending to fulfil them. In much of the field, a major issue is now to find out whether policies (i.e., rules, plans, programs, and projects) really meet existing commitments. Evaluation in general, and especially evaluating the environmental effectiveness of policies – whether or not they meet the environmental bottom-line – increasingly becomes a prerequisite for further progress in environmental management and policy (Mickwitz, 2003; Crabbé and Leroy, 2008).

The problem can be framed in two quite different ways, depending on the status conferred to dissension about the goals and means of environmental policies (Scrase and Sheate, 2002). First, one may regard politics and dissension as confined to goal-setting, and so as having been settled by institutional commitments (international treaties, EC directives, national laws). From that perspective, the implementation of environmental commitments is seen as a joint challenge shared by all actors. The explanation for, and the remedies to, gaps in the implementation is then to be looked for in difficulties such as coordination problems, poor organisational design and administration, technical obstacles, insufficient methods and indicators, etc. Second, one may, on the other hand, view environmental political commitments as just one step in an on-going struggle that is underpinning the transition to a more ecological society. From that perspective, the implementation of any environmental policy is no less political, and thus strategic and dissension ridden, than was the negotiation of the commitments the policy is meant to meet. Furthermore, the evaluation of environmental policy must then itself be seen as subject to the same dissension and struggle.

Our experience of, and research on, environmental policies and management clearly converge with Scrase and Sheate's analysis and conclude in favour of the second, more strategic, perspective on environmental commitments, policies, programs and their evaluation. This has profound consequences for the theory and practice of evaluation. It justifies critical reluctance towards methods that rely on consensus-building between stakeholders convened “around the table”, if these methods run the risk of providing a majority to stakeholders eager to “drown the fish” of the environmental effectiveness of policy. As an alternative, we have proposed a concern-focused evaluation (Mermet et al., 2010), based on a “strategic environmental management analysis framework” (Mermet et al., 2005). Concern-focused evaluation can be summarised in the following four points:

  • –. defining the focal concern;
  • –. developing criteria and synthesizing observed variations in environmental quality;
  • –. identifying policies that contribute to the concern being met or not. The “actual policy” regarding that concern consists of the whole set of public policies that impact it positively or negatively (e.g., in matters of wetland conservation in arid regions: agricultural development and irrigation programs, subsidies to several crops and intensive agricultural practices, hydroelectric development projects, wetlands conservation programs, etc.);
  • –. evaluating policies specifically aimed at meeting environmental commitments. By assessing the “intentional policy”, by understanding how it actually plays out in the wider picture of “actual policy”, the evaluator will see all the classical, non-dissensual causes of implementation gaps.

But he will also uncover causes and responsibilities that are often obscured by consensual framings and methods of evaluation. Furthermore, he will clearly situate the multiple aspects of implementation and its difficulties in the strategic, dissensual scenes and struggles of policy implementation.

This article is based on research carried out over a period of six years that has combined diagnostic analysis of the degradation of wetlands and ecosystems in the Senegal valley and policy evaluation of environmental programmes associated with hydropower development at the Manantali dam on the Senegal River in West Africa (Leroy, 2006). The purpose of the research was to develop and support detailed elaboration of the theoretical and methodological “strategic environmental management analysis framework”. We will not be going back over how this case study allows us to consolidate the four phase logic of concern-focused evaluation, as this has already been explained in a previous article (Mermet et al., 2010) but we will, however, be going further into how these evaluations are conducted, by providing the guidelines and tools that the evaluator will be using. The article will be organized in two main parts before the final discussion. The first reviews in a pragmatic way six challenges that the “on-board” evaluator has to take up, and proposes guidelines to steer his way through the confusion and turmoil of volatile environmental issues. The second accompanies the evaluator through the case study of the Senegal River reflects more systematically on steps and tools for the enterprise and shows how these solutions can be integrated into an “on-board” evaluation framework. Finally, we will discuss some of the theoretical, methodological and practical issues raised by this approach.

Six challenges and guidelines for the evaluator who wants to get at the environmental bottom-line

Evaluating environmental public policy involves analyzing managerial arrangements that transform externality problems into strategic settings. That is why the general methodology of our research on the Senegal River (Leroy, 2006) is in line with pragmatic sociological methodology to account for the shaping of these strategic settings, based on a longitudinal case study, as highlighted by other authors (Coutouzis and Latour, 1986; Barbier, 2008). Working on the process of organisation as it evolved, confronts the evaluator with a variety of vested interests and motivations, quite often contradictory or conflictual, with a multitude of rationalities, and with controversies of a strategic nature connected with the organisational process (Chanal et al., 1996). Given that the zone in question revealed conflicts, as in most situations of environmental management, particularly concerning the use of the rivers resources, we decided to understand the asymmetry of power which develops, and the controversies and conflicts that cause problems in the context of a more ecological management of the river. Drawing upon the extreme diversity of interests revealed by these controversies and conflicts is another methodological aspect that is provided by the comprehensive sociology and by the sociology of collective action (Crozier and Friedberg, 1980) that we have mobilized in this research. Our methodology, in which the triangulation of data plays a vital role, consists of a combination of:

  • –. socio-ethnological inquiries resulting from interviews with the actors and organisations implicated in Senegal River's environmental management, and participant observation;
  • –. documentary analysis: a retrospective analysis of the numerous documents produced by the actors throughout the process of managing (activity reports, strategic documents, procedural documents, etc.) as well as the documents of reference that justify their actions (scientific articles, literature and surveys relating to the problem they are trying to resolve, legal texts, etc.);
  • –. qualitative analysis of inter network communication (bulletins, website, the Internet, campaigns, etc.);
  • –. fieldwork observation when possible (for a more detailed presentation of our methodology see Leroy, 2006).

As in our previous experiences when evaluating wetland's public policy in France (CIME, 1994; Mermet, 1996), or as in the evaluation of integrated coastal management programmes (Billé, 2007), we were faced with certain actors who may obstruct the evaluation by concealing certain problems and responsibilities. It is therefore essential that the evaluator, like the researcher, is aware of criticisms arising during the evaluation process which could be used for strategic ends as well as rhetorical arguments intended to disqualify problems, to remain approximative, to avoid clarification, or to jeopardise the debate. Based on these studies we have identified six potential pitfalls confronting any evaluator who is intent on getting at the environmental bottom-line of policies, programs and projects. Since they combine different levels – tactics of the evaluation situation, methodology, position of the evaluator and framing of the evaluation – they may come out as heterogeneous. But taken together they provide pragmatic and experience-tested complementary perspectives on the challenges of “on-board” evaluation. As we present each of the six challenges, we introduce it with the presentation of a simple guideline suggesting how it may be overcome.

Refuse consensus as a sine qua none condition for implementation and evaluation

There is a strong tendency to assess policies in terms of goals that are accepted by all or most parties or stakeholders. In many situations, this hinders evaluation of environmental management because there is little consensus on the level of ambition with which an environmental goal should be pursued, or on the nature of environmental goals. A frequent compromise is to redefine the aims of the policy through stakeholder's participation and pluralistic or deliberative evaluation methodologies (Monnier, 1992; House and Howe, 1998). This often results in constantly re-negotiating criteria amongst a constituency of stakeholders, many of whom hold both a strong position of power and a reticence towards ambitious environmental goals. This may result in relegating environmental effectiveness in evaluations, often to the point where it is lost sight of and disappears from the agenda.

To overcome this problem, the evaluator should be prepared to base his environmental evaluation on environmental goals that do not always receive wide support in the field where he conducts his work. He can then base his evaluation on environmental criteria that are viewed as clear and compelling by those actors who are committed to environmental goals, even if such goals do not attract wide support at a given time, in a given field situation. In the Senegal valley research presented below (Leroy, 2006), this was done by focusing on international commitments taken by the riverside states on wetland protection, water management and biodiversity in an arid region. Although interviews showed such standards were met with reluctance by many actors in the field, they provide a clear and legitimate base for an environmental evaluation of how the ecosystem is really managed.

Start from clear environmental commitments, rather than from one program

Another tendency is to focus evaluation on one given program, rather than on the environmental problem at hand. This is rarely relevant for environmental evaluation because most environmental problems have complex causes and are subject to many policies and programs. Analysing just one of them for evaluation, it may be difficult or irrelevant to connect it with the environmental bottom-line.

So the evaluator may rather start not from one program or the other, but from the environmental bottom-line he has set as the basis for his evaluation. For example, in the 1994 French wetland policies evaluation (CIME, 1994), it rapidly became clear through a survey of all public programs affecting the condition of wetlands that the decreasing quantity and ecological quality of French wetlands is the result of policies from the Ministries of agriculture (drainage, fir plantations in bogs, dams and pumping for irrigation), of industry (hydroelectricity, gravel extraction) or of public works (dredging and filling for canals, harbours, or other infrastructure), rather than of conservation programs which were directly enforced on only approximately 7% of the total wetland area at the time of the evaluation. To evaluate with the environmental bottom-line in view, it is crucial to consider the whole set of policies with an impact. Otherwise, one could well end up with a situation where the degradation of wetlands is imputed to the ineffectiveness of wetland protection programs, when it is mainly caused, for instance, by an agricultural drainage policy. Blaming a bad environmental bottom-line on those few actors who struggle in favour of the environment seems less useful than clearly assessing the impact of other policies that are responsible for environmental degradation.

Aim at sufficient proof rather than at unachievable datasets and calculations

Environmental systems are complex and change with time. Characterising an ecosystem, measuring how its quality evolves, can rapidly turn into technical puzzles. How exact and exhaustive should the data be for an adequate assessment of the environmental bottom-line? In our work, we have repeatedly noticed situations where excessive demands on the quantity or quality of data were bogging down the evaluation. This can happen as a result of arbitrarily high standards by experts, or through actors who are hostile to environmental evaluation. This has proven instrumental in the high impact of the wetland policy evaluation study (CIME, 1994). Actors hostile to the evaluation first claimed that one could not assess variation of wetlands conditions, without a precise map of wetlands at the beginning and the end of the evaluation period. But if a wetland is known to be between 2,000 and 3,000 hectares in surface (the lack of a precise figure being due to difficulties and debates about limits on fringes of the wetlands), if there is an 800-hectare drainage program in the heart of the wetland, and if no other changes are manifest, then it is clear enough that the wetland will be reduced by about 800 hectares. So the evaluator can insist that the evaluation does not yield to demands for irrelevant levels of precision, data quality, or levels of proof, but focus on the best feasible demonstration in the real situation.

Firmly link environmental political commitments to ecological criteria

In evaluation situations, pulling together political commitments and ecological and technical data, often seems like linking together two disconnected spheres, with different languages, actors, logics and experts. The gap is often so wide that this can become an obstacle to evaluations that clarify the environmental bottom-line. For instance, political commitments often both leave a wide margin for interpretation and include such long lists of items, or “to-do lists”, that the key environmental items are lost.

Assessing the environmental bottom-line, however, does require that one clearly connect political commitments on the environment and scientific/technical criteria. Just as the evaluator had to clearly posit which environmental commitments he was going to use as the bottom-line for his evaluation, he also has to translate them into ecological and technical criteria. Our findings have shown that in most situations, an appropriate diagnosis can identify a very limited number of issues that are crucial if the ecological problem is to be solved, and that are core obstacles, i.e. the “hard issues”, which environmental policies come up against. Defining these few issues that connect knowledge on the main needs of ecosystems and the most relevant environmental political commitments requires an in-depth screening and interviews both on the political and technical side. But once this is done, one can focus the evaluation on the decisive hard issues that will make an essential difference for the environmental bottom-line.

Assess the relevance of programs, goals and outcomes to crucial ecosystem issues

Another challenge is the gap between commitments with a wide scope (for instance sound management of water at the watershed scale) and tools and programs with a much more restricted scope (for instance program subsidising residual water purification at the household level). If the gap is very wide, evaluation may tend to opt for one of three strategies, each of which miss the environmental bottom-line. If it focuses only on the scope of one program, it may lose sight of the main environmental issues. If it embarks upon the task of assessing all programs relevant for an environmental issue at once, it may end up bogged down in an unfeasible project. If it renounces the evaluation of given projects and programs and declares itself satisfied with a more general view of the situation, it tends to miss the crucial concrete issues of implementing environmental policy in the field.

What the evaluator can do here is to strike a balance by focusing on those few projects and programs that are most relevant to account for the degree to which the environmental bottom-line is met or not. Their in-depth analysis is likely to reveal essential information and understanding about the difficulties and opportunities. The Senegal valley evaluation presented below will illustrate how the evaluator can link political commitments and action programs and look for relevance – that is, both to choose which programs should be analysed, and to evaluate each program based on a clear understanding of what can make it relevant or otherwise in terms of the environmental bottom-line.

Embed management systems and evaluation tools to concrete situations

A last potential pitfall lies in the tendency to isolate administrative and technical environmental management systems from the threefold background of the political scene and environmental commitments, the concrete social and administrative contexts of program implementation, and the real environmental field conditions. A rational approach to the implementation of environmental programs, at the risk of caricature, can be summarized in the following way: once political commitments have been made, they could be translated into technical criteria and indicators; those could then be used directly both for the design of programs and for their evaluation. If the system is coherent enough from the start, there should be no need to go back either to political debates on aims, or to the intricate analysis of social and organisational implementation, nor to open new debates on the science of the case. One would just have to gather information on the indicators and, from there on, evaluation would just be a technical and administrative matter of compiling and treating data.

Whereas such approaches may seem to provide a rational way of staying focused on the bottom-line right from the start, they run a major risk of failing. Indeed, by staying on technical and apparently more objective ground, they do not escape political and stakeholders’ pressures to avoid or water down criteria that may expose insufficiencies and hard issues. Rather, the pressures are played out in technical form (methodology, data, etc.) and are difficult to discuss openly if technical (methodological) and political questions are not dealt with in parallel. Systematic and continuous systems of indicators tend to induce much higher demands on data than what is sufficient to demonstrate important changes in the environment bottom-line in an external evaluation. Also, these approaches are vulnerable to the pressure of (political) time. After an interval of several years between the inception of the system and its use for evaluation it can be easy to evade or redefine political commitments presented as obsolete.

The evaluator can choose the opposite path. Rather than trust – or distrust – management and monitoring systems, he can systematically revive their connections with political environmental commitments. To do so he may re-examine (a) the meaning of previous political commitments; (b) the real administrative and social functioning of the corresponding programs; and (c) the concrete environmental situation in the field – and then connect the three dimensions. The impact of an evaluation lies precisely in cross-examining these elements. Does it hold well in the current situation (political, administrative and social, ecological)? If not, why not? And what re-commitments are called for if we are finally to take up the environmental challenge? In that sense, evaluation is profoundly different from the administrative follow-up of action programs deriving from past political commitments. It is closer to laying the foundation for a new, or renewed, commitment, and thus, to strategic planning of the future.

Evaluating an environmental management program in the Senegal valley: an “on-board” approach and framework

Each of the six challenges we have just reviewed can be difficult to overcome, but they are also interconnected. They all have to be dealt with every time one wants clear answers on the environmental bottom-line. Inversely, if one challenge is poorly met, it tends to make the others more difficult. For instance, unclearly stated goals confuse the definition of technical criteria and the acquisition of data. So an effective evaluation has to rely on a methodology that meets these challenges in an organised way. We will now introduce, illustrate and discuss evaluative research on environmental management programs in the Senegal valley (Leroy, 2006). Of course, designs can, and will, vary depending on environmental and policy issues, and on evaluation use intentions in the context of the evaluator's strategic situation (Patton, 2008). But whatever the detailed tools and steps of the methodology chosen, the “on-board approach” has three phases:

  • –. phase 1: deconstruction, to go beyond the apparently consensual environmental management that is usually presented;
  • –. phase 2: reconstruction, when the evaluator posits criteria for his analysis and assessment;
  • –. phase 3: assessment, when he confronts the realities of the field in relation to the environmental commitments, the bottom-line.

The deconstruction phase: concrete situations in place of management discourse

Evaluation starts from a roughly stated environmental problem and a complex field situation. In the Senegal valley, the central environmental issue is the impact of two recently built dams, and of on-going projects to modify their management and potentially complement them with new hydraulic works.

The Senegal River, 1,700 km long, is shared by four main riparian states: Senegal, Mauritania, Mali, and Guinea. It flows through mostly desert regions. Its natural hydrology reflects the conditions of the upper basin, and being tropical is characterized by a period of high water (July-October) with a flood peak usually in September (3,515 m3/s characteristic flood flow) and a period of low water (4 m3/s characteristic low water flow). The absorption of the flood is very important (low slope and large overflow in the floodplain) and the flood propagation between entry into the floodplain (Bakel) and the delta 800 km downstream (Dagana) takes over a month (Rochette, 1974; Albergel et al., 1993). These averages hide a huge inter-annual variability that has been modified with the commissioning of the dams in 1987. Yearly floods create a complex of aquatic and terrestrial habitats which are essential for biodiversity (Trochain, 1940; Bourlière et al., 1976; Ba and Noba, 2001), birdlife (Morel and Morel, 1990), fisheries (Reizer, 1974), grazing and for flood recession agriculture (Boutillier and Schmitz, 1987). The Senegal valley and the estuarine zones were areas of extraordinary ecological richness which have degraded over a century of agricultural development (dykes, irrigated areas) which has accelerated over the last 20 years, following the Saharan drought, when two dams were built (Leroy, 2006). Upstream, the Manantali dam was built in the 1980s and came into operation in the 1990s, with the triple purpose of complementing low water flows for: (i) irrigation along the valley; (ii) navigation; and (iii) hydropower. Its main impact is the disappearance of the natural annual flood which is vital for wetland ecosystems along the valley, for water habitat and fisheries, for groundwater and vegetation, for flood recession agriculture and more generally traditional, ecosystem-services based activities, such as fishing, farming and range management (Boutillier et al., 1962; Lericollais and Schmitz, 1984; Schmitz, 1986). Downstream, the Diama dam blocks the dry-season intrusion of sea water along the river bed, with the purpose of preserving and increasing freshwater resources, especially for massive irrigated agriculture projects in the delta. Its main impacts are severe disturbance of the estuarine ecosystem and replacement of most associated habitats by intensive agriculture. The damage to ecosystem functioning and biodiversity is considerable, yet mitigated by the creation of protected areas that are artificially supplied with water, mostly to conserve some habitat for migratory birds. Both dams and large-scale hydraulic engineering works have been carried out under the authority of the Organisation for the Development of the Senegal River (OMVS), an intergovernmental organisation grouping Mali, Senegal and Mauritania. Created to promote the economic development of the Senegal River basin, the OMVS hosts the technical team in charge of planning and management the dams, and its governance relies on negotiation between the member states.

The main current project is to equip the Manantali dam with a 200 MW capacity hydro-electric power generation, that would provide about 800 GWh in a average hydrology year, to be dispatched to the respective capitals of Senegal, Mauritania, and Mali, through 1,400 km of transmission lines. International funding for the project, by the World Bank and several donor countries, has been made conditional on the satisfactory implementation of a comprehensive “Program for Environmental Impact Mitigation and Monitoring” (PASIE). How effective is this program, in environmental terms? And what are the prospects for the ecosystems of the Senegal valley under current development projects?

The deconstruction phase starts from reading official documents. To push beyond a management discourse emphasising consensus and environmentally responsible choices, one has to open the black box of political intrigue, of intricate management systems and programs, of highly complex ecosystem issues. In the Senegal valley, this was done through an extensive study of existing documents, both managerial and academic, and a program, in the first year of investigation, of 50 interviews with policy-makers, managers, engineers, natural and social scientists, local resource users. This work revealed a plethora of information, social and political, anthropological, managerial, hydrological and ecological. After a year of such open investigation, the importance of the accumulated information brought to light major contradictions and controversies between sources about the facts and diagnosis of the case, serious gaps in knowledge and data, contradictory or incompatible points of view from various actors on values, management issues, options and prospects. In the Senegal valley, this includes inter-alia debates on complex hydrological data about annual floods and their effects, abundant but heterogeneous and incomplete information on valley ecosystems, complex social, political, administrative and managerial processes that several management documents fail to comprehend. The evaluator now accumulates ample and relevant material, but in a thoroughly deconstructed way. He seems to be worse off than he would have been, had he taken for granted the official management discourse and built his evaluation from there.

The reconstruction phase: re-defining relevance and effectiveness

A priority is therefore to reconstruct a clear picture of the ecological criteria that will serve to assess the environmental bottom-line against which “actual” and “intentional” policies will be evaluated. This is largely a matter of determining a small set of key environmental claims, of crucial drivers and outcomes for ecosystems, and of combining them to define criteria. The key issue here is relevance: establishing a very small number of criteria with high relevance both to environmental commitments and ecological issues.

In the Senegal valley evaluation, the vital lead for this reconstruction was provided by the international conventions that helped us to build a normative framework for the assessment of programs decided by governments, intergovernmental organisations and international donors. Moreover, large international environmental agreements produce relevant bodies of work that link political commitments with managerial doctrines and technical understanding and standards. For instance literature produced by the Ramsar convention includes commitments on the conservation of wetlands, but also identifies key technical issues like wetland functions and preservation of lateral and vertical exchanges between hydrosystems and wetlands, as well as the management concept of “wise use”, that is, favouring human activities that rely on, and do not damage, the ecological functioning of wetlands. The result of this first effort of reconstruction is presented in table 1. Senegal valley environmental projects, programs or policies will be assessed as to their effective contribution in fulfilling this set of commitments.

Table 1 Environmental commitments and stakes in the Senegal River valley.

Environmental commitments Environmental stakes for the Senegal River valley
Preservation of hydro-system functions
(Ramsar Convention, Convention on Biological Diversity-CBD, Agenda 21 article 18, Global Water Partnership and World Water Council)
1- Release sufficient water from the Manantali reservoir to maintain an artificial flood (to allow for flooding of the valley).
2- Limit dyke building and other unfavourable construction or alteration (to allow for lateral and vertical exchanges).
Conservation of wetlands
(Ramsar Convention, CBD, Convention to Combat Desertification–UNCCD)
3- Safeguard and restore wetlands in terms of ecological quality as well as surface area.
Conservation of biological diversity and of threatened wild species
(CBD, Bonn Convention, Berne Convention, World Heritage Convention, Algiers Convention, Ramsar Convention, Convention on International Trade in Endangered Species-CITES, Agenda 21 article 15)
4- Limit degradation and disappearance of natural resources: maintain viable populations of species in natural surroundings or re-populate, restore habitats and control pollution.
Wise use of ecosystems and multi-usage management at community level
(Ramsar Convention, CBD, Agenda 21 articles 26)
5- Favour knowledge, innovations, and practices which preserve, maintain, and enhance natural ecological and hydraulic functioning of wetland ecosystems.

However, defining the issues fundamental to environmental bottom-line evaluation goes only half way. One still has to define the benchmarks against which the bottom-line will be measured. Here the issue is one of effectiveness: do programs make enough of a difference in meeting the relevant environmental commitments? As discussed above, defining the benchmark risks bogging down evaluation if excessive demands of precision are made, relative to limits in the available knowledge and data.

In the case of the Senegal valley, as in some previous studies (Cattan et al., 1996; Poux et al., 1996), we adopted a very simple set of relative benchmark levels. The program under assessment either leads (or contributes to):

  • –. rapid degradation;
  • –. gradual degradation;
  • –. stabilisation of ecosystem state and functions;
  • –. restoration of ecosystem state and functions.
These criteria allow us to produce a “relative evaluation” which is why we designate this evaluation as an “on-board approach”. The criteria define the variations, that is to say the “trends” and not the “state” of the ecosystem, in terms of slowing down, stabilisation, or acceleration of degradation processes. The programs, projects and plans to be evaluated can therefore be positioned within a standardised framework of which the evaluative reference points are the major environmental issues.

For each of the key targets defined in table 1, based on a triangulation of available data and understanding from the deconstruction phase, we defined qualitative and quantitative criteria to help with the benchmarking of programs’ aims and outcomes.

Regarding the crucial issue of the annual flood, for instance: a lack of artificial flooding, or a high proportion of years without such flooding (1 in 2 or 3 years) is interpreted as rapid degradation; a weak artificial flood (150,000 flooded hectares or 40,000 hectares of flood recession agriculture), or more years without flooding than in the natural hydrological pattern, is interpreted as slower degradation; a regular annual artificial flood with a fair duration and volume (230,000 hectares flooded, or 70,000 hectares of flood recession agriculture will count as a stabilisation on this issue; a regular annual flood of high amplitude (330,000 hectares, or 110,000 hectares of flood recession agriculture) will be seen as a form of ecological restoration.

Regarding the “wise use” issue, policies effectively promoting technical and economic activities that do not rely on ecosystem functions and that require infrastructure works that hinder such functions are assessed as rapid degradation; policies that plan for the gradual disappearance of activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture…) are assessed as resulting in slow degradation; policies that support and revitalize such activities are seen as restorative.

The same construction of criteria for all five key issues leads to a synthetic “on-board evaluation framework” cross-referencing relevance and effectiveness, that is, key issues and the level to which they are met by evaluated programs (table 2).

Table 2 « On-board » environmental evaluation framework for Senegal valley environmental commitments.

Rapid degradation Slow degradation Stabilisation Restoration
Annual flood Lack of artificial flooding
(flood coming only from uncontrolled tributaries, high proportion of years without flooding)
Weak artificial flood
(or more years without flood than in the natural hydrological pattern)
150,000 flooded hectares (or 40,000 hectares of flood recession agriculture)
Regular artificial flood with a fair duration and volume
230,000 hectares flooded (or 70,000 hectares of flood recession agriculture)
Regular artificial flood of high amplitude
330,000 hectares flooded (or 110,000 hectares of flood recession agriculture)
Lateral and vertical exchanges Acceleration of major dyke building and infrastructure works and dredging in the river bed with mitigation of impacts (or development at current rate without reduction of impacts) Development of dyke building and infrastructure works in the river bed with mitigation of impacts Termination of dyke and river bed works limiting lateral and vertical exchanges or equivalent compensation Termination of major dyke and river bed works and restoration of degraded areas
Wetlands Infrastructure works in the river bed without wetlands protection Infrastructure works in the river bed with wetlands protection, restoration of existing protected wetland areas No net loss (in surface or quality), termination of destructive practices or equivalent restoration Termination of destructive practices and restoration of degraded wetlands
Biodiversity Uncontrolled use with negative impact on the preservation of fauna and flora and disruption of natural ecosystem functions Maintenance of certain species in protected areas, gradual degradation of functional ecosystems Maintenance of biodiversity at current levels. No net loss Preservation of functional ecosystems and restoration of habitats, regeneration of species reproductive potential.
Wise use Promoting technical and economic activities that do not rely on ecosystem functions and that require infrastructure works that hinder such functions Gradual disappearance of activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…) Maintenance at current levels activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…) Support and revitalize activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…)

The assessment phase: appraising programs against the environmental bottom-line

The reconstruction phase thus provides a clear set of criteria against which the environmental effectiveness of management and policies can be assessed. As discussed above, such management must be conceived both at the program level – because the effective operation of such programs is the concrete basis of management – and at the level of the overall human action on the ecosystem, i.e. the sum of all impacting programs and activities which we defined as the “actual” management of the ecosystem.

To illustrate program assessment, we will focus on the Program for Environmental Impact Mitigation and Monitoring PASIE. Identified by states and donors as the main tool to attain environmental goals in managing the river Senegal, it is the means through which funding is presented as conditioned on good management of the environment, and receives major funding (in the 1999 OMVS plan, 17.5 million US dollars, about 4% of the cost of the hydro-power plant project). The assessment phase requires a detailed analysis of the design of the program, of its organisational operation and of its implementation (allocation of funds, redefinition of priorities, etc.).

The PASIE is a six-fold program:

  • –. mitigation of impacts of the power-plant project;
  • –. involuntary resettlement and compensation program;
  • –. optimisation of reservoir management;
  • –. health component;
  • –. monitoring, coordination, communication;
  • –. auxiliary actions.

On close examination, the first two concern the construction of the installations and the direct, momentary impact of the works, including resettlement and land acquisition in order to establish the transmission towers and substations, access roads and construction work areas. The fourth is about public health policy (all aspects of proposed plans to improve human health through ecosystem management and restoration have been rejected at an early stage by OMVS). The sixth is about rural electrification and development, with no relevance to the main environmental issues. The fifth has a limited element of ecosystem management, through the establishment of an Environmental Monitoring Office within OMVS (Observatoire de l’environnement), planned at a cost of 1.8 million US dollars. The only part of the project with a bearing on the management of the valley ecosystems is the third. It receives the lowest funding: approximately 15% of the programs budget. The initial terms of reference for this third part of the program revolved around two issues:

  • –. building a rainfall-runoff model to optimise the ecological use of water released for floods by coordinating it with the natural floods of non-dammed affluents of the river;
  • –. evaluating floods from the point of view of “the theoretical needs of natural habitats and of human uses relying on them”, so that release for artificial flooding will “re-establish a considerable part the traditional floodplain functions” (World Bank, 1997: 35) and will “allow flood recession agriculture, but also aquifers recharge, grazing land and forests regeneration, fish reproduction, wetlands management” (OMVS, 1996, section E-1: 3; OMVS, 1999, part C: 6).

Such terms of reference have a clear relevance to the key environmental issues of the Senegal valley. Implementation, however, led to major redefinition.

The rainfall-runoff model was dropped from the program at the initiative of the French research institute in charge of it, on technical, cost and feasibility grounds. The final recommendation for this part of the programme is a fixed-date release for artificial flooding because “it will have only minor consequences on electricity production”. Such a fixed-date release does not take advantage of synergies with natural flooding (and thus leads to smaller levels of flood, the amount and cost of water released remaining constant), and leads, in years with a long interval between natural and artificial floods, to very negative consequences for ecosystems and flood recession agriculture. The proposed release is also conditional on the technical consideration of the water level in the dam for ensuring optimal electricity production, which results in adopting a no-release policy even in years with sufficient rainfall for flooding.

The second main topic of the terms of reference is redefined during the program study phase as a need to characterise the “uses” of water. It effectively focuses mainly on water for agriculture, both for irrigation and for flood recession agriculture. The first has no need of annual floods, and no connection with ecosystem functioning – on the contrary, new irrigation perimeters are often built at the expense of wetlands, forests, or extensive grazing land. As for flood recession agriculture, they find it hard to see it as sustainably relying on ecosystem functioning. They propose to optimise the flooding of cultivated areas and to retain water in a controlled way through “simple hydraulic works”. In other words, they recommend a transition towards controlled irrigation by submersion, which would effectively disconnect flood recession agriculture from the ecological functioning of hydrosystems and wetlands. On the other aspects of “the needs of natural habitats”, only the issue of fish populations’ viability has been investigated seriously. The Canadian experts in charge of that part of the study recommend management aims for artificial flooding between 234,000 and 200,000 flooded hectares, corresponding to what we defined above as “stabilisation” or “restoration” levels. These recommendations, to our knowledge, have not been adopted by the dam management in preparation by OMVS. All other aspects (grazing areas and activities, forests, aquifer recharge, water quality, health connected with ecosystem management) were only granted means for light expertise (between 5 and 10 person-days each). They could produce no new information or analysis, and made no difference in the output of PASIE.

Our careful analysis of the PASIE studies and their cross-checking with other and previous hydrological studies clearly shows that there is opposition between the aims of maximising the profitability of electricity production and of ensuring a sufficient flood (in amplitude, length and frequency) to sustain ecosystems and traditional uses in the valley. Finally, OMVS has had to include some artificial flooding as one of the objectives of the dam's management. It has done so under the social and political pressure of poorer farming communities supported by NGOs who depend crucially on flood recession agriculture and cattle-raising in the valley. But the amplitude, length and frequency of the flood have been reduced to as low a level as possible, the priority being electricity production and irrigation.

When measured against our evaluation frame, the PASIE leads to slow degradation on some issues and rapid degradation on others (table 3). One can choose, in examining the PASIE, to focus mainly on efforts to attenuate the impacts: they slow down the already rapid degradation of the ecosystems in the valley. Is it a compromise? The PASIE also states that artificial flooding should be maintained for a transitory period of 20 years, until agriculture relies entirely, the planners expect, on irrigation.

Table 3 « On-board » environmental assessment of the Program for Environmental Impact Mitigation and Monitoring.

Rapid degradation Slow degradation Stabilisation Restoration
Annual flood Lack of artificial flooding
(flood coming only from uncontrolled tributaries, high proportion of years without flooding)
Weak artificial flood
(or more years without flood than in the natural hydrological pattern)
150,000 flooded hectares (or 40,000 hectares of flood recession agriculture)
Regular artificial flood with a fair duration and volume
230,000 hectares flooded (or 70,000 hectares of flood recession agriculture)
Regular artificial flood of high amplitude
330,000 hectares flooded (or 110,000 hectares of flood recession agriculture)
Lateral and vertical exchanges Acceleration of major dyke building and infrastructure works and dredging in the river bed with mitigation of impacts (or development at current rate without reduction of impacts) Development of dyke building and infrastructure works in the river bed with mitigation of impacts Termination of dyke and river bed works limiting lateral and vertical exchanges or equivalent compensation Termination of major dyke and river bed works and restoration of degraded areas
Wetlands Infrastructure works in the river bed without wetlands protection Infrastructure works in the river bed with wetlands protection, restoration of existing protected wetland areas No net loss (in surface or quality), termination of destructive practices or equivalent restoration Termination of destructive practices and restoration of degraded wetlands
Biodiversity Uncontrolled use with negative impact on the preservation of fauna and flora and disruption of natural ecosystem functions Maintenance of certain species in protected areas, gradual degradation of functional ecosystems Maintenance of biodiversity at current levels. No net loss Preservation of functional ecosystems and restoration of habitats, regeneration of species reproductive potential.
Wise use Promoting technical and economic activities that do not rely on ecosystem functions and that require infrastructure works that hinder such functions Gradual disappearance of activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…) Maintenance at current levels activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…) Support and revitalize activities based on a sustainable use of ecosystem functioning (fisheries, non-intensive cattle-raising, flood recession agriculture, traditional forestry…)

When carefully assessed beyond its fine façade of demanding procedures and reasonable compromises, the intentions and results of the PASIE contrast with the aims for wetlands conservation and sustained wise use, for functional hydrosystems, to which the states of the Senegal valley and their donors are committed (table 3).

Do other programs or activities close up the gap? We studied two of them in detail, in Senegal (Leroy, 2006): the Left-bank Land Management Plan (PDRG, 1994) and the Regional Environmental Action Plan (MEPN, 1997). Although they contain some positive features that may slow down the degradation of ecosystems and lead to some restoration locally (Humbert et al., 1995), they do not have the potential of reversing the trend. During the deconstruction phase, we looked systematically for actors and strategies in favour of a more environmental management of the Senegal valley ecosystems (such as the study of environmental health on intestinal schistosomiasis, or the study of the Strategic reflection group commissioned by the Senegalese Ministry of Agriculture, or the Validation of the environmental assessment required by Canadian cooperation [Leroy, 2006: p. 328-334]). The best of their strategic intervention as it is translates in the results of the PASIE, which they fought to reorient as much as was in their power to do, with the results we just assessed. The compromise on the Senegal valley ecosystems is one of gradual degradation. It is highly fragile, when considering the pressures from the irrigation, electricity and navigation. In the Senegal valley today, riparian States and their donors are not on the way to fulfilling their international environmental commitments.

Conclusion and discussion

We will not elaborate here on the reasons (historical, political, financial, bureaucratic) which effectively confer a low priority to environmental commitments, these points are developed in depth in the detailed case study (Leroy, 2006). Our main points here have been (1) to underline that environmental evaluation of policies is based on a duty of clarification, of enabling those parties and stakeholders who take environmental commitments seriously to assess what actions are effectively taken to fulfil them; (2) to show the challenges and pitfalls such clarification has to overcome and (3) to suggest a coherent, strategic approach to overcome them. This “on-board” approach raises some fundamental issues that we will now briefly discuss.

Based on the Senegal case and the preceding review of challenges to the evaluator who wants to get at the environmental bottom-line, let us take stock of our main proposals.

Getting at the environmental bottom-line when evaluating policies requires the framing of evaluative questions and the treatment of content with a focus on environmental effectivity concerns. This is in essence the “concern-focused evaluation” and its four steps for constructing the evaluation: define a focal concern, translate it into ecological and technical criteria, identify and analyse all policies with an impact on the concern (“actual policy”), then evaluate policies that are targeted on the focal concern (“intentional policy”) confronting actual and intentional policy to understand actual results and the strategic situation of actors with a serious environmental intent (Mermet et al., 2010).

The concern-focused approach, however, centers on the core of evaluation methodology: the framing of evaluative questions, and the terms of evaluation used (in our case, to serve most directly those actors who are intent on obtaining environmental effectiveness from policies). This leaves open essential aspects of the evaluator's intervention. Here, we have been concerned with situations where a significant part of the stakeholders perceive the effort to clarify the environmental bottom-line of policies as inappropriate interference. Whether conducting an evaluation commissioned by stakeholders who have not been able to create a consensus on the environmental issue, or whether conducting non-commissioned evaluative research (as in the Senegal case above) the evaluator is then “on-board”, embarked on an adventure that is at once strategic and methodological, of clarifying environmental efficiency in the face of resistance.

Let us briefly discuss the four aspects of this “on-board” situation.

Concern-focus and pluralism: it is often claimed that when focusing on clear environmental criteria and benchmarks, the evaluator may not reflect the balance of priorities of the community (local, national, international). Indeed, we do not know a priori which parties and stakeholders take environmental commitments seriously. Far from lacking pluralism, the clarification it produces is a crucial input to revive pluralistic debate on environmental commitments and actions which is lacking in so many situations of dubious consensus around environment and sustainable development. We cannot expect the adherence of all parties. Evaluation approaches aimed at obtaining a consensus-based assessment of environmental policy run the risk of being diverted from the environmental bottom-line by the very forces which generate the environmental problem in the first place. A clarified public debate should confront an environmental evaluation with other (agricultural, geopolitical, etc.) evaluations. The evaluator is “on-board” in that he cannot keep out of the fray: clarifying environmental commitments and achievements does make him an additional party in the controversy. In Senegal, the PASIE provides a good example of an environmental program in a context where pluralistic debate is insufficient, and where the evaluation creates new possibilities to engage others actors in such debate

Evaluation as a form of strategic intervention, based on strategic understanding: a major theoretical issue in evaluation focusing on the environmental bottom-line is that it cannot rely on theories and concepts that conceive of environmental management as primarily (or even, as only) an exercise in coordination and cooperation (based on the premise that we all want good environmental quality, but are not yet organised well enough to achieve it). Such theoretical backgrounds support evaluations that focus on compromise points and procedures, not on the environmental bottom-line. The latter needs theories that acknowledge the political and adversarial dimensions fundamental to most environmental issues, instead of euphemising them and strive to clarify them for the discussion and treatment of environmental problems. In our work, the strategic environmental management analysis framework provides such a conceptual foundation (Mermet et al., 2005). It is essential for the on-board approach to be able to mobilize theoretical resources sensitive to the strategic dimensions, such as the sociology of organisations (Crozier and Friedberg, 1980), environmental geopolitics (Le Prestre, 2005), the anthropology of development (Olivier de Sardan, 1995) or critical and political sociology (Boltanski and Chiapello, 1999).

Between managerial approaches and immersion in the field: at the centre of the on-board evaluator's situation rests the awareness that the implementation of policies is subject to deep contingencies such as field conditions, ecological, technical, political, cultural, social, etc. Linking together a real understanding of the field reality of environmental policies and the constructed abstraction of environmental management tools and systems raises important theoretical and methodological challenges. First, the evaluator must embark on comprehensive approaches that allow deconstruction of the situation and provide irreplaceable materials for the (re)construction of the evaluation. Then, a normative and management-oriented approach is necessary for positing criteria, benchmarks and assessments. Mobilizing them together is a challenge, not only for the demands on time and expertise in diverse disciplines (from management to anthropology, from sociology to planning and law), but also because these are two profoundly different perspectives, mind sets even. Comprehensive approaches promote immersion in the field, rather than the researcher arriving up front with a load of normative (environmental) demands (Brosius, 2006, expresses this very clearly in the example of conservation issues). Managerial approaches have a normative basis; they welcome excursions into the real confusion of the field, but only so far as the benefits for clarification are rapidly visible. The tension is apparent for instance in our evaluation of programs, which has to be conducted both from an internal point of view (understanding the real functioning of the program on its own, internal terms and dynamics) and from an external point of view (analysing it based on external expectations, independent of their having meaning and support for program members). For more details we refer the reader to Leroy (2006).

An on-board evaluation framework for assessment in the midst of shifting references and situations: evaluation of environmental effectiveness of policies almost always occurs in a context of complex natural and social dynamics that combined with strategic games the stakeholder's play, leads to ever shifting references. An essential issue here is to posit a small set of well-defined, synthetic criteria that will allow the evaluator to synthesize into a clear assessment the complex information and understanding gathered in the evaluation's field- and paperwork. This is the object of the “on-board evaluation framework” used in the Senegal valley evaluation. Two further points are to be underlined here. First, the framework is based on an assessment with regard to dynamic, relative, notions of efficiency. When discussing any serious political matter (think of unemployment, wages, taxes…), everyone readily admits that the crux of the debate is laying down clearly, about a given policy or reform, who is set out to win or lose, who is going to get closer to his goals and principles and who is not. Treating an environmental issue seriously does not consist of pondering about absolute, agreed upon reference states, but about whether a given policy works towards or against the goals expressed by those actors who want progress on a given environmental issue. This is what the apparent simplicity of the on-board evaluation framework sets out to do, as a synthetic tool in situations where environmental efficiency tends to be lost in strategic translations. Second, the choice of the criteria (the lines in the table) is of course essential. Beyond their small number and their clear definition, each of them has to make sense both in terms of political commitments and in technical and ecological terms (the case studied here clearly shows that the optimization of electrical production “weak artificial flood”, and the development of irrigated agriculture “development of dyke building and infrastructure works in the river bed” are technically incompatible with the preservation of wetlands “no net loss”, and are therefore against any political commitment to maintaining the ecological quality of the floodplain). Much of their usefulness lies in their capacity to firmly link technical and political debates. Like a good compass, or a GPS, they help to navigate and hold the line of environmental effectiveness in the midst of the accumulated and tangled maps of contemporary environmental policy problems

To sum up, we could paraphrase the title of Eileen Shapiro's (1996) book on management fads, and state that a crucial issue today is to assess in the age of integrative discourses that embrace so much and sometimes achieve so little. The concern-focused evaluation approach, the analysis of the “on-board” evaluator's situation and the “on-board evaluation framework” presented here, are resources which combine courage with method. But even once purpose and method are clear, the evaluator will still have to manage, in the real and messy world of environmental policy and evaluation, the pitfalls we reviewed in the first part of the paper. Here, we hope the six experience-based guidelines we propose may also be of use. From deconstruction to assessment and through the construction of the evaluation, they provide complementary mottos to resist being led astray by the combination of complexity and strategic opposition to clarification of the environmental bottom-line.