Evaluation Unit

Cross-section evaluation

of independent evaluations in 2008

in the thematic priority area Water

Synthesis report

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water'

Published by: Deutsche Gesellschaft für

Technische Zusammenarbeit (GTZ) GmbH Evaluation Unit

Postfach 5180 65726 Eschborn

T +49 61 96 79-1408

F +49 61 96 79-801408 E evaluierung@gtz.de

Prepared on behalf of:

Federal Ministry for Economic Cooperation and Development

Internet: www.gtz.de

Produced by: GFA Consulting Group GmbH

Jan Sass / Annika Schöpe

Eschborn, Oktober 26th 2009

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' i

Contents

Summary 8

1. Introduction 15

1.1 Background, objectives and object of the synthesis report 15

1.2 For the reader's guidance 17

1.2.1 Overview of the evaluated projects and programmes 17

1.2.2 Methodological approach 22

1.2.3 Comparability of the evaluation reports 23

2. Objectives and activity areas 25

2.1 Brief description of the evaluated projects and programmes 25

2.1.1 Interim evaluations 25

2.1.2 Final evaluations 29

2.1.3 Ex-post evaluations 31

2.2 Analysis of objectives systems 32

2.2.1 Activity areas, priority areas and indicators 33

2.2.2 Intervention and results levels 37

3. Rating According to DAC Criteria 39

3.1 Relevance 40

3.1.1 Relevance rating 40

3.1.2 Weighting of relevance 46

3.2 Effectiveness 46

3.2.1 Effectiveness rating 46

3.2.2 Weighting of effectiveness 52

3.3 Overarching development results (impact) 52

3.3.1 Assessment of overarching development results (impact) 52

3.3.2 Weighting of overarching development results (impact) 56

3.4 Efficiency 56

3.4.1 Efficiency rating 56

3.4.2 Weighting of efficiency 62

3.5 Sustainability 63

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' ii

3.5.1 Sustainability rating 63

3.5.2 Weighting of sustainability 67

3.6 Synopsis 68

3.6.1 Weighting of DAC criteria 68

3.6.2 Overall rating of evaluated projects and programmes 70

4. Rating of cross-cutting development themes 75

4.1 Poverty reduction and MDGs 75

4.1.1 Conceptual problems 77

4.1.2 Mainstreaming in development strategies 79

4.1.3 Poverty marker in the offers and its distribution 79

4.1.4 Target group differentiation and poverty analysis 81

4.1.5 Conclusions 82

4.2 Gender equality 83

4.2.1 Gender markers of the evaluated projects and programmes 83

4.2.2 Conclusions 87

4.3 Effects on the partners' capacity for action 88

4.3.1 Capacity development within the context of the evaluated projects and programmes 88

4.3.2 Conclusions 92

5. GTZ's concept of sustainable development 94

5.1 GTZ's concept of sustainable development within the context of the evaluated projects

and programmes 94

5.2 Conclusions 96

6. Sectoral assessment 97

6.1 Cooperation and task-sharing among German interface organisations 97

6.2 Embedment of development measures in the activities of the international donor

community 99

6.3 Implementation of the multi-level approach in advisory projects and programmes 101

6.4 Utilisation of GTZ's knowledge management methods 103

7. Contract and cooperation management 104

7.1 Modes of delivery 104

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' iii

7.2 Steering and cooperation with partner institutions 105

8. Summary of conclusions and lessons learned 107

8.1 Summarising conclusions 107

8.2 Lessons learned from the individual evaluations 108

8.2.1 Relevance of the development measures 108

8.2.2 Effectiveness of the development measures 109

8.2.3 Impact of the development measures 110

8.2.4 Efficiency of the development measures 110

8.2.5 Sustainability of the development measures 111

9. Recommendations 113

9.1 General recommendations 113

9.2 Recommendations for steering ongoing projects and programmes 114

9.3 Recommendations for planning new projects and programmes 115

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' iv

Annexes

Annex 1: Terms of reference for the synthesis report on water

Annex 2: (Standard) terms of reference for individual evaluations

Annex 3: GTZ Guidelines on Evaluating the Success of Projects and Programmes

Annex 4: Rating of individual criteria and overall rating of 15 evaluated projects

List of illustrations and tables

Figure 1: Distribution of activity areas in the evaluated projects/programmes 33

Figure 2: Proportions of urban and rural priorities in the

evaluated projects/programmes 34

Figure 3: Trends in the thematic orientation of objectives systems 35

Figure 4: Direct results levels of the evaluated projects/programmes 38

Figure 5: Distribution of relevance ratings 41

Figure 6: Distribution of effectiveness ratings 48

Figure 7: Distribution of overarching development results rating 54

Figure 8: Distribution of efficiency ratings 58

Figure 9: Distribution of sustainability ratings 64

Figure 10: Distribution of overall ratings 72

Figure 11: Average rating according to type of evaluation 74

Table 1: Overview of evaluated projects/programmes 18

Table 2: Assessment scale for the DAC criteria relevance, effectiveness,

impact and efficiency, and for the overall rating 39

Table 3: Assessment scale for the DAC criterion sustainability 40

Table 4: Relevance ratings of evaluated projects/programmes 41

Table 5: Effectiveness ratings of evaluated projects/programmes 47

Table 6: Effectiveness correlations 52

Table 7: Rating of overarching development results of

the evaluated projects/programmes 54

Table 8: Efficiency ratings of the evaluated projects/programmes 58

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' v

Table 9: Comparison of term, level of funding and efficiency ratings 59

Table 10: Sustainability ratings of the evaluated projects/programmes 64

Table 11: Sustainability correlations 67

Table 12: Average weighting of DAC criteria 69

Table 13: Weighting of DAC criteria according to time (= type) of evaluation 69

Table 14: Overall ratings of evaluated projects/programmes 71

Table 15: Distribution of overall ratings according to success level 72

Table 16: Comparison of average weightings and ratings of DAC criteria 73

Table 17: Average overall ratings and DAC criterion ratings according to region 74

Table 18: Overview of contributions by evaluated projects/programmes to the

achievement of the MDGs 76

Table 19: Poverty markers of the evaluated projects/programmes 80

Table 20: Gender markers of the evaluated projects/programmes 85

Table 21: Levels of capacity development in the evaluated projects/programmes 90

Table 22: Consideration of the characteristics of sustainable development

in the evaluated projects/programmes 95

Table 23: Cooperation between the evaluated projects/programmes

and German DC interface organisations 98

Table 24: Contractors and subcontractors involved in the evaluated

projects/programmes 104

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' vi

List of abbreviations

AURA Development-policy Framework for Contracts and Cooperation

BGR Federal Institute for Geosciences and Natural Resources

BMZ German Federal Ministry for Economic Cooperation and

Development

CIM Centre for International Migration and Development

DC development cooperation

DED German Development Service

EcoSan ecological sanitation

EVORAP Eastern and Volta Region Assistance Programme, Ghana

FC financial cooperation

GICA Gestión Integrada de Cuencas y Acuíferos, Mexico

(Management of Water Catchment Areas for the Río Lerma in the Toluca

Valley and for the Río Balsas, Mexico)

GIS geographical information system

GTZ Deutsche Gesellschaft für Technische Zusammenarbeit GmbH

GTZ-IS GTZ International Services

IGWDP Indo-German Watershed Development Programme, India

IM irrigation management

InWEnt Capacity Building International, Germany

IWRM integrated water resources management

IWSTR Improvement of Water Supply in the Tigray Region, Ethiopia

KfW Kreditanstalt für Wiederaufbau / KfW Entwicklungsbank

LDC least developed country

MDGs millennium development goals

NGO non-governmental organisation

NWI Nile Water Initiative -

Planning and Management of Water Resources in the Nile Basin,

Africa supraregional

OECD-DAC Organisation for Economic Cooperation and Development

- Development Assistance Committee

PDEA Programme d‟Eau en Algérie

(Integrated Water Management Programme, Algeria)

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' vii

PHV/MI Programme Hydraulique Villageoise Maradi, Niger

(Rural Water Supply in Maradi, Niger)

PPR project progress review

PROAGUA Programa de Agua y Alcantarillado, Peru

(Drinking Water and Sanitary Programme, Peru)

SPR sector policy and regulation

SWAp Sector Wide Approach

TC technical cooperation

ToR terms of reference

UMS Upgrading of Municipal Services, Turkey

WDPC Wastewater and Solid Waste Management in Provincial Centres, Viet

Nam

WM wastewater management

WM/S wastewater management/sanitation services

WMIA Water Management in Irrigated Agriculture, Jordan

WRM water resources management

WS (drinking) water supply

WS&S water supply and sanitation

WSP Water Sector Programme, Yemen

WSRP (KEN) Water Sector Reform Programme in Kenya

WSRP (ZMB) Water Sector Reform Programme in Zambia

WS/WM water supply/wastewater management

WSWDP Water Supply and Wastewater Disposal Project (Kosovo)

WU water utility

ZOPP objectives-oriented project planning

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 8

SUMMARY

In the overall review of all evaluations per-

formed in the thematic priority area 'water', it

proves decisive for the success of these

measures that the respective project or pro-

gramme be in harmony both with the sector

strategies of the German Federal Ministry for

Economic Cooperation and Development

(BMZ) and with the respective national policies

and poverty reduction strategies, and that the

projects/programmes follow the key guidelines

on regional cooperation and integration, capac-

ity development and integrated water re-

sources management (IWRM). It is of particu-

lar importance to systematically pursue the

concept of sustainable development from the

planning stage onward, at the same time har-

monising the objectives dimensions within the

project/programme by way of a multi-level ap-

proach. In addition to a systematic approach to

capacity development, this also calls for closer

attention to the duration of advisory services

with an eye to efficient follow-up assistance.

The present cross-section evaluation of fifteen

projects and programmes which were evalu-

ated in the thematic priority area „water‟ in

2008 contains an analysis of the evaluation

reports on:

eight interim evaluations (Africa suprare-

gional, Algeria, Yemen, Jordan, Kenya,

Peru, Zambia and Viet Nam);

three final evaluations (Ghana, Kosovo

and Mexico);

four ex-post evaluations (Ethiopia, India,

Niger and Turkey).

The selection of evaluated projects and pro-

grammes constitutes a non-representative

random sample taken from the overall portfolio

of the Deutsche Gesellschaft für Technische

Zusammenarbeit (GTZ) GmbH.

The submitted synthesis report offers some

important points of reference apart from a

summary of performance assessments, espe-

cially with regard to the methodological orienta-

tion of the evaluated projects/programmes, and

consideration of the principles and concepts of

German development cooperation. Poverty

reduction, adherence to the principles of sus-

tainable development and systematic devel-

opment of capacity for action within the partner

structure are all major cross-cutting themes for

which this evaluation can provide insights and

findings.

The sector-specific priority areas of the fifteen

evaluated projects and programmes cover a

broad, diverse range, from sector reform and

policy advisory services to municipal water

supply and wastewater disposal, rural water

supply and water resources management, up

to and including irrigated agriculture. Eight of

the evaluated projects/programmes fall into the

category of programme building, and six of

them are cooperative projects/ programmes as

defined by German development cooperation.

Most of the correspondingly defined objectives

include two or more partial objectives and/or

sub-sectors of the water sector (interventions,

e.g. in water supply and wastewater manage-

ment, water resources management, consid-

eration of urban and rural sub-sectors, combi-

nation of policy advisory services and process

optimisation). The underlying objectives sys-

tems are therefore very complex and charac-

terised by a large number of indicators and, to

a certain extent, long results chains.

The main focus of the underlying definitions of

objectives lies on the quantitative improvement

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 9

of municipal drinking water supply. Only a mi-

nor share of the projects/programmes focus on

the rural sub-sector and the areas of wastewa-

ter management and sanitation services.

On the basis of their design, all evaluated pro-

jects/programmes generally contribute to the

achievement of MDG 7 ('Ensure environmental

sustainability'). In addition, the evaluated pro-

jects contribute to the achievement of the fol-

lowing MDGs:

10 projects/programmes (66%) contribute

to the achievement of MDG 1 ('Eradicate

extreme poverty and hunger'), for exam-

ple by creating jobs and potentials for

economic activities and promoting irri-

gated agriculture;

11 projects/programmes (73%) contribute

to the achievement of MDG 4 ('Reduce

child mortality'), for example by reducing

the incidence of water-induced diseases,

in particular diarrhoea, and by improving

hygiene in overall infant care;

8 projects/programmes (53%) contribute

to the achievement of MDG 5 ('Improve

maternal health'), for example by reducing

the incidence of water-induced diseases,

improving natal hygiene, increasing the

availability of hygienically safe drinking

water and providing for adequate sanita-

tion services;

5 projects/programmes also contribute to

the achievement of MDG 2 ('Achieve uni-

versal primary education'), 5 projects con-

tribute to MDG 3 ('Promote gender equal-

ity and empower women'), 1 project con-

tributes to MDG 6 ('Combat HIV/AIDS,

malaria and other diseases') and 1 project

contributes to MDG 8 ('Develop a global

partnership for development').

The projects/programmes evaluated within the

scope of the independent evaluations were

rated on the basis of a six-point scale for the

DAC criteria1 'relevance', 'effectiveness', 'im-

pact' and 'efficiency', plus an overall rating:

'very good' (1): very good results, signifi-

cantly better than expected

'good' (2): good results, fully in line with

expectations, with no significant defects

'satisfactory' (3): satisfactory results; fal-

ling short of expectations but with positive

results dominant

'unsatisfactory' (4): unsatisfactory results;

significantly below expectations, and

negative results dominate despite identifi-

able positive results

'inadequate' (5): clearly inadequate re-

sults; despite several positive partial re-

sults, the negative results clearly domi-

nate

'useless' (6): the project/programme is

useless, or the situation has, if anything,

deteriorated.

The 'sustainability' criterion was rated accord-

ing to the following four-point scale:

'very good' (1): With a high degree of

probability, the overall success of the pro-

ject/programme (positive to date) will con-

tinue unchanged or even increase.

'good' (2): With a high degree of probabil-

ity, the overall success of the pro-

ject/programme (positive to date) will de-

crease only minimally and remain signifi-

1 OECD-DAC:

Organisation for Economic Cooperation and Develop-

ment – Development Assistance Committee

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 10

cantly positive overall (normal situation –

to be expected)

'satisfactory' (3): With a high degree of

probability, the overall success of the pro-

ject/programme (positive to date) will de-

crease significantly but remain positive.

'inadequate' (4): The overall success of

the project/programme was inadequate at

the time of evaluation, and there is a high

degree of probability that it will not im-

prove.

In the above sense, ratings 1 - 3 are indicative

of a successful assessment.

Overall, 80 % of the evaluated projects/ pro-

grammes were rated as 'successful' in terms of

implementation. Only three development

measures (Ethiopia, Niger and Turkey) were

regarded as 'unsuccessful' in the sense of the

underlying performance assessment. In those

projects, the calculated average ratings were in

the lower satisfactory range - due to either

inadequate sustainability (Ethiopia and Niger)

or, in the case of Turkey, unsatisfactory contri-

butions to the achievement of overarching

results.

The average overall rating of all 15 develop-

ment measures covered by the cross-section

evaluation, at 2.7, lies within the 'satisfactory'

range.

Success level Rating Number Percentage

'Successful'

'very good' (level 1) 0 0%

'good' (level 2) 8 53%

'satisfactory' (level 3) 4 27%

'Unsuccessful'

'unsatisfactory' (level 4) 3 20%

'inadequate' (level 5) 0 0%

'useless' (level 6) 0 0%

In the assessment of projects/programmes

according to DAC criteria, „relevance‟ received

the best ratings with an average of 2.0, includ-

ing 11 individual ratings in the 'very good' (5

projects/programmes) and 'good' (6 pro-

jects/programmes) range. The 'effectiveness'

ratings of the evaluated development meas-

ures reached an average of 2.5. Again, most of

the individual success ratings were 'good' (9

projects/programmes). By contrast, the aver-

age ratings for the criteria 'impact' (2.7), 'effi-

ciency' (2.8) and 'sustainability' (2.9) were

more plainly situated at and around success

level 3, where most of the individual ratings

were 'satisfactory' and 'unsatisfactory'.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 11

Assessed DAC criterion Average rating

Relevance – 'Are we doing the right thing?' 2.0

Effectiveness – 'Are we achieving the objectives of the devel-

opment measure?' 2.5

Impact – 'Are we contributing to the achievement of overarching

development results?' 2.7

Efficiency – 'Are the objectives being achieved cost-effectively?' 2.8

Sustainability – 'Are the positive results durable?' 2.9

In the analysis of the objectives systems of the

evaluated projects/programmes, it becomes

clear that, above all, only a small number of

development measures include 'poverty rele-

vance' in their formulation of objectives and/or

include a 'poverty dimension' via appropriate

indicators: only two programmes (Kenya and

Zambia) have a recognisably formulated pov-

erty relevance in their definition of objectives.

Four programmes (Yemen, Kenya, Peru and

Zambia) have indicators for dealing with the

poverty dimension. This stands in contradiction

to the underlying service packages offered to

the BMZ, in which twelve projects/programmes

include poverty relevance, albeit with different

weightings, by way of the corresponding DAC

marker. Only a single programme (Africa su-

praregional) was classed EPA (general devel-

opment approach), while two pro-

grammes/projects (Kosovo, Turkey) neither

have a poverty marker in their respective offer,

nor a relevant recommendation included in

their evaluation report.

A further inconsistency arises from the fact that

ten of the evaluated projects/programmes

carry the DAC marker G1 (gender equality as a

secondary objective), but only three of those

ten development measures (Yemen, Kenya

and Zambia) have a corresponding indicator

for the 'gender dimension'.

With regard to their ecological impacts, all

projects/programmes were rated as UR1 or

better (India, Jordan and Mexico: UR2), while

only four (4) have indicators related to the 'en-

vironmental dimension' (India, Yemen, Jordan

and Kenya).

All in all, the assessment of all 15 individual

evaluations showed that the ratings of the de-

velopment measures on the basis of DAC

markers had, for the most part, not been per-

formed as specified and that corresponding

objectives and indicators had often not been

formulated with regard to these markers. On

most of the projects/ programmes reliable pov-

erty, gender or environmental impact studies

are not available. Instead, the respective re-

sults are presumed, i.e. aggregated up via long

results chains.

The various evaluations attached different

levels of importance to the DAC criteria (as

measured against the upgrading or downgrad-

ing of the standard weighting 2.0). Particular

importance was attached to the criteria 'rele-

vance' and 'sustainability' (average weighting

2.3) and to 'effectiveness' (2.2).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 12

DAC criterion Average weight-

ing

Number of upgrades

(weighting factor 3)

Number of down-

grades (weighting

factor 1)

Relevance 2.3 5 0

Effectiveness 2.2 3 0

Impact 2.1 2 1

Efficiency 2.0 2 1

Sustainability 2.3 6 1

In line with the impressions gained by the

evaluators who performed the individual

evaluations, the important success factors for

the 'relevance' of development measures in-

clude allowance for regional cooperation (par-

ticularly in the field of integrated water re-

sources management, IWRM), conformance of

the respective development measures with the

BMZ sector strategy and/or with the Millennium

Development Goals (MDGs), and systematic

attention both to a 'multi-level approach' and to

the principles of capacity development in the

design of the measures.

Regarding the relevance rating of the pro-

jects/programmes, the fact that only a small

portion of the measures were directly intended

to improve the living conditions of poorer popu-

lation groups, and that most of the pro-

jects/programmes had performed no poverty,

gender or environmental impact analyses,

clearly had a negative effect on the rating.

The evaluated projects/programmes mainly

achieve their direct results at the technical

improvement level and in terms of supply stan-

dards, but less so at the institutional level and

in connection with statutory change processes

and the creation of sustainable capacities.

There is little correlation between the em-

ployed volume of funding and the achievement

of objectives. The duration of advisory services

appears to be a major success factor for effec-

tiveness. The qualitative aspects of achieved

objectives (e.g. altered behaviour, satisfaction,

etc.) are still not collected consistently. As a

consequence, numerous results are presumed

without appropriate evidence.

In only three of the evaluated pro-

jects/programmes (Ghana, Peru and Zambia)

were any concrete links established between

the outcomes and the intended, or even unin-

tended, indirect results (impact). In those

cases institutionalised monitoring systems

were used to acquire the corresponding data at

an early stage and at regular intervals. In most

of the evaluations and projects/ programmes,

the overarching results were assessed on the

basis of assumptions and abstractions via long

results chains, without the corresponding in-

puts having been registered or investigated.

One success factor of prime importance for the

efficiency of development measures is the

embedment of TC interventions in sector inter-

ventions per se. Here, all evaluations identified

a definite need for improvement. Often, unsat-

isfactory efficiency is mainly attributable to

inadequate coordination and/or consensus-

building between DC organisations regarding

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 13

the use of resources, in addition to steering

and monitoring problems.

Sustainability is considered to be an extremely

important criterion, also in the course of im-

plementation, and correspondingly influences

the overall rating. A correlation was estab-

lished between the duration of the measure

and, hence, the duration of advisory services

and the sustainability rating of the respective

measure. The main success factors for sus-

tainability are the design of systematic capacity

development approaches and the considera-

tion of follow-up assistance phases - the latter

primarily in the case of cooperation between

TC interventions and financial cooperation

(FC) investments. As a rule, the basic chal-

lenge here is to flank and track the relatively

short implementation phases required for the

construction work with adequate measures to

create and/or strengthen appropriate opera-

tional and administrative structures.

The methodological approach of the evaluated

projects/programmes was given a positive

overall rating. Obviously, it was not possible in

some of the projects/programmes to implement

a systematic multi-level approach against the

backdrop of very complex partner structures.

The results constellation of the evaluated pro-

jects and programmes is focused at the meso

level in the area of personal capacity building

and organisational advisory services. System-

building and networking results are achieved

mainly by the more complex sector pro-

grammes (Yemen, Kenya, Zambia), in which

the multi-level approach was mainstreamed

with well-balanced attention to the economic,

social and ecological objectives dimensions.

Capacity development approaches formed part

of all projects/programmes. In many cases,

however, the underlying strategies were still

too unsystematic and non-transparent, particu-

larly for the partner structures. The time peri-

ods allowed for establishing capacities are

sometimes too short to achieve sustainable,

structurally relevant results. Due not least to

pronounced concentration on the meso level,

TC work sets high standards for steering by

the partner institutions. The recommendations

in the individual evaluations concentrate mainly

on the need to systematically broaden or focus

the corresponding interventions in order to

foster system building and to initiate or support

change processes.

Some of the projects' and programmes' impor-

tant innovations are already being dissemi-

nated via GTZ‟s knowledge management.

Nevertheless, the use of best practices in

German TC is still poorly developed.

The mode of service delivery was assessed as

appropriate in most of the evaluations. All in

all, both the German interface organisations

and the German consulting sector are very

broadly involved in delivering services on the

basis of an increasingly shared formulation of

objectives. The conceptual design of the hu-

man resource inputs underlying the implemen-

tation is still viewed as suboptimal sometimes.

In many places the integration of all service

providers into the overall programme, the flow

of information, the steering and, to a certain

extent, even the qualifications of the assigned

personnel continue to cause difficulties.

Cooperation agreements between the DC ac-

tors improve the joint planning and steering of

interventions. By contrast, the recording of

results via adequate monitoring systems as a

basis for further-improved steering still consti-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 14

tutes a weak point of most pro-

jects/programmes.

Important recommendations from the individual

evaluations refer primarily to the conceptual

orientation of the projects/programmes:

Include poverty and target-group analyses

in the planning of projects/programmes;

Closer orientation of the projects/ pro-

grammes to pro-poor results and change

processes;

More systematic attention to capacity de-

velopment at all levels of intervention;

Closer attention to networking and system

development in the multi-level approach

to ensure sustainability;

Improved recording of results via monitor-

ing systems and the use of project pro-

gress reviews and evaluations for a more

systematic appraisal of results;

More intensive coordination and steering

of German DC contributions in the water

sector in the partner countries on the ba-

sis of cooperation agreements.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 15

1. Introduction

1.1 Background, objectives and

object of the synthesis report

GTZ extended and optimised its evaluation

system in 2005. The adjusted evaluation sys-

tem is now consistently oriented towards the

evaluation criteria and principles of the Devel-

opment Assistance Committee of the Organi-

sation for Economic Cooperation and Devel-

opment (OECD-DAC). It is based both on self-

evaluations and independent evaluations2.

Within the scope of independent evaluations,

GTZ itself now commissions independent third

parties to perform the evaluations. Implemen-

tation contracts are concluded with independ-

ent research institutions, consulting firms and

external evaluators. The commissioning party

is the GTZ Evaluation Unit, which is organised

independently of the operative departments

and answers directly to the Office of the Man-

aging Directors. The unit bears overall respon-

sibility for planning and steering GTZ‟s evalua-

tion programme.

Independent evaluations serve especially to

meet accountability obligations regarding the

use of allocated resources, to improve the

management of projects and programmes and

to promote learning, both within the pro-

ject/programme and throughout the company.

Observations presented in this synthesis report

on the thematic priority area 'water' are based

exclusively on independent evaluation reports

concerning the relevant projects and pro-

grammes.

2 Until 2007, independent evaluations were still commonly

referred to as 'external evaluations'.

The planning and implementation of evalua-

tions is subject to the evaluation principles of

the Development Assistance Committee of the

Organisation for Economic Cooperation and

Development (OECD-DAC3):

Usefulness: The choice of project and the

terms of reference (ToR) reflect the inter-

est in obtaining specific findings both on

the part of those responsible for a project

and of the decision-makers. The evalua-

tion findings are communicated to all

stakeholders and incorporated into knowl-

edge management.

Participation: The partner institutions in

the partner country are involved in every-

thing from the formulation of the terms of

reference to the conclusion of the evalua-

tion.

Credibility: The evaluation system and the

evaluation programme are transparent,

and the evaluation findings, both positive

and negative, are published. The Evalua-

tion Unit, the commissioned research in-

stitutes and/or consulting firms, and the

evaluators performing the assessment are

all independent.

On the basis of methodically verified data,

evaluations are intended to provide substanti-

ated information about key criteria of the pro-

ject/programme assessment. The underlying

evaluation criteria are in line with the OECD-

DAC recommendations and are agreed as

binding between the BMZ and the German

implementing organisations:

3 'Organisation for Economic Cooperation and Develop-

ment - Development Assistance Committee'.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 16

the relevance of the implemented devel-

opment measure – 'Are we doing the right

thing?'

the results achieved by the pro-

ject/programme (effectiveness) – 'Are we

achieving the objectives of the develop-

ment measure?'

the respective contribution to overarching

development results (impact) – 'Are we

contributing to the achievement of over-

arching development results?'

the efficiency of the employed resources –

'Are the objectives being achieved cost-

effectively?'

the sustainability – 'Are the positive re-

sults durable?'

In addition, the evaluations are intended to

provide information on specific corporate and

development-policy issues. This refers in par-

ticular to how the development measures con-

tribute toward poverty reduction, the achieve-

ment of MDGs, gender equality and sustain-

able development. Moreover, appropriate

technical/sectoral issues are also evaluated.

Each year, the GTZ commissions approxi-

mately 30 independent evaluations (interim,

final and ex-post evaluations) with thematic

priority areas (2008: 'decentralisation' and 'wa-

ter'). Based on the reports concerning individ-

ual evaluations, the Evaluation Unit orders the

implementation of a 'cross-section evaluation4'

for each thematic priority area, which then

serves as the point of departure for a corre-

sponding 'synthesis report'. The latter is in-

4 Until 2008, cross-section evaluations were referred to as

'meta-evaluations'.

tended to facilitate institutional learning within

the GTZ and reporting to the BMZ. The results

and findings are incorporated into the GTZ's

knowledge management and are made avail-

able for public relations work.

The object of this synthesis report is the cross-

section evaluation of fifteen global evaluations

concerning the thematic priority area 'water'

from the year 20085. This includes:

eight interim evaluations (Africa suprare-

gional, Algeria, Yemen, Jordan, Kenya,

Peru, Zambia and Viet Nam);

three final evaluations (Ghana, Kosovo

and Mexico);

four ex-post evaluations (Ethiopia, India,

Niger and Turkey).

The cross-section evaluation encompasses the

following aspects:

overview and summary of the results of the

individual evaluations based on a quantita-

tively and qualitatively comparative as-

sessment;

identification of recurring strengths and

weaknesses and/or factors of success and

failure;

identification of overarching lessons

learned and recommendations for imple-

menting comparable future

projects/programmes and for the strategic

5 Altogether, sixteen (16) independent evaluations were

performed in 2008 for the thematic priority area 'water'.

The report on the jointly implemented ex-post evalua-tion of the cooperative project 'Rural Water Supply

East-Kilimanjaro, Tanzania' had not yet been coordi-

nated between the GTZ and the KfW at the time of the cross-section evaluation and was therefore not in-

cluded in the assessment.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 17

orientation of the GTZ's sector portfolio

'water'6.

1.2 For the reader's guidance

1.2.1 Overview of the evaluated pro-

jects and programmes

Table 1 on the following page surveys the

evaluations and corresponding projects and

programmes covered by the cross-section

evaluation. For the sake of straightforward,

systematic representation of the results, the

projects/programmes listed in this synthesis

report are arranged according to the type of

evaluation and/or alphabetically according to

the country, independently of any specific,

technical/ sectoral orientation or other criteria.

While all projects/programmes have to do with

the thematic priority area 'water', the specific

objectives and range of activities of the various

development measures differ substantially in

terms of content. The projects/programmes

can be basically differentiated with regard to

set priorities as belonging to the sectors or

sub-sectors (drinking) water supply, wastewa-

ter management/sanitation services, water

resources management and sector pol-

icy/regulation. Six (6) of the pro-

jects/programmes can be attributed to the pri-

ority area 'water supply', two (2) pro-

jects/programmes to the area 'water resources

management', and one (1) project/programme

to the area "wastewater management'. All

other projects/programmes deal with more

6 Selected results of the cross-section evaluation repre-

sented in connection with the GTZ Staff Conference in

Hohenroda on 6 July 2009.

than one priority area, i.e. have combined ob-

jectives definitions:

water resources management and sector

policy/regulation (1 project/ programme);

water resources management and irriga-

tion (1 project/programme);

water resources management and water

supply (1 project/ programme);

water supply and wastewater manage-

ment/sanitation services (1

project/programme);

water supply, wastewater manage-

ment/sanitation services and water re-

sources management (2

projects/programmes).

In the light of their objectives and respective

target groups, four of the evaluated pro-

jects/programmes can be attributed to the rural

sub-sector, while seven of the pro-

jects/programmes are situated in the urban

sub-sector. The objectives of four projects/

programmes relate both to the rural sub-sector

and to the urban sub-sector (Africa suprare-

gional, Algeria, Yemen and Ethiopia).

The regional distribution of the evaluated pro-

jects/programmes places eight of the pro-

jects/programmes in Africa, four of which relate

to the area 'Sub-Saharan Africa'. Two pro-

jects/programmes each from the regions Latin

America, Middle East and Southeast

Europe/Turkey were evaluated. Finally, one

project/programme from the region 'Asia' was

included in the cross-section evaluation.

Table 1: Overview of evaluated projects/programmes

Development measure Country Term Funding

volume7

Priority areas

Interim evaluations

Nile Water Initiative (NWI)

Planning and Management of Water Resources in the

Nile Basin

Africa

(AFR suprare-

gional

01/2002 - 2/2011 120 months EUR 10.0 M WRM

(urban & rural)

Integrated Water Management

Programme (PDEA)

Algeria

(DZA) 10/2003 - 2/2011 99 months EUR 9.7 M

WRM & SPR

(urban & rural)

Water Sector Programme (WSP) Institutional Develop-

ment of the Water Sector-

Yemen

(YEM) 07/2006 - 6/2015 114 months EUR 13.5 M

WS/WM & WRM

(urban & rural)

Water Management in Irrigated

Agriculture (WMIA)

Jordan

(JOR) 10/2006 - 9/2015 111 months EUR 2.5 M

WRM & IM

(rural)

Water Sector Reform Programme, Kenya (WSRP-KEN) Kenya 10/2003 - 2/2013 123 months EUR 13.9 M WS/WM & SPR

7 Includes either the total cost of TC (German contribution) in final and ex-post evaluations, or the planned total TC funding volume (German contribution) up to the end of the respective ongoing phase

of implementation in interim evaluations. Both in representation and analysis, the employment of funds is only accounted for in the case of 'Rural Water Supply in Maradi, Niger', since the evaluation was performed jointly on the basis of the harmonised system of objectives, and the FC inputs were implemented by the GTZ.

Development measure Country Term Funding

volume7

Priority areas

(KEN) (urban)

Drinking Water and Sanitary Programme (PROAGUA) Peru

(PER) 01/2002 - 2/2014 156 months EUR 8.2 M

WS

(urban)

Water Sector Reform Programme, Zambia (WSRP-ZMB) Zambia

(ZMB) 01/2004 - 2/2012 108 months EUR 8.1 M

WS/WM

(urban)

Wastewater and Solid Waste Management in Provincial

Centres (WDPC)

Viet Nam

(VNM) 02/2005 - 1/2013 107 months EUR 8.8 M

WM

(urban)

Final evaluations

Improvement of Water Supply - Eastern and Volta Re-

gion Assistance Programme (EVORAP)

Ghana

(GHA) 07/1998 - 6/2008 126 months EUR 6.2 M

WS

(urban)

Infrastructure Programme -

Water Supply and Wastewater

Disposal (WSWDP)

Kosovo

(KOS) 07/2000 - 6/2009 108 months EUR 7.7 M

WS

(urban)

Management of Water Catchment

Areas for the Río Lerma (GICA)

Mexico

(MEX) 04/2002 - 1/2008 70 months EUR 2.4 M

WRM & WS

(rural)

Ex-post evaluations

Improvement of Water Supply, Tigray Region (IWSTR) Ethiopia 11/1996 -09/2004 95 months EUR 2.9 M WS

Development measure Country Term Funding

volume7

Priority areas

(ETH) (urban & rural)

Indo-German Watershed Development Programme

(IGWDP)

India

(IND) 04/1989 - 6/2005 183 months EUR 12.8 M

WRM

(rural)

Rural Water Supply in Maradi (PHV/MI) Niger

(NER) 02/1995 -12/2003 107 months EUR 20.1 M

WS

(rural)

Upgrading of Municipal Services (QADI) Turkey

(TUR) 11/2002 - 2/2006 50 months EUR 1.7 M

WS

(urban)

WM: wastewater management/sanitation services WS: (drinking) water supply

SPR: sector policy and regulation WRM: water resources management

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 21

Altogether, eight of the fifteen reviewed meas-

ures fall into the category of programme build-

ing and are based on various parallel and/or

previous measures. Six of the pro-

jects/programmes (Yemen, Ghana, Kenya,

Niger, Peru and Vietnam) are 'cooperative

projects' between the GTZ and the KfW in

which the achievement of objectives is based

on close, synergetic intermeshing of the TC

and FC instruments. In the case of the 'Water

Sector Reform Programme, Kenya', the Ger-

man Development Service (DED) is formally

involved in the cooperation agreement. In 'In-

stitutional Development in the Water Sector,

Yemen', not only the DED but also the Federal

Institute for Geosciences and Natural Re-

sources (BGR), Inwent - Capacity Building

International, Germany and the Centre for

International Migration and Development (CIM)

are involved in the implementation at various

working levels by way of cooperation agree-

ments.

Only three (3) of the evaluated projects/ pro-

grammes (Africa supraregional, Algeria and

Turkey) were, or are still being, implemented

without contracts being awarded to the Ger-

man consulting sector. In all other pro-

jects/programmes, consulting firms assume

responsibility for implementing sub-sectors or

components. Two of the evaluated pro-

grammes (India and Mexico) were imple-

mented completely by a consulting firm work-

ing on behalf of the GTZ. In the 'Rural Water

Supply in Maradi, Niger' project, the FC advi-

sory services were provided by GTZ Interna-

tional Services on the basis of a negotiated

contract.

The overall volume of German TC contribu-

tions for these fifteen development measures

evaluated in the course of 2008 amounts to

EUR 128.5 million. Hence, the average amount

of funds employed per project/programme

comes to approx. EUR 8.6 million. The lowest

volume of all (EUR 1.7 million) - in combination

with the shortest term (50 months) - was em-

ployed for the 'Upgrading of Municipal Ser-

vices, Turkey' project. The largest volume

(EUR 20.1 million) was used for the 'Rural

Water Supply in Maradi, Niger' project. The

longest term, however, was that of the „Inte-

grated Indo-German Watershed Development

Programme, India' (183 months).

Three of the projects/programmes were termi-

nated early, i.e. ahead of the terms originally

planned:

'Rural Water Supply in Maradi, Niger': The

third TC promotion phase (follow-up

phase) January 2004 - December 2004

was not carried out due to funding prob-

lems on the part of the GTZ

'Management of Water Catchment Areas

for the Río Lerma, Mexico': The third pro-

motion phase January 2008 - March 2010

was not implemented because the pro-

gramme does not correspond to the prior-

ity areas defined for bilateral cooperation

with Mexico

'Upgrading of Municipal Services, Turkey':

Planned duration January 2000 - Decem-

ber 2005, began late due to the lack of a

programme agreement, and the term was

abbreviated due to design deficits and in-

tervention by other projects/ programmes.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 22

1.2.2 Methodological approach

The fifteen evaluation reports in their accepted

form served as the basis for the cross-section

evaluation and for producing the synthesis

report. In addition, the respective offers which

the GTZ had submitted to the BMZ were used

as a source of supplementary information as

well as for verifying the information contained

in the evaluation reports (particularly with re-

gard to objectives systems, markers, project

terms and funding volumes). The cross-section

evaluation rests exclusively on those two

sources of data for documenting the specific

situation of each project/programme; no addi-

tional studies or research were carried out. It

must therefore be called to the reader's atten-

tion that the results of the cross-section

evaluation depend very heavily both on the

quality of the evaluation reports and on the

informational content of the GTZ offers (see

also the following Section 1.2.3).

The evaluation reports were subjected to com-

prehensive quantitative and qualitative analy-

sis. The quantitative analysis was based firstly

on the objectives systems (objectives and indi-

cators) established for the development meas-

ures and, secondly, on the evaluation of indi-

vidual projects and programmes according to

the DAC criteria (see also Section 1.1) based

on the 'GTZ Guidelines on Evaluating the Suc-

cess of Projects/Programmes' (see Annex 3).

In that context, core statements based on the

respective unweighted ratings of the DAC crite-

ria were elaborated for the synthesis report,

because the weightings in the individual

evaluation reports were regarded as too incon-

sistent for general evaluation purposes. The

practice of weighting the DAC criteria was

discontinued by the Evaluation Unit in 2009 on

the basis of the non-transparent explanations

given for upgradings and downgradings.

The quantitative findings described further on

are the result of statistical analyses that should

be viewed in the light of the small random

sample (15 projects/programmes) and of the

highly heterogeneous content of the projects/

programmes (in combination with considera-

tion of three different types of evaluation).

While the stated averages are based on the

respective arithmetic mean values drawn from

the individual evaluations, potential dependen-

cies were accounted for by way of calculated

rank correlations (Spearman)8.

The qualitative analysis is based on the reflec-

tions and assessments of the development-

policy cross-cutting themes and sectoral as-

sessments documented in the evaluation re-

ports and on the expressed recommendations.

The evaluators made a qualified, though still

subjective, selection and interpretation of the

core statements in the evaluation reports. At

the same time, the argumentation chains in the

evaluation reports were subjected to a critical

review. That, too, constitutes a subjective re-

flection which, in a specific case, logically

leaves room for debate and alternative inter-

pretation.

The reference material for the qualitative

analysis of the evaluation reports consisted of

8 A rank correlation coefficient is a non-parametric meas-

ure of correlation - that is, it assesses how well an arbi-

trary monotonic function describes the relationship be-tween two variables without making any assumptions

about the frequency distribution of those variables. The

rank correlation coefficient is robust with respect to outliers and can be applied to non-linear correlations. Spearman’s rank correlation coefficient:

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 23

the usual GTZ and BMZ background docu-

ments, in particular the 'BMZ Water Sector

Strategy' (2006), the 'Guidelines for Successful

Poverty Reduction in the Work of GTZ' (2007),

'Sustainable Development - GTZ‟s Concept'

(2006, see also Annex 5) and the 'GTZ‟s Un-

derstanding of Capacity Development: A Guid-

ing Framework for Corporate Action' (2007,

see also Annex 6).

In the portrayal of factors of success and fail-

ure for evaluating individual DAC criteria (see

also Section 3), several examples were bor-

rowed from the evaluated projects/ pro-

grammes and explained. This is in line with the

expectations of the sectoral structure of the

GTZ‟s 'Water' Section and is intended to make

the respective ratings more understandable

with the help of practical information and an

explanation of the relationships involved.

1.2.3 Comparability of the evaluation

reports

The uniform structure of the evaluation reports

as well as the uniform assessment criteria

based on the DAC criteria ensure the compa-

rability of the individual evaluation reports cov-

ered by the cross-section evaluation (see also

Sections 1.1 and 1.2.2). The corresponding

terms of reference (see also Annex 2) and the

'GTZ Guidelines on Evaluating the Success of

Projects and Programmes' (see also Annex 3)

contain appropriate, uniform key questions for

use by the evaluators of the individual projects.

At the same time, it is prescribed that, in ad-

vance of any evaluation, the results constella-

tion of the project or programme be estab-

lished and the methodological approach of the

evaluation mission be accordingly harmonised

with it (Inception Report). The Evaluation Unit

is not only responsible for training the evalua-

tors but also for the quality of the respective

reports. To that end, the unit organises pre-

liminary talks and evaluation meetings to be

attended by the evaluators and representatives

of the participating organisational units of the

sectoral and regional structure. The results of

each individual evaluation are also subject to

comment by the participating implementing

structure.

It should, however, be kept in mind that, de-

spite extensive standardisation, the compara-

bility of the resulting evaluation reports remains

limited. The evaluated development measures

are naturally very heterogeneous both in terms

of their content and design. Also, the evalua-

tions were performed by evaluators from dif-

ferent institutions armed with different empirical

data and standards of comparison. Despite the

effort to remain objective, their respective in-

terpretations and assessments are very sub-

jective.

The 'atmospheric' circumstances of the respec-

tive evaluations are also important in this con-

nection. Acceptance and comprehension prob-

lems on the part of project staff regarding the

instrument of independent evaluation, in com-

bination with inappropriate, insensitive actions

on the part of the evaluators can disrupt the

course of evaluation and produce unbalanced

results. Ultimately, the quality of the evaluation

results depends crucially on collective attention

to the questions raised. Not infrequently, 'at-

mospheric disturbances' between the evalua-

tion teams and the project teams culminate in

evaluation reports that are marred by incom-

pleteness or unsatisfactory quality. In the pre-

sent, concrete evaluation sequence, such

problems were noticed in the evaluations of the

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 24

'Integrated Water Management Programme,

Algeria' and 'Water Sector Reform Programme,

Zambia'. In those two cases, the local project

team perceived the behaviour of the evaluators

as being inappropriate and inadequately

geared to cooperation. In the case of the

evaluation in Algeria, the project team also

considered the evaluators' analysis and con-

clusions to be incorrect.

The assessment of the provided evaluation

reports also brought to light that, despite very

intensive quality assurance on the part of the

Evaluation Unit, it was not possible to rectify all

of the weak points of the individual evaluation

reports with regard to „technical‟ aspects of the

evaluation process. In some cases, the evalua-

tors tasked with performing the cross-section

evaluation would have come to different con-

clusions about the quantitative rating of the

individual projects and programmes than the

individual evaluators. It must be unequivocally

noted that not all of the ratings and weightings

are transparent and/or consistent with the in-

formation (and corresponding key questions)

contained in the reports. Regarding the qualita-

tive ratings, the evaluation reports' comparabil-

ity can be summarised in a single sentence:

'not every report makes statements about ar-

eas on which statements should be made.'

Relevant statements are missing in a signifi-

cant share of the reports, particularly in the

analysis of objectives systems and the as-

sessment of important cross-cutting themes. At

the same time, the conceptual principles con-

tained in the relevant guidelines and recom-

mendations on capacity development or sus-

tainable development, for example, are not

always given consistent consideration in the

assessment.

The only consistently identifiable information

about the planned and implemented use of

funds in the various projects and programmes

covered by the reviewed evaluation reports,

relates to the German TC inputs with reference

to the GTZ offers submitted to the BMZ for

those projects and programmes. As far as

actually implemented partner contributions and

FC funding are concerned, a large portion of

the information contained in the evaluation

reports is based on assumptions and esti-

mates. Concrete data are only stated for some

of the projects and programmes. Conse-

quently, no analyses of the utilisation of funds

by the implementation and cooperation part-

ners were performed within the scope of the

cross-section evaluation.

Against that backdrop, analysis of the fifteen

evaluation reports on the priority area 'water'

only allows limited statements to be made

regarding the GTZ‟s overall portfolio for that

sector. The selected development measures

do not constitute a representative selection of

projects and programmes within the sector.

Nevertheless, recurrent core statements and

identified trends do make it possible to draw

conclusions about the need for further action

and/or about factors of success and failure. In

parallel, but only to a minor degree, good ap-

proaches and 'products' (best practices) in the

implementation of individual projects/ pro-

grammes can be identified and recommended.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 25

2. Objectives and activity areas

2.1 Brief description of the evalu-

ated projects and programmes

As mentioned in Section 1.2.1, the technical

orientation, setting of priorities and objectives

systems of the evaluated development meas-

ures differ in part substantially. Only a small

portion of the projects and programmes are

based on mutually comparable concepts (e.g.

Yemen, Kenya and Zambia). However, in view

of the quite complex objectives systems in

some cases, it does not appear expedient to

differentiate the projects/ programmes accord-

ing to sectoral priorities, nor would that be of

relevance for this cross-section evaluation. In

the following section, the projects and pro-

grammes are therefore grouped according to

type of evaluation and presented in the form of

brief individual summaries.

2.1.1 Interim evaluations

The following eight projects/programmes were

the subject of interim evaluation:

Nile Water Initiative – NWI

Planning and Management of Water Re-

sources in the Nile Basin, Africa suprare-

gional

Overall objective: 'The Water Policies of the

Nile Basin countries have converged closer

towards joint cross-border management of the

Nile.'

The purpose of this project is to integrate the

challenge of integrated, transboundary man-

agement of the Nile into the national water

policies by means of reform-policy advisory

services, hence achieving convergence of the

riparian policies. The IWRM approach provides

the foundation. The lead executing agency is

the Nile Basin Initiative (NBI), based in En-

tebbe, Uganda. NBI was established in 1998 to

encourage regional dialogue between the Nile

riparian states and, in doing so, reduce con-

flicts over the allocation of limited water re-

sources. The project's implementing organisa-

tions are the ministries of water affairs in the

NBI member countries (Egypt, Ethiopia, Bu-

rundi, Democratic Republic of the Congo,

Kenya, Rwanda, Sudan, Tanzania and

Uganda). The target groups are the water us-

ers of the Nile watershed. Being oriented to-

wards transboundary water resources man-

agement, the project is in conformance with

the BMZ strategies for regional development

cooperation in Africa. Since work is being done

at the regional (transnational), the national

political level as well as the institutional level,

both the macro level and the meso level are

involved. Basically, three different areas are

combined: confidence building, capacity devel-

opment among personnel at the water minis-

tries, and revision of water policies and legal

frameworks in the NBI member countries. Indi-

rectly, collaborative planning of investment

projects is to be simplified, hence promoting

economic development throughout the region.

Integrated Water Management Programme,

Algeria – PDEA

Overall objective: 'The capacities of the water-

sector institutions (Ministry of Water, regional

authorities) for integrated water management

are improved.'

This programme's priority area is to provide

water-policy advice. By incorporating the sus-

tainable use of resources into the country's

water policy, the programme aims to promote

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 26

the efficient use of water and curtail the other-

wise increasing use of untreated water. Capac-

ity development in the sector institutions is

being promoted by advisory and training

measures at the national, regional and local

levels (multi-level approach). The main themes

of action are integrated water resources man-

agement, sector planning, regulation of water

supply and sanitation (WS&S), organisational

development and knowledge management.

The lead executing agency is the Algerian

Ministry of Water, which is responsible for

managing the entire country's water resources.

The ministry is also the implementing organisa-

tion for WS&S planning and regulation. Other

implementing organisations include the na-

tional water utility, regional river basin agen-

cies, the water management agency in Béchar

and the community of Béni Abbès. The target

group is the entire population of Algeria. Im-

proved water resource development and man-

agement should lead indirectly to improve-

ments in the people's hygiene and health situa-

tion.

Water Sector Programme, Yemen - WSP

Institutional Development of the Water Sec-

tor

Overall objective: 'Municipal water supply,

wastewater management and water resources

management in the programme region is im-

proved.’

This cooperative programme aims to improve

urban WS&S and WRM in Yemen by providing

technical, process and organisational advisory

services. The five components of the coopera-

tive programme include a reform of the institu-

tional framework conditions, human resource

development, the development of independent,

commercialised WUs, IWRM and strengthen-

ing of local actors. The programme pursues a

multi-level approach. Its thematic priority areas

at the national level are decentralisation and

commercialisation, both of which are to be

implemented and institutionalised at the local-

level water utilities (WUs). The regional-level

priority area is the generation of WRM plans,

which are to be mainstreamed at both the na-

tional and the local level. At the regional level,

16 WUs receive advisory services geared to

enhancing their operational efficiency in terms

of organisational structure, technical processes

and procedures, and financial/commercial

matters, plus the introduction of a management

information system (MIS). At the local level,

governorate and district administrative authori-

ties receive technical and methodological sup-

port to create water catchment committees and

water user groups and conduct public aware-

ness campaigns and planning water resource

development measures. The lead executing

agency and implementing organisation is the

Ministry of Water and Environment. Additional

implementing organisations include the Na-

tional Water Resources Authority and the Min-

istry of Local Administration. Together with the

WUs and representatives of the local admini-

strations, technical and managerial personnel

from those institutions function as intermediar-

ies. The target group are the people of Yemen.

Efficient Water Resources Management in

Irrigated Agriculture in the Jordan Valley

and Highland Areas, Jordan – WMIA

Overall objective: 'Governmental water institu-

tions, operators and farmers manage the water

resources in an efficient and sustainable man-

ner.'

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 27

Component objective: 'The efficiency of water

resources management in irrigated agriculture

in the Jordan Valley and the Highlands is in-

creased.' 'Governmental water institutions,

operators and farmers manage the water re-

sources in an efficient and sustainable man-

ner.'

The evaluation was based solely on compo-

nent 2 of the 'Management of Water Re-

sources Programme, Jordan', which evolved

from four previous projects in 2006. For that

reason, the component objective is of rele-

vance. The thematic priority areas of the pro-

gramme are WS&S, irrigated agriculture and

WRM planning. The programme itself is de-

voted to the sustainable use of existing water

resources and to a fair reconciliation of conflict-

ing interests between households, the industry,

tourism and agriculture. It takes a multi-level

approach. At the national level, it contributes to

decision-making through capacity development

within the ministry and a critical examination of

the institutional framework conditions. Regional

WUs receive advisory support in technical and

operational matters. Their technical and man-

agement personnel are kept abreast of cutting-

edge technology and management practice by

appropriate training. At the local level, the pro-

gramme cooperates closely with irrigation

farmers. The programme component com-

prises three subcomponents: participatory

irrigation management with a focus on institu-

tion building and capacity development among

water-user associations, the use of treated

wastewater for irrigation with a focus on farmer

training, and groundwater management with a

focus on reduction of extraction rates. The use

of treated wastewater for irrigation purposes

makes the allocation of scarce freshwater re-

sources more flexible. A system for monitoring

the quality of soil, water and field crops was

introduced as part of the programme. The lead

executing agency is the Ministry of Water and

Irrigation. The implementing organisations are

the Jordanian Water Authority and the Ministry

of Agriculture. The programme's target groups

are the entire population of the country and, in

particular, irrigation farmers in the Jordan Val-

ley and surrounding highland areas.

Water Sector Reform Programme, Kenya –

WSRP-KEN

Overall objective: 'Sustainable access of the

urban poor to safe drinking water and basic

sanitation is increased and the water resources

management improved.'

The aim of this programme is to improve water

resources management (WRM), access to safe

drinking water and basic sanitary services for

the urban poor. Poverty relevance is a central

advisory element. The five components of this

programme are: support for the Water Ministry

in sector reform, regulation of the water sector

and pro-poor financing, commercialisation of

water utilities (WUs), capacity building for the

Water Resources Management Authority, and

introduction of recycling-oriented sanitation

services (EcoSan) - all of which reflect the

programme's multi-level approach. The lead

executing agency and implementing organisa-

tion is the Ministry of Water and Irrigation.

Other implementing organisations include the

regulatory committee, the trust fund, the Water

Resources Management Authority (WRMA)

and the commercialised water utilities (WUs) -

plus, at the local level, the water user associa-

tions and catchment area advisory committees.

The target group encompasses the water us-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 28

ers and the consumers of WS&S services in

poor urban areas. Access to WS&S services is

improved by the construction of water kiosks

and public sanitation centres. Indirectly, the

programme helps to improve the hygiene and

health situation and, hence, the living condi-

tions of the country's population.

Drinking Water and Sanitary Programme,

Peru – PROAGUA

Overall objective: 'The management of drinking

water supply and wastewater disposal in se-

lected cities of Peru is sustainably improved.'

This programme is geared to improving access

to drinking water while lowering the cost of

supplying water and improving its quality. Ad-

visory services provided to sector institutions

and water utilities are generally geared to im-

proving the operative and commercial man-

agement of eight WUs, implementing inte-

grated WS&S investments at two WUs to

strengthen the 'management of the social envi-

ronment', expanding the extent of community

participation, and generally improving the train-

ing structures for WUs. The programme works

with WUs at the meso level and with sector

institutions at the macro level. It aims to im-

prove the efficiency of water utility operations

and services in cities. At the sector level, a

national system of capacity development struc-

tures for advisory support and training, i.e. for

further development within the sector, is to be

created. The programme promotes sector re-

form. The lead executing agency is the Peru-

vian Vice Ministry of Construction and Sanita-

tion within the Ministry of Housing, Construc-

tion and Sanitation. The implementing partners

are the National Superintendence of Sanitation

Services, the WUs in selected small and me-

dium-size towns and cities and the National

Association of Water Utilities. The target group

is the population of the various WU service

areas, especially poor people living in urban

peripheries. The programme is intended to

improve living conditions and to reduce the

frequency of illnesses caused by contaminated

water.

Water Sector Reform Programme, Zambia –

WSRP-ZMB

Overall objective: 'The poor population's ac-

cess to safe drinking water and sanitary facili-

ties is improved and integrated water re-

sources management is introduced.'

The programme aims to improve the supply of

drinking water and sanitation services to poor

people and to introduce the IWRM approach. It

pursues a multi-level approach, as reflected in

the programme components. Those compo-

nents include policy advisory services and

regulation, advisory services for implementing

poverty-reduction approaches in the trust fund,

advisory services for the WUs for improving

their operative management, support for the

ministries with regard to sector coordination

and planning, and policy advisory services in

connection with IWRM. At the macro level, the

programme provides advisory services for

sector policy. At the meso level, it supports the

development of institutions, financing instru-

ments and national information systems. At the

micro level, it advises water supply and sanita-

tion companies and supports the development

of water-user groups. The lead executing

agency is the Ministry of Energy and Water

Development. The implementing organisations

are the Zambian National Water Supply and

Sanitation Council with its Water Watch

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 29

Groups, three commercial WUs, the Depart-

ment of Water Affairs and Water Board within

the ministry, the trust fund and the Ministry of

Local Government and Housing. The target

group is the poor peri-urban and rural popula-

tion. The programme is geared to improving

sanitation and hygiene conditions. The use of

unsafe water sources is to be further curbed by

the installation of water kiosks.

Wastewater and Solid Waste Management

in Provincial Centres, Viet Nam – WDPC

Overall objective: Municipal service providers

in six provincial centres are operating the im-

proved rainwater and sewage systems cost

efficiently and according to plan.

The project aims to improve wastewater and

solid waste management according to a con-

cept based on the multi-level approach. At the

Ministry of Construction, capacities for action in

the wastewater and solid waste sector are

being strengthened, and human resource train-

ing is to be supported by providing advisory

services to the technical and management

personnel. Provision of institutional advice to

the municipal enterprises is intended to ensure

proper operation of the surface-runoff and

sewage systems. Targeted training aims to

qualify the staff to operate and maintain the

systems. In addition, the legal basis for regulat-

ing the management of wastewater and solid

waste is being established at the macro level.

The Ministry of Construction is the lead execut-

ing agency, and the municipal wastewater

management companies (utilities) in the six

provincial capitals are the implementing or-

ganisations. The target group is the population

of the six partner communities. The project

contributes indirectly to health promotion by

improving settlement hygiene.

2.1.2 Final evaluations

Three of the fifteen projects/programmes were

concluded only recently but already subjected

to final evaluation:

Improvement of Water Supply - Eastern and

Volta Region Assistance Programme,

Ghana – EVORAP

Overall objective: The continual, needs-driven

supply of hygienically sound drinking water to

the populations of selected small towns in the

Eastern and Volta Regions is ensured.

The intention of this programme was to con-

tribute to a qualitatively and quantitatively im-

proved drinking water supply in the Volta and

Eastern region. Its multi-level approach en-

compassed provision of technical, organisa-

tional and policy advisory services. The the-

matic priority areas of the advisory services

and training at the national, regional and local

levels were the operation and management of

provincial water supply systems, organisational

development and hygiene education. At the

macro level, the advisory services were geared

to reforming the sector policy and legal frame-

work. At the micro level, the programme was

oriented towards rehabilitating the drinking

water supply networks in the selected small

towns with the active participation of the local

population. The lead executing agency was the

national Community Water and Sanitation

Agency (CWSA - a regulatory and controlling

body responsible for water supply in the rural

areas and small towns). The implementing

organisations were the District Water and Sani-

tation Teams (DWST) and the Water and Sani-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 30

tation Development Boards (WSDB). The tar-

get group of the programme comprised the

populations of 29 small towns in the Eastern

and Volta Regions, including the staff of the

sector institutions. As an indirect impact, the

project was intended to contribute to an im-

provement in the hygiene and health situation

of the population and, hence, their quality of

life.

Infrastructure Programme - Water Supply

and Wastewater Disposal, Kosovo -

WSWDP

Overall objective: Operational efficiency in the

regional water companies Peja and Prizren is

enhanced.

This programme was geared to enhancing the

operational efficiency of the regional water

companies in Peja and Prizren. It was strictly

confined to the WU level. The companies in

question received support for their organisa-

tional development, financial and customer

management, mapping of their WS&S net-

works and drinking water connections, plant

evaluation, water loss management, and the

introduction of a geographical information sys-

tem (GIS). A management information system

(MIS) was also introduced in order to improve

the companies' financial decision-making basis

by means of an auditable bookkeeping system.

A maintenance database was constructed and

maintenance plans prepared in order to ensure

trouble-free operation over the long term, as

well as the preservation of the plants them-

selves and the investments they represent.

WS&S is intended to operate with commercial

efficiency based on the operating-cost-

recovery principle and, later on, with full cost

coverage. The lead executing agency was the

Government of Kosovo, and the implementing

organisations were the regional water utility

companies in Peja and Prizren. The target

group was the population of both regions. Indi-

rectly, the programme aimed to prepare the

companies to become independent legal enti-

ties and be responsible for WS&S manage-

ment. The people were supposed to gain en-

sured access to WS&S.

Management of Water Catchment Areas for

the Río Lerma in the Toluca Valley and for

the Río Balsas, Mexico – GICA

Overall objective: Management of the aquifer

catchment in Toluca Valley by water sector

actors is improved.

This project aimed to improve water resources

management in the state of Mexico by creating

an institutional structure for a water resources

management plan and by improving the quali-

fications of actors in the water sector. This

encompassed process advisory services, train-

ing and follow-up assistance of pilot measures.

The project concept was based on a multi-level

approach, meaning that the work was per-

formed at state, regional and local levels. Wa-

ter resources management activities at state

institutions were systemised, advisory services

and training offered to partner organisations

and principal actors, and water users mobilised

and educated about the conscious use of wa-

ter. The lead executing agency was the na-

tional water commission (CONAGUA), and the

regional CONAGUA office in the state of Mex-

ico was the implementing partner. The target

group was the population of the project region.

Indirectly, the project intended to promote sta-

bilisation of decentralised water resources

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 31

management while improving the region's

drinking water supply situation.

With regard to the comparability of these

evaluation reports, it should be noted in con-

nection with the 'Infrastructure Programme

Water Supply and Wastewater Disposal, Kos-

ovo' that the final evaluation was performed

eight months before the end of the project term

(June 2009), i.e. in October and November of

2008. Hence, in the opinion of the evaluators, it

cannot, in the literal sense, be regarded as a

final evaluation.

2.1.3 Ex-post evaluations

Four of the evaluated projects/programmes

were concluded some time ago and therefore

assessed within the framework of an ex-post

evaluation:

Improvement of Water Supply in the Tigray

Region, Ethiopia – IWSTR

Overall objective: A reliable and sustainable

water supply management system for provin-

cial and rural areas is established, is opera-

tional at selected locations and can be repli-

cated by the Ethiopian agencies concerned.

This project aimed to introduce a water supply

management system in the provincial and rural

region. The project focused mainly on meas-

ures at the regional administrative level and

consisted largely of training and the introduc-

tion of software for spare-parts management at

repair shops. Other contributions included the

drilling of wells and the supply of pumps and

large-scale water meters. The lead executing

agency was the Ministry of Water Resources.

The implementing organisation was the Tigray

Water Resources Development, Mines and

Energy Bureau. The target group was the

population of 8 pilot and 19 replication towns.

An indirect aim of the project was to improve

the supply of water throughout the Tigray Re-

gion.

Indo-German Watershed Development Pro-

gramme, India – IGWDP

Overall objective: Governmental and nongov-

ernmental implementing organisations are

applying methods for improving the sustainabil-

ity and breadth of impact in integrated, gov-

ernment-sponsored watershed management

programmes.

The project was geared to improving public

programmes for management of watershed

areas. It provided advisory services to gov-

ernmental and non-governmental institutions

and self-help groups and implemented capac-

ity development measures within the partner

organisations. The project pursued a multi-

level approach and operated at national, re-

gional and local levels. The lead executing

agency and implementing organisation was the

Ministry of Agriculture. Ministerial staff and

evaluators from other governmental and non-

governmental organisations served as media-

tors. The target group was made up of land

users in selected, representative watersheds.

The indirect aim of the project was to improve

the supply of food and household income.

Rural Water Supply in Maradi, Niger –

PHV/MI

Overall objective: Continuous, year-round sup-

ply of safe drinking water to the rural popula-

tion of the programme region is assured.

This project aimed to improve the supply of

drinking water in rural areas of the Maradi re-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 32

gion by installing/rehabilitating dug wells. The

measures comprised the application of new

and appropriately modified dug-well construc-

tion techniques and the creation of a basis for

controlled, self-responsible, resource-

conserving operation of the wells by the village

communities. The project therefore intervened

primarily at the local level. The lead executing

agency was the Ministry of Water. The regional

water directorate served as the implementing

partner, and the village communities were the

target group. By increasing the supply of water

and reducing the time required for fetching

water, the project was to have the indirect ef-

fect of expanding/securing the economic activi-

ties and increasing household income accord-

ingly.

Upgrading of Municipal Services, Turkey

Overall objective: The inter-institutional, pro-

fessional network and the relevant training

measures in the water supply, wastewater

disposal and waste management sector are

improving the political, institutional and techni-

cal framework conditions for the provision of

municipal services in selected cities.

The project aimed to improve the framework

conditions for WS&S. By means of a thorough

administrative reform and attendant decentrali-

sation, the municipalities and/or the concerned

WUs were to be given additional opportunities

for engaging in self-responsible management.

The efforts focused on establishing an inter-

institutional, professional network. The project

operated mainly at the municipal administrative

level. The lead executing agency was the Min-

istry of the Interior, and the implementing or-

ganisations were the Institute of Public Admini-

stration for Turkey and the Middle East and the

Turkish Association of Towns and Municipali-

ties. The target group consisted of the popula-

tions of the towns concerned. The project

measures addressed policy-makers as well as

technical and administrative personnel of city

and municipal facilities. Indirectly, the project

was intended to improve the quality and sus-

tainability of municipal services and as well as

public awareness of the problem through in-

tensified public relations activities.

2.2 Analysis of objectives systems

A major share of the objectives definitions

underlying the evaluated projects and pro-

grammes contain at least two partial objectives

and/or sub-sectors of the water sector (see

also Section 1.2.1). Eight of the evaluated

projects/programmes have a very complex

objectives system made up of phases, compo-

nent objectives and corresponding indicators.

The programme 'Institutional Development of

the Water Sector, Yemen', for example, has a

total of 22 overall-objective and component-

objective indicators, and the 'Water Sector

Reform Programme, Kenya' has a total of 24

overall-objective and component-objective

indicators.

In the evaluation reports, the objectives sys-

tems formulated for the projects and pro-

grammes were given different ratings, but most

of them were classified as suitable. It was ob-

served in most of the evaluations that, in the

course of implementation, the indicator sys-

tems in particular required adjustment with

respect to the formulations in the GTZ offers,

because the levels of intervention and, often,

the results constellations tend to vary. For

three of the projects/programmes (Kosovo,

Niger, Turkey), the respective evaluation mis-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 33

sion reformulated the objectives systems for

evaluation purposes, because the stated ob-

jectives, indicators and results chains were

considered imprecise with regard to results

constellations and causalities and did not allow

comprehensive documentation of results di-

mensions. It must be noted in that connection

that some of the projects/programmes (in par-

ticular with regard to the final and ex-post

evaluations) were conceived and designed

prior to introduction of the Development-Policy

Framework for Contracts and Cooperation

(AURA) and were therefore still subject to the

old objectives-oriented project planning

(ZOPP) system. No results models from the

implementation phase are available for these

projects/programmes, Instead, the correspond-

ing models were subsequently formulated

within the scope of evaluation.

The present objectives systems are discussed

below with regard to the setting of priorities in

their activity areas and results levels. It should

be pointed out that in-depth reflections on the

respective definitions of objectives are only to

be found in some of the evaluation reports (8

of 15). None of the evaluation reports include a

pertinent analysis of how well the poverty,

gender and environmental dimensions are

included in the formulation of objectives.

2.2.1 Activity areas, priority areas

and indicators

Figure 1 below illustrates the distribution of

priority areas and activity areas based on the

ormulated objectives systems. There is a con-

spicuous focus on two areas: 'water supply'

and 'water resources management', while only

a very small portion of the pro-

jects/programmes have the activity area

'wastewater management/ sanitation services'

as part of their objectives system.

Urban areas account for approx. 60 % of the

set priorities in the projects and programmes,

and it should be noted that four of the pro-

jects/programmes are or were active in both

urban and rural areas (Figure 2).

The heterogeneity of the projects/ programmes

makes it hard to identify clear-cut trends in the

development of objectives systems. It is ap-

parent, however, that the themes '(drinking)

water supply', 'water resources management'

and 'water management' have been continu-

ously included in most objectives systems

since 1995/1996. The themes 'wastewater

management' and 'sanitation services', by con-

trast, were only found in the objectives defined

for two projects/ programmes, beginning in

2003.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 34

Figure 2: Proportions of urban and rural priorities in the evaluated pro-jects/programmes

Since 2001, the urban water sector has been

given increasing attention in the objectives

systems of projects and programmes, while

the rural water sector has found very little con-

sideration at the objectives level. This corre-

sponds to the trends established in the pro-

gramming of new projects and programmes

according to BMZ directives.

In the opinion of the evaluators, only little at-

tention was paid to the dimensions 'poverty',

'gender' and 'sustainability' at the formulation-

of-objectives level. Poverty relevance was only

included at the objectives level of two evalu-

ated projects/programmes (Kenya and Zam-

bia). The concept of sustainability is mentioned

in four of the reviewed objectives systems

(Ethiopia, India, Kenya, Peru), but since 2004,

that dimension has been scarcely included at

all in the formulation of objectives (namely in

only one of nine objectives systems formulated

since then).

9%

43%17%

26%4%

Sector Policy / Regulation

Water Supply

58%

42%

urban

rural

Figure 1: Distribution of activity areas in the evaluated

projects/programmes

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 35

Figure 3: Trends in the thematic orientation of objectives systems

At the level of indicators for achievement of

objectives (decisive for the determination of

effectiveness), the reviewed objectives sys-

tems clearly emphasise the results constella-

tion 'drinking water supply - standards of sup-

ply - operational efficiency'. Corresponding

indicators were formulated for ten of the fifteen

development measures, albeit to varying de-

grees. Altogether, some 48% of the indicator

formulations can be allotted to that results

constellation. This is somewhat surprising,

since only one programme (Kosovo) displayed

a direct connection between the actual formu-

lation of objectives and improvements in the

operation of water utilities. In addition, only six

of the projects/programmes intervene at the

WU level in the sense of improving the supply

of drinking water. (Despite the formulation of

corresponding indicators, no direct correlation

was established for the projects/programmes

in Algeria, Ethiopia, Niger and Mexico.) Con-

spicuously, too, indicators concerning the

'wastewater management/sanitation services'

sector were formulated for only three of the

projects/programmes (Algeria, Yemen and Viet

Nam), whereas the project in Jordan had one

indicator relating to the area 'reuse of re-

claimed water'.

The results constellation 'sector policy - regula-

tion - governance' is found in the indicator sys-

tems of two programmes (Africa supraregional

and Algeria).

IWRM principles were taken into account in the

indicator systems of only five of the evaluated

projects/programmes (Algeria, Yemen, Kenya,

Mexico, Zambia), despite the fact that the con-

cepts 'water resources management' and/or

'water management' were included in the for-

Water Supply

Management of Water

Resources /

Water Management

Wastewater

Management /

Sanitation

urban

20081995 2000

rural

constant since 1995

constant since 1996

increasing since 2003

decreasing strongly since 2001

increasing strongly since 2001

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 36

mulation of objectives for nine of the evaluated

projects.

In the objectives systems of the evaluated

projects/programmes, indicators designed to

register poverty, gender and environmental

dimensions are included on a very small scale

that is not considered appropriate in principle

to the objectives of German DC. Only four

projects/ programmes have an indicator for

determining the poverty dimension (Yemen,

Kenya, Mexico and Zambia), and only three

(Yemen, Kenya and Zambia) register the gen-

der dimension via appropriate indicators. Like-

wise four projects/programmes (India, Yemen,

Jordan and Kenya) have objectives systems

that include indicators for the environmental

dimension. This stands in stark contradiction to

the fact that, as dealt with in Sections 4.1 and

4.2 below, most of the projects/programmes

had DAC markers identifying the poverty, gen-

der and environmental dimensions as secon-

dary objectives.

All told, it can be said of the project/ pro-

gramme indicator systems that only a very

small share of the formulated appraisal factors

relate to concrete, results-based 'outcomes'

(e.g. supplies, poverty reduction, health). At

the same time, measurable factors for estab-

lishing indirect results are also missing.

Good examples of (objectives-level) indicators for documenting the poverty dimension:

'Institutional Development of the Water Sector, Yemen': 'A survey conducted among the popula-

tion living in the assisted municipalities - with special focus on representatives of local women's

groups - confirms that measures in water supply and sewerage have contributed to an im-

provement in their overall living conditions’.

'Water Sector Reform Programme, Kenya': 'At least 1.6 million poor people in the assisted

towns have more access to safe, affordable drinking water’ (report by water sector trust fund).

'Water Sector Reform Programme, Zambia': 'The share of people in poor urban settlements with

access to hygienically safe drinking water at socially acceptable prices has risen from 54%

(2005) to 80% (reports by the sector information system)’.

All projects and programmes have very long

results chains, and the actual results constella-

tion of the respective project or programme is

therefore projected behind the 'attribution gap'

(range of indirect and highly aggregated re-

sults) in the results model. Results counting as

'indirect' can therefore only be 'sensed', not

tangibly registered. The question arises as to

whether the cautious formulation of indicators

could be attributable to a certain desire to 'stay

on the safe side'. A little more courage to 'face

the consequences' of choosing and formulating

indicators would be appropriate, since many

evaluation reports refer to 'palpable' results in

the relevant project context, whereas the cor-

responding databases are missing due to im-

precise formulation of indicators or a lack of set

priorities. In the opinion of the evaluators, the

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 37

evaluated projects and programmes can be

presumed to have achieved far more results

than were actually recorded.

2.2.2 Intervention and results

levels

German TC projects and programmes are

designed according to a results model in which

relationships are established between the im-

plemented activities and the ensuing direct and

indirect results. The GTZ results model de-

scribes the connection between a project or

programme and development-policy changes

by way of the project's or programme's active

inputs, activities, outputs, the use of those

outputs by other actors, the direct benefits

deriving from such use (outcomes or direct

results) and other forms of developmental pro-

gress that can be described on the basis of

plausible hypotheses as indirect benefits of the

respective project or programme.

By way of analogy to the major differences in

thematic and conceptual orientation, the over-

view of the evaluated projects and pro-

grammes in the water portfolio shows a very

heterogeneous pattern with respect to the di-

rect results level of the respective projects and

programmes. Of decisive importance here too

is the structuring of a 'multi-level approach' in

the design of the projects/programmes. The

following illustration (Figure 4) attempts to

show how the anticipated direct results of the

selected projects and programmes concentrate

on the meso level with respect to the develop-

ment and strengthening of sector institutions

and their intermediaries, or to 'networking'

within the water sector. Some 75% of the pro-

jects and programmes achieve results in that

area. Roughly two thirds of the projects and

programmes are geared to improving sector

management and regulation. However, only

about one third of the projects and pro-

grammes aim to have a direct effect on

changes in the framework conditions and re-

requisites for development at the macro level.

The proportion of projects having concrete

micro-level results with regard to quality-of-life

improvements for the population is situated at

around 40%. Another two thirds of the projects

and programmes achieve results in the devel-

opment of capacities and competencies among

the intermediaries and target groups, hence

establishing an essential basis for improving

living conditions by use of investments and

sector-specific know-how.

The diagram also visualises the fact that only

three of the programmes (Yemen, Kenya and

Zambia) have a complete, conceptually main-

streamed multi-level approach spanning all

results levels. It is also apparent that, with the

exception of the three projects/programmes

'Nile Water Initiative, Africa supraregional',

'Infrastructure Programme, Kosovo' and 'Rural

Water Supply in Maradi, Niger', all projects and

programmes have substantial results constella-

tions that span all levels. On the basis of the

selected projects and programmes, the results

orientation appears not to be subject to any

time-dependent trends.

As far as the programme „Nile Water Initiative,

Africa supraregional' is concerned, it should be

explained that the multi-level approach is dif-

ferentiated within the regional context. Within

the context of a supraregional programme, the

macro level is the level occupied by intergov-

ernmental bodies, while the meso level is con-

cerned with the respectively highest national

institutions.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 38

The 'Rural Water Supply in Maradi, Niger' pro-

ject purposely dispensed with a multi-level

approach in the project design. This was at-

tributed to excessive weakness on the part of

the lead executing agency and a great need for

direct intervention at the target-group level

(construction of dug wells for the supply of

water). Planned prior to 1995, this develop-

ment measure constitutes a type of project for

which a rationale can hardly be found any-

more, as it predated both the introduction of

the Development Policy Framework for Con-

tracts and Cooperation - AURA (2002) and the

orientation of TC towards results and sustain-

able development.

Figure 4: Direct results levels of the evaluated projects/programmes

In-depth information about the objectives

and results of the individual projects and

programmes can be found in the respec-

tive brief descriptions contained in Section

2.1. Further analyses and observations re-

garding the overarching development-

policy objectives of the projects and pro-

grammes are contained in Section 3.1

(relevance rating), 3.3 (impact rating) and

4.1 (poverty reduction and MDGs).

Macro Level

Meso Level

Micro Level

Improvement of Living

Conditions

Capacity Development

of Intermediate

Organisations

Capacity Development

of Sector Institutions

Improvement of

Framework Conditions

Af

ri

ca

N

.A

.

Al

ge

ri

a

Ye

me

n

Jo

rd

an

Ke

ny

a

Pe

ru

Za

mb

ia

Vi

et

na

m

Gh

an

a

Ko

so

voM

ex

ic

o

Et

hi

op

ia

In

di

a

Ni

ge

r

Tu

rk

ey

Capacity Development

of Target Groups

Sector Coordination

and Regulation

"Sector Networking"

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 39

3. Rating according to DAC Criteria

In the following sections, the evaluated pro-

jects and programmes are rated according to

the DAC criteria, and the results are corre-

spondingly summarised. To improve the trans-

parency and traceability of the respective rat-

ings, the factors determining success and fail-

ure for rating the individual DAC criteria are

explained by means of several examples from

the evaluated projects and programmes.

Within the scope of the independent evalua-

tions, the ratings for the DAC criteria 'rele-

vance', 'effectiveness', 'impact' and 'efficiency‟,

as well as the overall rating, are based on a

six-point scale (see also Annex 3: GTZ Guide-

lines on Evaluating the Success of Projects

and Programmes):

Table 2: Assessment scale for the DAC criteria relevance, effectiveness, impact and efficiency,

and for the overall rating

Success level Rating Description

'Successful'

'very good'

(1) Very good results, significantly better than expected.

'good'

(2)

Good results, fully in line with expectations, with no sig-

nificant defects.

'satisfactory'

(3)

Satisfactory results; falling short of expectations but with

positive results dominant.

'Unsuccessful'

'unsatisfactory'

(level 4)

Unsatisfactory results; significantly below expectations,

and negative results dominate despite identifiable positive

results.

'inadequate'

(5)

Clearly inadequate results: despite several positive partial

results, the negative results clearly dominate.

'useless'

(6)

The project/programme is useless, or the situation has

deteriorated on balance.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 40

The criterion 'sustainability' is rated according to the following four-point scale:

Table 3: Assessment scale for the DAC criterion sustainability

Sustainability level Description

'very good'

(level 1)

With a high degree of probability, the overall success of the project/programme

(positive to date) will continue unchanged or even increase.

'good'

(level 2)

With a high degree of probability, the overall success of the project/programme

(positive to date) will decrease only minimally and remain significantly positive

overall (normal situation – to be expected)

'satisfactory'

(level 3)

With a high degree of probability, the overall success of the project/programme

(positive to date) will decrease significantly but remain positive.

'inadequate'

(level 4)

The overall success of the project/programme is inadequate at the time of

evaluation, and there is a high degree of probability that it will not improve.

According to the applicable rating principles, a

project or programme can only be given a

'successful‟ overall rating in terms of develop-

ment policy if its direct results (effectiveness),

indirect results (impact or „overarching devel-

opment results‟) and sustainability all achieve a

rating of at least „satisfactory‟ (level 3).

In the present evaluation sequence for the

priority area 'water', a decision was taken and

justified in each evaluation and for each of the

five DAC assessment criteria as to whether it

was „very important‟ (weighting 3), „important‟

(weighting 2) or „less important‟ (weighting 1)

within the specific context of the project or

programme. In the absence of a special reason

making it more or less important, a given crite-

rion was held to be „important‟ (standard

weighting 2). In the evaluation of 'Rural Water

Supply in Maradi, Niger' (joint GTZ/KfW ex-

post evaluation), the performance assessment

for all DAC criteria was given a standard

weighting of 2.

3.1 Relevance

3.1.1 Relevance rating

The term relevance defines the extent to which

the objectives of a development measure

match the needs of the target groups, the poli-

cies of the partner countries and partner insti-

tutions, global development goals and the ba-

sic development-policy orientation of the Ger-

man Federal Government.

'Are we doing the right thing?'

With 2.0 ('good') as its average rating, rele-

vance emerged as the most successful evalua-

tion criterion. In fourteen of fifteen pro-

jects/programmes, the development measure

was successfully aligned with the policies of

the partner country, the needs of the respec-

tive target groups, the global development

objectives, and the basic development-policy

orientation of the German Federal Govern-

ment. Five of the projects/programmes (Africa

supraregional, Jordan, Peru, Viet Nam and

India) received a relevance rating of 'very

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 41

good'. Four of the projects/programmes are

still ongoing and were rated within the scope of

an interim evaluation. Only one programme

(Algeria) was rated 'unsatisfactory' according

to an interim evaluation (see also Table 4 and

Figure 5).

The average relevance rating varies according

to when the evaluation took place: While the

average rating in the interim evaluations, at

1.8, was still well within the good range, it

dropped to 2.3 or 2.5 in the final and ex-post

evaluations. That trend is attributable to the

fact that ratings given in connection with in-

terim evaluations are still strongly geared to-

wards the intended, conceptually targeted

orientation of the respective project or pro-

gramme, as measured against the develop-

ment-policy directives. To the extent that de-

viations or limitations are recognised at that

early stage, a need for realignment is usually

formulated with reliance on the respective per-

formance capability, without downgrading the

rating. This way, interim evaluation is able to

inject fundamentally positive impulses into the

projects and programmes, and realignment is

still possible thanks to the remaining length of

term. For a final evaluation, however, the situa-

tion differs in that corrections are no longer

possible, and any given limitations must, ulti-

mately, be rated with respect to the actually

achieved development-policy orientation. The

respective rating is then, apparently, more

critical in connection with an ex-post evaluation

(as compared to final evaluation), particularly

with regard to the orientation of interventions

towards the actual requirements of the target

groups.

33%

40%

20%7%

very good

good

satisfactory

not satisfactory

Figure 5: Distribution of relevance ratings

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 42

Table 4: Relevance ratings of evaluated projects/programmes

Country Project/Programme Type of

evaluation Rating Weighting

Africa su-

praregional NWI

Interim

evaluations

1 2

Algeria PDEA 4 3

Yemen WSP 2 3

Jordan WMIA 1 2

Kenya WSRP-KEN 2 3

Peru PROAGUA 1 3

Zambia WSRP-ZMB 2 3

Viet Nam WDPC 1 2

Ghana EVORAP Final

evaluations

3 2

Kosovo WSWDP 2 2

Mexico GICA 2 2

Ethiopia IWSTR

Ex-post

evaluations

2 2

India IGWDP 2 2

Niger PHV/MI 3 2

Turkey QADI 3 2

Overall (average) 2.0 2.3

Orientation towards key questions for as-

sessing the 'relevance' of a development

measure leads to the success factors. The

evaluated development measures were

rated „positive‟ in the individual evalua-

tions, if

they are in line with the policies and

strategies of the partner country and part-

ner institutions;

poverty analyses are conducted, and

poverty reduction was specified as a cen-

tral objective in accordance with poverty

reduction strategies;

thematic priority areas of BMZ for the wa-

ter sector are included in the design of the

measure and are in line with the sector-

specific policies, country strategies and

position papers (basic development-policy

orientation of the German Federal Gov-

ernment);

conformance with international standards

and conventions is given;

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 43

both the partners and the target group

attach high priority to the measures;

integrative and cooperative elements are

included, and the region in question is re-

garded as an integral whole in the case of

transboundary initiatives;

capacity development measures are in-

cluded;

a multi-level approach is accounted for;

gender analyses were conducted, and a

gender-responsive concept is in place.

Within the scope of the evaluations, the

following factors determining failure led to

downgrading of the relevance rating. De-

velopment measures were rated „nega-

tive‟ if

the core problem is not recognised, hence

resulting in erroneous setting of priorities;

the national strategies stand in conflict

with the basic development-policy orienta-

tion;

there is a conflict of objectives between

the short-term and long-term orientation of

the project or programme;

no relevance to poverty reduction is es-

tablished, and no poverty analysis was

conducted;

a gender-responsive conceptual design

and a gender analysis are lacking;

a multi-level approach is inadequately

implemented in that too few interventions

are provided at the target-group level (mi-

cro level).

Regarding the conduct of a poverty analysis, it

must be conceded that it is hardly possible to

perform the relevant investigations for regional

programmes as complex as the „Nile Water

Initiative, Africa supraregional'. In principle, the

requisite time and effort would be prohibitive –

a fact that should be considered for the rele-

vant rating.

The 'Indo-German Watershed Development

Programme, India' received a very good rele-

vance rating. The objective of the programme,

namely to improve the living conditions of peo-

ple living in the water catchment areas, was in

accordance with the essential national and

international issues and priorities of the Gov-

ernment of India at all levels. The programme

was tailored to the particular requirements of

the target groups, who were involved in its

planning and implementation from the very

beginning. All partner institutions and involved

intermediaries attached high priority to the

programme. With its priority set on the devel-

opment and piloting of innovative methods and

techniques, the programme was in line with the

sector-specific policies of the partner countries,

with the policy of the Government of India and

with that of the non-governmental organisa-

tions (NGOs) participating as intermediaries.

This includes both the BMZ country strategy

for India, in which 'watershed management' is

stated as a priority, and the GTZ sectoral ap-

proach for rural development. Moreover, it

addressed central concerns of the UN conven-

tion to combat desertification and strategies for

adapting to climate change. The programme

complied with generally established technolo-

gies and encompassed income-generating

measures on private and public property, so

that both property owners and the landless

population benefited. The multi-level approach

was rendered evident by interventions at the

national, regional and local levels.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 44

The 'Nile Water Initiative' (Africa suprare-

gional) also received a very good relevance

rating. As part of the Shared Vision Pro-

gramme of the Nile Basin Initiative on the inte-

grated development and administration of

shared water resources, the project helps ri-

parian states reform their national water poli-

cies and adjust them to the shared vision. The

project is in line with the MDGs and the BMZ

water sector strategy with its prioritisation of

IWRM, river basin management and trans-

boundary cooperation in Africa. The relevant

evaluation also comes to the positive conclu-

sion that the project pursues the GTZ strategy

for capacity development and the multi-level

approach in the multinational context. The

project concept is such that the effects on wa-

ter users in the Nile Basin (micro level) are

achieved via long results chains. The project's

strong presence at the macro and meso levels

is of decisive importance for its relevance.

Actually, the intended impacts on the target

group should have been described more

closely in the evaluation in order to better sub-

stantiate the rating of relevance as 'very good'.

The 'relevance' of the development meas-

ures in Yemen, Kenya and Zambia was rated

'good' in the interim evaluations. In the opinion

of the evaluators, however, a rating of 'very

good' would have been appropriate, since

those measures represent exemplary models

in terms of relevance:

The slightly downgraded rating for

'Institutional Development of the Water

Sector, Yemen' in the relevant evaluation

was founded on the premise that irrigated

agriculture, as a sub-sector of relevance

to water-resource problems, remained

unaccounted for in the priority area strat-

egy paper on the grounds of develop-

ment-policy directives. However, German

DC activities in this sub-sector would not

necessarily be expedient, because other

donors already cover this theme. Conse-

quently, that aspect should not have a

negative impact on the relevance rating.

With regard to the priorities of the lead

executing agency and implementing part-

ner and to conceptual embedment in the

country, as well as the sectoral and multi-

sectoral strategies of German DC, the

programme is highly relevant. It pursues a

successful approach to capacity devel-

opment and draws support from the suc-

cessful endeavours of women's groups. A

poverty study was conducted in advance

of programme building.

The 'Water Sector Reform Programme,

Kenya' is not only in accordance with Ke-

nyan water-sector policies and strategies,

but also with international themes and

standards. The programme enjoys high

priority at the lead executing agency, the

implementing organisations and the target

groups. It is suitable for solving central

development issues, contributes directly

toward improving the living conditions of

the poor population and is therefore in line

with the basic development-policy orienta-

tion of the GTZ. Its special focus on

gender equality is reflected by the water

kiosk approach. The entire reform pro-

gramme, with its water kiosk concept,

EcoSan concept and envisaged pro-poor

water rates structure, is geared to improv-

ing water services for the urban poor and

therefore has a strong relevance to pover-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 45

ty, even though no separate poverty anal-

ysis was conducted.

The 'Water Sector Reform Programme,

Zambia' is suitable for solving the coun-

try‟s central development issues because

it is aimed at improving the supply of wa-

ter and sanitation services, IWRM and

poverty reduction. It is in line not only with

national sector policies and strategies, but

also with international themes and stan-

dards as well as with the basic develop-

ment-policy orientation of the GTZ. The

programme's poverty relevance is illumi-

nated by its pro-poor orientation in peri-

urban rural areas. This includes the de-

velopment of pro-poor tariffs. The concept

is gender-responsive, and the responsible

institutions and target groups attach high

priority to the programme, as reflected by

the increased budget from the govern-

ment and by an increasing willingness to

pay for water and sanitation.

The 'Integrated Water Management Pro-

gramme, Algeria' stands as an example of a

programme with a poor relevance rating. As

analysed by the evaluation team, the measure

is in line with the sector policy and strategies of

the partner country, and it supports the impor-

tant process of reform in the Algerian water

sector, but the short-term supply of water – as

opposed to the sustainable management of

water resources – is seen as its most important

function. Accordingly, there is a conflict of ob-

jectives between the rather short-term orienta-

tion of the Algerian policy and the requirements

of sustainability pursuant to the IWRM ap-

proach in accordance with the BMZ sector

strategy. Consequently, the programme is not

in line with the basic development-policy orien-

tation of the client. Capacity development is

emphasised as a solution to the competency

deficits, although the core problem is a lack of

institutional mainstreaming with regard to sus-

tainability. Though they constitute part of the

target group, future generations in Algeria will

not benefit from the programme. Neither a

poverty analysis nor a gender analysis was

performed. While the partner does have a posi-

tive opinion of the programme's concept, the

majority of the key questions for assessing the

programme's 'relevance' could not be an-

swered positively.

In many evaluation reports, a general weak

point with regard to the relevance of projects

and programmes is stated as the fact that only

a small share of the interventions are situated

at the target group level and geared to directly

improving the living conditions of the poor

population (see also Section 2.2.2). In fact,

most projects lack the appropriate planning-

phase analyses of target groups, poverty and

the gender situation.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 46

3.1.2 Weighting of relevance

No clear-cut patterns emerge with regard to

the weighting of relevance in the individual

evaluation reports. The relevance criterion had

an average weighting factor of 2.3 (interim

evaluations 2.6, final evaluations 2.0, and ex-

post evaluations 2.0). In five of the evaluations,

the factor was upgraded (weighting factor 3;

see also Table 4) always in connection with

interim evaluations. This leads to the conclu-

sion that, either at or after the end of the

measures, more importance was attached to

criteria other than relevance, whereas the stra-

tegic orientation of the projects and pro-

grammes played a decisive role in the course

of implementation.

Main reasons for enhanced weighting of rele-

vance were, for example:

the fact that it is one of GTZ's largest wa-

ter programmes, therefore enjoys high

priority, and is characterised by good con-

ceptual embedment in the entire sector

(Yemen);

the objective of improving the supply of

drinking water to the urban population

while protecting the water resources

(Kenya);

the fact that the development measure

does not focus on solving problems of a

local nature, with limited content, instead

addressing all levels and pursuing a sys-

tem-changing capacity development ob-

jective (Peru);

the objective of improving the supply of

drinking water to the urban population, in

combination with the fact that the water

sector enjoys priority in the Zambian na-

tional development plan, and that Ger-

many is the lead donor in the water sector

(Zambia).

In the case of the 'Integrated Water Man-

agement Programme, Algeria', the higher

weighting of relevance was not entirely under-

standable, and no corresponding substantia-

tion was evident.

In the above context, three of the pro-

jects/programmes were conspicuous in that

their relevance was given higher weightings,

but their relevance rating was actually too low.

Apparently, the evaluators' analysis is marred

by major uncertainty in the use of the available

tools.

3.2 Effectiveness

3.2.1 Effectiveness rating

Effectiveness is seen as the extent to which

the direct results (objectives) of the develop-

ment measure are being achieved (comparison

of actual situation with targets).

'Are we achieving the objectives of the devel-

opment measure?'

On the whole, the evaluated projects and pro-

grammes achieved an average effectiveness

rating of 2.5 ('good'). In more than half of the

projects and programmes good achievement of

objectives was established (9). Five others

performed satisfactorily, and only one project

(Niger) was rated 'unsatisfactory' in terms of

achieving objectives, based on comparison of

the actual situation with targets (see also Table

5 and Figure 6).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 47

The average effectiveness rating varies ac-

cording to the timing of the evaluation: In the

interim and final evaluations, the average rat-

ing was 2.3, dropping to 3.0 in the ex-post

evaluations. Apparently, the degree to which

the objectives are achieved is interpreted

somewhat more critically with increasing 'dis-

tance' to the actual implementation.

Table 5: Effectiveness ratings of evaluated projects/programmes

Country Project/Programme Type of

Evaluation Rating Weighting

Africa su-

praregional NWI

Interim

evaluations

2 2

Algeria PDEA 3 3

Yemen WSP 2 2

Jordan WMIA 2 2

Kenya WSRP-KEN 2 2

Peru PROAGUA 2 3

Zambia WSRP-ZMB 2 2

Viet Nam WDPC 3 2

Ghana EVORAP Final

evaluations

2 2

Kosovo WSWDP 2 2

Mexico GICA 3 3

Ethiopia IWSTR

Ex-post

evaluations

3 2

India IGWDP 2 2

Niger PHV/MI 4 2

Turkey QADI 3 2

Overall (average) 2.5 2.2

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 48

Figure 6: Distribution of effectiveness ratings

Success factors for improving effectiveness,

i.e. achievement of objectives:

Development measures were given a 'positive'

rating if

the project/programme partners and GTZ

programme management all share a

common understanding of the objectives

and cooperation;

the target group understands and appre-

ciates the benefits of the measure and

exercises its rights and obligations;

the national authorities accept revised

policies and regulations;

guidelines are generated and best prac-

tices disseminated and recognised by the

partner structure;

'meso-level environment management' is

formally mainstreamed in the organisa-

tional structure and implemented at the

micro level in the form of such concrete

measures as hygiene education, informa-

tion campaigns, customer surveys and/or

focus-group discussions;

capacity development reflects the quality

and coherence of the training and capac-

ity-building package , and systematic as-

certainment of needs enables better coor-

dination of supply and demand;

TC and FC are optimally intermeshed in

cooperative projects and programmes, i.e.

synergy effects are generated and the re-

sults of TC measures can be multiplied by

FC investments, and vice-versa.

The effectiveness of the 'Water Sector Re-

form Programme, Kenya' was rated 'good'. A

comparison of the actually achieved indicators

with the targets shows that all four overall-

objective indicators were partially achieved and

that full achievement within the programme

term (2013) is probable, since the overall de-

velopment is demonstrably positive. At the time

of evaluation, 6 of 20 component indicators

had already been fulfilled. Since the pro-

gramme began, sector coordination by the

ministry (component 1), regulation by the regu-

latory authority and pro-poor financing by the

trust fund (component 2) improved. Commer-

cialisation of WUs is being broadly imple-

mented (component 3). Strengthening of per-

formance capabilities at the WRM authority is

reflected in an improved contribution to sector

regulation and management (component 4).

0%

60%

33%

7%

very good

good

satisfactory

not satisfactory

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 49

Initial EcoSan pilot measures were introduced

(component 5).

As explained in the preceding Section 2.2.1,

the reviewed objectives systems clearly at-

tached priority to the results constellation

'drinking water supply – standards of supply –

operational efficiency'. All in all, approximately

one half of all indicator formulations relate to

that results constellation. In 'Rural Water

Supply in Maradi, Niger', for example, 6 of

the 7 formulated objectives-level indicators

relate to supply standards (supply coverage

rate, minimum supply volume, water quality)

and operation (maintenance/repair, water-well

hygiene, prevention of overuse).

The results constellation 'sector policy - regula-

tion - governance' is found in the indicator sys-

tems of only two projects and programmes

(Africa supraregional and Algeria). Conse-

quently, the 'recorded' direct results of the

evaluated projects and programmes were pri-

marily achieved as technical improvements

and improvements in the standards of supply,

and less so in terms of institutional and legal

change processes and the creation of sustain-

able institutional capacities. In addition, it

proves enormously difficult to document indica-

tors which were formulated first and foremost

for the policy-making area, if these are impre-

cisely worded and excessively ambitious, and

have no follow-up documentation available. In

the interim evaluation report on the 'Integrated

Water Management Programme, Algeria',

for example, no plausible contributions regard-

ing the indicators for the areas of sector devel-

opment planning, investment planning, regula-

tion and water resources policy planning were

documented.

The recording of qualitative aspects (changes

in behaviour, quality, satisfaction, etc.) in the

achievement of objectives is still a weak point.

Relevant results are usually 'assumed' or 'an-

ticipated' instead of substantiated. This point

has been criticised by numerous evaluators

conducting individual evaluations, especially

because the projects and programmes are

widely assumed to have more results than are

actually being recorded. Such assumptions

derive mainly from random interviews with

target-group representatives, which are held in

connection with all such evaluations and are

accordingly interpreted by the respective

evaluators.

Since only a very small fraction of the indica-

tors formulated for the evaluated projects and

programmes relate to the recording of con-

crete, results-based outcomes (e.g. supply

situation, poverty reduction, health), it is rec-

ommended that, in future, more indicators be

formulated and more appropriately designed

studies and documentations be prescribed,

e.g. in reporting and in advance of scheduled

evaluations. It is also important in this connec-

tion that the appropriate indicators be docu-

mented with concrete facts, figures and data. A

number of best practices cited in the current

set of evaluated projects and programmes are

available to that end.

Factors contributing to failure reduce the effec-

tiveness of development measures: In the

evaluations under review, the achievement of

objectives received a 'negative' rating if, for

example:

the structural context of the sector im-

pedes sustainable capacity development;

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 50

a number of different but overlapping as-

sistance programmes undermine the

stipulations (conditionalities) of the donor

countries;

delayed outflow of FC funds in connection

with cooperative projects and pro-

grammes makes the shared TC/FC indi-

cators unachievable;

adjustment and further development of

the technical conceptual design are ne-

glected during the implementation phase;

further education/training and subsequent

follow-up assistance for water user asso-

ciations by the responsible authorities is

neglected;

rapid population growth is not accounted

for in the design of the measure or in its

formulated objectives;

conflicts within the GTZ team cause de-

lays in implementation and substantially

disrupt the working relationships;

internal conflicts have a negative influ-

ence on the partner, and partner(s) have

a negative influence on the measure.

Some good examples of documented objectives achievement and results:

In the 'Eastern and Volta Region Assistance Programme, Ghana', the establishment and intro-

duction of an integrated database (InfoSys) for continuous data acquisition and monitoring in

the water sector was facilitated and a uniform monitoring system established (MOM – 'Mainte-

nance – Operation - Monitoring'). These instruments have since proven comparatively effective

for use in monitoring the sustainable operation of water supply systems and have been nomi-

nated as best practice for countrywide introduction. Additionally, the programme regularly re-

ceived statistics on the occurrence of water-induced diseases such as typhus, bilharziasis,

dracunculosis/Guinea worm, malaria and diarrhoea from hospitals in the project region. Those

data were then analysed to detect effects on the health & hygiene situation.

In the 'Drinking Water and Sanitary Programme, Peru', a strategic planning and management

procedure (MPPI) was instituted at the participating WUs. A complementary monitoring system

(SIME) was developed and a knowledge-management procedure established within the com-

plex institutional structure. The corresponding instruments have enabled routine data availability

and dissemination. They have since been implemented as best practices in neighbouring coun-

tries, too. Through the GTZ knowledge-management structures, the instruments have been

made available to a broad audience.

The 'WMIA Jordan' project, for example, was

geared to reducing groundwater extraction

(indicator: component objective). However, no

fresh data of relevance are available. While

some information is available concerning re-

duced abstraction rates from the aquifer in

question, the extent to which this is attributable

to the project remains unclear (measurability

and attribution problem).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 51

The 'Rural Water Supply in Maradi, Niger'

project was an example of failure. The effec-

tiveness of this development measure was

rated as 'unsatisfactory'. Only four of its eight

indicators were fully or partially met. The main

reason for the negative rating was the lack of

durable self-administration structures and the

resultant poor operating standards. At the end

of the project, water committees were manag-

ing the operation of about 85% of all the con-

structed wells (indicator: objectives level). At

the time of ex-post evaluation in 2008, 90% of

the water user committees had ceased their

activities. In 2003, women accounted for 50%

of all water-well committee members (indicator:

objectives level). By 2008, that was true of only

10% of the still-active committees. Since most

(with a few exceptions) of the water user com-

mittees who had been taught how to keep the

well environs clean and the extracted water

hygienic, have discontinued their activities, the

well systems are no longer being kept clean or

free of cattle (indicator: objectives level). In this

project, important coordination processes be-

tween TC and FC have failed to materialise, as

has conceptual adaptation regarding a stricter

focus on capacity development among the

target group and intermediary institutions.

For the 'Integrated Water Management Pro-

gramme, Algeria', the effectiveness of the

development measure was given a 'satisfac-

tory' rating based on comparison of the actual

situation with targets. However, on the as-

sumption that existing cooperation problems

will be eliminated, only 3 of 12 indicators were

identified in the evaluation report as being

'potentially achievable'. At the time of the

evaluation, i.e. six years into the measure, not

a single indicator could be classified as

'achieved'. Indeed, the achievability of 8 of the

12 indicators was assessed as 'questionable'

or 'hardly probable'. Consequently, the corre-

sponding rating is basically inconsistent, and

the programme should be regarded as an ex-

ample of failure.

The level of funds provided by German TC

appears to have very little influence on the

effectiveness of the projects and programmes.

No corresponding statistical connection can be

ascertained on a rank correlation basis. There

does, however, appear to be a closer link be-

tween the duration of implementation and the

achievement of objectives. Simultaneously, the

effectiveness of projects and programmes

appears to exert substantial influence on both

the impact and the sustainability of the meas-

ures (see also Table 6).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 52

Table 6: Effectiveness correlations

Characteristic

Correlation

value/

Rank correlation

Funding level (overall) – effectiveness - 0.03

Funding level (per month of implementation) –

effectiveness + 0.06

Duration of implementation – effectiveness - 0.38

Effectiveness – impact + 0.53

Effectiveness - sustainability + 0.67

3.2.2 Weighting of effectiveness

Similarly to the weighting of relevance, there

are no clear patterns to be seen with regard to

the weighting of effectiveness in the individual

evaluation reports. This criterion was weighted

with an average factor of 2.2 (interim evalua-

tions 2.3, final evaluations 2.3, ex-post evalua-

tions 2.0), which was upgraded in three

evaluations (weighting factor 3; see also Table

5). This was the case in two interim evalua-

tions and one final evaluation.

In the opinion of the evaluators, it is surprising

that the actual achievement of objectives on

the basis of the formulated objectives systems

did not enjoy greater importance in the as-

sessment of projects and programmes. Basi-

cally, one would expect final and ex-post

evaluations to rate the achievement of objec-

tives, along with sustainability, as the most

crucial characteristic. At the same time, closer

analysis of the observed objectives achieve-

ment trends enables quicker identification of

potentially required conceptual adaptations,

especially in interim evaluations key question:

'Why will the objectives probably not be

achieved?').

Principal reasons for stronger weighting of

effectiveness in the present evaluations in-

cluded, for example:

implementation of the measures is objec-

tives-oriented and in shared understand-

ing with the partner (Algeria);

effectiveness is a central criterion for final

evaluation (Mexico);

it is of decisive importance that the essen-

tial results of the measure be evident in

the interim evaluation after completion of

two phases, i.e. after nearly six years, so

that they can be consolidated during the

last phase of the programme (Mexico).

3.3 Overarching development re-

sults (impact)

3.3.1 Assessment of overarching

development results (impact)

Overarching development results (impact)

relate to the 'extent to which the project or

programme helps to achieve the planned over-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 53

arching objectives, and other indirect results

occur‟.

Are we contributing to the achievement of

overarching development objectives and re-

sults?

The evaluated projects and programmes

achieve an average impact rating of 2.7 ('satis-

factory'). While two thirds of the projects and

programmes (10) were found to have made

satisfactory contributions to the achievement of

overarching development results, one pro-

gramme (Yemen) received a 'very good' rating,

and two projects/programmes were rated

'good'. Only a single project (Turkey) received

an 'unsatisfactory' rating for its contribution to

the achievement of indirect results (see also

Table 7 and Figure 7).

The average impact rating again varies accord-

ing to the point in time of the evaluations: The

average ratings in the interim and final evalua-

tions were situated at 2.5 and 2.7, respectively,

falling to a poorer 3.3 on average in the ex-

post evaluations. As with the effectiveness

rating, it would appear that here, too, the

achievement of indirect results is interpreted all

the more critically with the passing of time

since implementation. Particularly with regard

to ex-post evaluations, this is understandable,

because once several years have passed

since a measure's completion, an appropriate

review can and should be performed.

It becomes apparent in the analysis of evalua-

tion reports, especially of interim and final

evaluations, that it is hardly possible to record

the results of the projects/programmes down-

stream of their 'attribution gap' and that little

attention is devoted to that area. With regard to

impact, the evaluation reports only offer either

conjecture on results constellations or as-

sessments of conditions that have not yet oc-

curred. Often, it is argued that the impact will

only emerge after a considerable length of

time, so no further investigations are possible.

In the opinion of the evaluators, however, the

projects/programmes could very well attempt

to substantiate indirect results when, for exam-

ple, concrete measures are implemented at the

intermediary and target-group levels. A typical

example of such a case would be the system-

atic collection and documentation of data on

water quality and the occurrence of water-

induced diseases in the course of implementa-

tion (Ghana). This could yield information on

results trends. Also, random surveys can be

conducted among the target groups at the

beginning and during implementation to shed

light on qualitative improvements in vital natu-

7%20%

67%

7%

very good

good

satisfactory

not satisfactory

Figure 7: Distribution of overarching development results rating

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 54

ral resources and altered behaviour (Yemen).

Such surveys can likewise provide important

insights into such matters as how changes in

the range of services offered by utility compa-

nies are perceived (Peru). In an ideal case, this

could be founded on the establishment of an

appropriate baseline.

Table 7: Rating of overarching development results of the evaluated projects/programmes

Country Project/Programme Type of

evaluation Rating Weighting

Africa su-

praregional NWI

Interim

evaluations

2 2

Algeria PDEA 3 2

Yemen WSP 1 2

Jordan WMIA 3 2

Kenya WSRP-KEN 2 2

Peru PROAGUA 3 2

Zambia WSRP-ZMB 3 2

Viet Nam WDPC 3 1

Ghana EVORAP Final

evaluations

2 3

Kosovo WSWDP 3 2

Mexico GICA 3 2

Ethiopia IWSTR

Ex-post

Evaluations

3 3

India IGWDP 3 2

Niger PHV/MI 3 2

Turkey QADI 4 2

Overall (average) 2.7 2.1

Success factors contributing to enhancement

of the overarching development results can be

summarised as follows on the basis of the

results of the reviewed evaluations: Develop-

ment measures are given a 'positive' rating if

the measure's contribution to the over-

arching result can be plausibly substanti-

ated, and the formulation of the overarch-

ing result leaves no room for interpreta-

tion;

micro-level results extending beyond the

target group occur, e.g. the founding of a

successful NGO;

structural impacts, e.g. the decentralisa-

tion of WS&S and the advancement of

sector reform beyond the level of objec-

tives achievement, are supported;

the overarching results are documented

not only by qualitative statements, but

also by quantitative statements founded

on a baseline study;

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 55

a contribution toward achievement of the

MDGs is made, and target group surveys

confirm a qualitative improvement of the

situation;

a role-model function emerges, and the

imparted approaches and concepts be-

come common property;

the negotiating positions in political dia-

logue and the ability to solve problems on

the part of both the target group and the

stakeholders are strengthened.

The impact of 'Institutional Development of

the Water Sector, Yemen' was rated 'very

good'. The objective of this programme is to

improve the health situation of the poor popula-

tion while securing an important part of the

target group's natural resource base. No quan-

titative statements can be made in that re-

spect, because no corresponding baseline

study is available. However, real qualitative

improvements are identified by way of informa-

tive surveys. For example, the fact that female

village multipliers have joined together to es-

tablish an NGO was regarded as positive. The

programme's structural impact above and be-

yond the objectives level, though, was decisive

for the 'very good' rating. Throughout the coun-

try, there has been a paradigm shift in the wa-

ter sector. The approaches and concepts

communicated by the reform policy have be-

come common property, with consequential

positive effects in other areas, too (e.g. the

decentralisation of administrative structures).

In that respect, the contribution made by the

development measure was definitely demon-

strable.

Factors determining failure with potential influ-

ence on impact can be summarised as follows:

Development measures are rated 'negative' if

the aggregated results are rendered un-

realistic by the programme (excessively

high expectations, very long results

chains) and can therefore only be as-

sumed ('perceived');

no verifiable contribution toward poverty

reduction is made;

there are no positive effects on other parts

of the broader sectoral and regional con-

text;

there are no broad-based effects from, for

example, model character, and the gen-

erated work results are only utilised by

people who were involved in their crea-

tion.

The impact of the 'Upgrading of Municipal

Services, Turkey' project, for example, was

rated as 'unsatisfactory'. Overarching results

had been expected with regard to better con-

servation of natural resources, safe access to

clean drinking water and greater participation.

The project failed to develop broad-based ef-

fectiveness, because the generated working

aids were only utilised by a few individuals

(presumably those who had been involved in

their creation). The low number of training

measures implemented in the municipalities

was cited as another reason for the poor rat-

ing. In addition to the existing financing con-

straints, the development of municipal infra-

structure is also perceived as constrained (an

argument that would also have been of rele-

vance for the effectiveness rating). The project

is only known to a small minority of municipali-

ties, and even in the pilot communities it has

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 56

not been possible to influence the quality of

municipal services. Note: This project is the

one with the shortest term. The term was

shortened after the project began, and the

project came to an abrupt end.

3.3.2 Weighting of overarching de-

velopment results (impact)

The impact of the projects and programmes

was weighted with an average factor of 2.1.

Again, there are minor differences between the

weighting assigned in the performed interim

evaluations (1.9) and that found in the final and

ex-post evaluations (2.3 in each case). In one

final and ex-post evaluation each, the impact

was weighted with a higher factor of 3, and in

one interim evaluation it was downgraded to

the factor 1 (see also Table 7).

Overall, then, major importance was ascribed

to the indirect results of projects and pro-

grammes. While these results constellations

frequently had not yet become identifiable at

the time of the interim and ex-post evaluations

(especially in the case of long results chains),

the evaluators believe that more importance

should have been attached to this criterion in

the ex-post evaluations, because indirect re-

sults do make real contributions to qualitative

performance assessment.

One explorable hypothesis in favour of not

attaching more importance to impact in evalua-

tions would be that, with regard to indirect re-

sults, the results chains of the TC projects and

programmes often have very long, abstract

formulations, and that many results are pro-

jected into a very highly aggregated area. Such

links are difficult to record, and robust data are

rarely available. Moreover, independent inves-

tigations of these are very complicated.

Principal reasons for altering the weighting of

impact in the evaluations:

upgrading: This criterion is of decisive

importance for the project or programme,

particularly with regard to institution de-

velopment and to projects involving ca-

pacity development (Ethiopia);

upgrading: Since the indirect results have

to do with improving the people's health

and hygiene situation and with generally

improving their quality of life while con-

tributing to the provision of basic services,

there is a chance that an especially large

number of people can benefit from the

positive results. An improved quality of life

for target groups should find expression in

this category (Ghana);

downgrading: Due to delays in the imple-

mentation of the first phase, most of which

could not be influenced, the TC contribu-

tion had not yet had a chance to provide

the full scope of planned outputs or,

hence, to achieve the anticipated results

(Viet Nam).

3.4 Efficiency

3.4.1 Efficiency rating

Efficiency is a measure of the degree to which

the resources invested in a development

measure are appropriate compared to the out-

puts and results achieved.

'Are the objectives being achieved cost-

effectively?'

The evaluated projects and programmes

achieved an average efficiency rating of 2.8

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 57

('satisfactory'). Six of the projects/programmes

were rated 'good' or 'satisfactory', while three

others (Ethiopia, Algeria and Niger) received

an efficiency rating of 'unsatisfactory' (see also

Table 8 and Figure 8). The projects and pro-

grammes examined in ex-post evaluation tend

to have been rated somewhat lower with re-

gard to the cost efficiency of their outputs.

Here, the average rating was 3.5, that is, an

entire point lower than the average rating (2.5)

of the projects and programmes in the interim

evaluations. The average efficiency of the pro-

jects and programmes in the final evaluations

was 2.7.

The relatively large discrepancy between the

efficiency ratings found in the ex-post evalua-

tions and those cited in the interim and final

evaluations is certainly also due to the fact

that, when evaluation is being performed dur-

ing or even at the end of an implementation

period, relatively little information is available

with regard to the specific level of funding (es-

pecially for investments: funds employed per

user) and about the actual achievement of

objectives. At the same time, for the purposes

of interim evaluation, less thought tends to be

given to alternative forms of service delivery

than to optimising the existing conceptual de-

sign.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 58

Table 8: Efficiency ratings of the evaluated projects/programmes

Country Project/Programme Type of

Evaluation Rating Weighting

Africa su-

praregional NWI

Interim

Evaluations

3 2

Algeria PDEA 4 3

Yemen WSP 2 2

Jordan WMIA 2 2

Kenya WSRP-KEN 2 2

Peru PROAGUA 2 2

Zambia WSRP-ZMB 2 2

Viet Nam WDPC 3 2

Ghana EVORAP Final

Evaluations

3 3

Kosovo WSWDP 2 2

Mexico GICA 3 2

Ethiopia IWSTR

Ex-post

Evaluations

4 1

India IGWDP 3 1

Niger PHV/MI 4 2

Turkey QADI 3 2

Overall (average) 2,8 2,0

Figure 8: Distribution of efficiency ratings

There appears to be no link between the level

of funding (total funds employed per month of

implementation) and the respective efficiency

rating. In the IWSTR Ethiopia project, for ex-

ample, the average monthly level of funding

was EUR 30,526, while EUR 97,980 was spent

0%40%

40%

20%

very good

good

satisfactory

not satisfactory

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 59

per month in PDEA Algeria and EUR 187,850

in PHV/MI Niger. The efficiency of those pro-

jects/programmes, however, was rated 'unsat-

isfactory'. On the other hand, both the WSP

Yemen programme and the WSRP Kenya

programme received 'good' efficiency ratings

despite their high levels of funding per month

of implementation (EUR 118,421 and EUR

113,008, respectively; see also Table 9).

Table 9: Comparison of term, level of funding and efficiency ratings

Country Project/Programme

Term

[months]

Level of funding

Rating Total

per

month

Yemen WSP 114 €13.5 M €118,421

'good'

Jordan WMIA 111 €2.5 M €22,523

Kenya WSRP-KEN 123 €13.9 M €113,008

Kosovo WSWDP 108 €7.7 M €71,296

Peru PROAGUA 156 €8.2 M €52,564

Zambia WSRP-ZMB 108 €8.1 M €75,000

Africa su-

praregional NWI

120 €10.0 M €83,333

'satisfactory'

Ghana EVORAP 126 €6.2 M €49,206

India IGWDP 183 €12.8 M €69,945

Mexico GICA 70 €2.4 M €34,286

Turkey QADI 50 €1.7 M €34,000

Viet Nam WDPC 107 €8.8 M €82,243

Ethiopia IWSTR 95 €2.9 M €30,626

'unsatisfactory' Algeria PDEA 99 €9.7 M €97,980

Niger PHV/MI 107 €20.1 M €187,850

The link between project term and efficiency

does not appear to be very distinct. Statisti-

cally, relevance also appears to have no plau-

sibly verifiable effect on efficiency. Viewed

against the backdrop of the factors of success

and failure discussed below as applied to effi-

ciency, however, some degree of correlation

should be presumed. Thus, not only the devel-

opment-policy alignment with sector and coun-

try strategies is decisive, but also the coordina-

tion actually performed with other do-

nors/actors (e.g. on the basis of a cooperation

agreement) and adjustment of the envisaged

interventions (e.g. on the basis of shared plan-

ning).

In connection with the reviewed evaluations, a

number of success factors were identified with

the capacity to enhance efficiency. Accord-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 60

ingly, a development measure was given a

'positive' rating if

coordination takes place between donors,

and the fields of endeavour are mutually

complementary;

coordination takes place with other Ger-

man DC measures, and potentials for fur-

ther cooperation are utilised;

synergies between TC and FC are sys-

tematically exploited in a cooperative pro-

ject/programme;

the personnel concept is convincing, i.e.

internal and external, short-term and long-

term advisory services (including those

rendered by national evaluators) are har-

monised, and synergies are fostered by

well-developed knowledge management;

necessary adjustments in the timing of

phases are coordinated with BMZ in due

time;

synergies result from interventions at dif-

ferent levels.

The efficiency of the 'Water Sector Reform

Programme, Zambia' was rated 'good'. The

programme is characterised by a high level of

donor coordination, which involves not only the

German donors but also the international do-

nor community and NGOs in the sector reform

programme. Promotion of sector development

was pursued by splitting the responsibilities

between the donors and identifying special

fields. German and Danish DC, for example,

assumed leadership in the field of IWRM. The

programme supports such programme-based

approaches as SWAp and joint financing

measures. Its high level of cost efficiency is

particularly apparent in the financing of water

kiosks, which make a major contribution to-

ward the achievement of objectives (invest-

ments amounting to EUR 8 per person). Only

the fact that the outputs geared to drafting the

relevant legislation for the water sector had not

brought the desired success gives rise to a

minor downward adjustment.

Nevertheless, some of the evaluations recog-

nised a few factors determining failure that

reduced efficiency. Accordingly, a development

measure was given a 'negative' rating if

its objectives, outputs and activities are

neither coordinated with those of other

projects and programmes nor designed

for task-sharing (joined-up development

cooperation, SWAp and basket funding);

consensus-building and coordination of

resource inputs between the various

German DC organisations is either inade-

quate or unmanaged;

the partner shows too little willingness to

implement;

there is too little exchange between the

programme management and the partner;

individual/personal interests impede co-

operation between donors;

the advisory approach is faulty and not

results-based;

the project‟s/programme‟s own resources

(personnel, material and equipment, etc.)

are used inefficiently, and antici-

pated/unforeseen additional costs accrue;

the complexity of the personnel concept

has a negative influence on the cost/effect

ratio, e.g. too many experts not only cost

more, but even have counterproductive

effects;

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 61

the continuity of the measure is inter-

rupted;

investments are lacking, e.g. when the

allocation of FC funds is delayed in a co-

operative project or programme.

The efficiency of the 'Rural Water Supply in

Maradi, Niger' project was rated 'unsatisfac-

tory'. The average cost of investment per

newly constructed or rehabilitated water well

(approx. EUR 16,000) increased two-fold to

roughly EUR 33,000 per well, if the consulting

and advisory services are also counted. Con-

sequently, the outlays for advisory services are

exaggerated in comparison with the achieved

results. With regard to the requisite advisory

and awareness-raising measures for the civil-

ian population, it would have been expedient to

cooperate with NGOs, but that was not the

case. Differences of opinion between the FC

and TC experts gave rise to problems affecting

the synergetic interaction and coordination of

the project structure. Experience drawn from

the implementation phase was not utilised for

refining the approaches and technical models.

In addition, the operation of parallel project

infrastructures for TC and FC caused addi-

tional costs.

The 'Improvement of Water Supply in the

Tigray Region, Ethiopia' project was also

given an 'unsatisfactory' efficiency rating. The

project began with a study of the water sector

in Tigray. That study was too general in nature

and of average quality. Manuals developed on

the basis of that study proved to be of limited

use and rather impractical. Overall, the first

implementation phase was deemed inefficient,

partially because the implementation partner

had difficulty fulfilling its obligations. At the

same time, the commissioned consulting firm

had problems delivering its services. Due to

domestic-policy difficulties, implementation

was interrupted for approximately three years

between the two phases. Much data and in-

formation was lost during that period. A requi-

site tariff study was implemented much too

late, i.e. in April 2004, and neither cooperation

with similar projects in the sector nor an ex-

change of knowledge took place.

Nearly all evaluation reports speak of a definite

and general need for improvement when it

comes to the actual mainstreaming of meas-

ures in the respective sector interventions. In

many places, it was noted that the DC inter-

ventions of the various actors were not opti-

mally intermeshed despite established coordi-

nation mechanisms. This was attributable, on

the one hand, to delimitation problems be-

tween the DC actors' respective interventions

and on the other, to the fact that the coordina-

tion mechanisms are often still of an essentially

informal nature, i.e. not anchored in concrete

agreements. Hence, the coordination process

frequently never proceeds beyond an ex-

change of information.

Regarding modes of delivery, it must be noted

that very few of the evaluation reports look into

the concept of service delivery. This is surpris-

ing in that, in 13 of 15 projects and pro-

grammes, not only GTZ personnel but also

consulting firms and NGOs were commis-

sioned to provide services. Consequently, it

would hardly be possible to perform a consis-

tent analysis in that respect. The following two

aspects regarding efficiency were mentioned in

a small number of evaluations:

the importance of qualified national

evaluators and/or consulting firms and

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 62

NGOs for capacity building and sector-

specific advisory services and training as

a potentially more efficient alternative to

the deployment of seconded experts and

national staff (Niger, Peru);

the design of external expert assignments

which was at times still suboptimal due to

substantial commitment of funds, prob-

lems with integration into TC structures,

management and flow of information, in

addition to qualifications sometimes re-

garded as inadequate, primarily with re-

gard to language proficiency (Algeria,

Yemen).

3.4.2 Weighting of efficiency

On the whole, the efficiency of the evaluated

projects and programmes was the criterion

given the lowest weighting, i.e. 2.0 on average.

There were conspicuous differences between

interim evaluations (average weighting 2.1),

final evaluations (2.3) and ex-post evaluations

(1.5). In two of the four ex-post evaluations, the

cost efficiency of objectives achievement was

classed as less important (weighting factor 1),

while that criterion was weighted more strongly

in one interim evaluation and one final evalua-

tion (see also Table 8).

Main reasons for changes effected in the

weighting of efficiency:

upgrading: The criterion is amenable to

direct influence by GTZ and the pro-

gramme's management (Algeria);

upgrading: The establishment of basic

principles for effective monitoring sys-

tems, the organisation and training of self-

help organisations and the significance of

institutional interaction can be highlighted

with the help of this criterion. In the con-

ceptual sense, these are crucial elements

of the measure (Ghana);

downgrading: If impact and sustainability

are both good, it is of lesser importance

whether or not the activities were per-

formed inefficiently. Even when activities

are performed with extreme efficiency, the

result cannot be satisfactory in combina-

tion with poor sustainability (Ethiopia);

downgrading: There is little room for ex-

erting influence on the input-output ratios,

total expenditures and the nature of out-

puts from intermediary organisations (In-

dia).

In the opinion of the evaluators, the arguments

cited in favour of giving more weight to effi-

ciency as a criterion are inconsistent. It is deci-

sive that the employed resources, the modes

of delivery and the corresponding outputs and

results be documented. Wherever relatively

high effort is required for establishing capaci-

ties or for catalysing processes of change and

bringing about improvements in quality (e.g.

water supply), the question arises as to the

cost efficiency of the employed instruments

and of potentially available, more efficient al-

ternatives – not, however, as to their strategic

importance or the extent to which they can be

influenced.

In the view of the evaluators, the reasoning

behind the effected downgrading of efficiency

must be seen in connection with unclear or

ineffective control mechanisms.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 63

3.5 Sustainability

3.5.1 Sustainability rating

Sustainability is a measure of the probability

that the positive results of the development

measure will continue beyond the end of assis-

tance.

'Are the positive results durable?'

Most of the evaluated development measures

were given positive ratings, i.e. deemed suc-

cessful, in terms of sustainability. The average

rating here was 2.9 and sustainability therefore

the DAC criterion with the poorest average

rating. Nine of the projects and programmes

were classed as 'satisfactory' and four as

'good'. Two of the projects/programmes

(Ethiopia and Niger) received an unsatisfactory

sustainability rating (see also Table 10 and

Figure 9).

Experience shows that the assessment of sus-

tainability tends to fall in the course of a project

cycle. The reviewed evaluation sequence was

also marked by a gradual deterioration of this

criterion's ratings: from 2.7 in the interim

evaluations to 3.0 in the final evaluations and

3.3 in the ex-post evaluations. Particularly at

the 'capacity development' results level, it often

takes until the end of implementation or even

longer for the structures to show whether they

will keep on developing and utilising the TC

outputs in the intended interest of achieving

objectives following completion of the project

or programme. At that point, sustainability as-

pects are no longer 'anticipated' but actually

recorded.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 64

Table 10: Sustainability ratings of the evaluated projects/programmes

Country Project/Programme Type of

evaluation Rating Weighting

Africa su-

praregional NWI

Interim

evaluations

2 2

Algeria PDEA 3 3

Yemen WSP 3 3

Jordan WMIA 2 2

Kenya WSRP-KEN 3 2

Peru PROAGUA 2 2

Zambia WSRP-ZMB 3 2

Viet Nam WDPC 3 1

Ghana EVORAP Final

evaluations

3 2

Kosovo WSWDP 3 3

Mexico GICA 3 3

Ethiopia IWSTR

Ex-post

evaluations

4 3

India IGWDP 2 3

Niger PHV/MI 4 2

Turkey QADI 3 2

Total (average) 2.9 2.3

Figure 9: Distribution of sustainability ratings

0%27%

60%

13%

very good

good

satisfactory

not sufficient

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 65

For the sustainability criterion, too, the per-

formed evaluations identified a number of suc-

cess factors with the capacity to enhance sus-

tainability. A development measure is consid-

ered sustainable if

sustainability is deemed important in the

course of implementation, and all (eco-

nomic, social, political and ecological) ob-

jectives dimensions are given timely, well-

balanced consideration at the design

stage;

participatory structures are established at

the target-group level (micro level) to en-

able participation in dialogue on the use of

water as a resource;

structural risks in further interventions are

identified;

the lead executing agency shows accep-

tance, and the basic instruments and

methods for the further development of

partner structures are created and inter-

nalised within those structures;

the aims and content of the existing plan-

ning remain intact despite possible

changes in responsibilities, objectives and

content;

sufficient personnel is trained in and for

the sector;

follow-up assistance is provided (espe-

cially for FC participation);

WS&S is recognised as a service and in-

tegrated into the day-to-day IWRM ap-

proach;

examples not only of 'best practice' but

also of 'lessons learned' are compiled and

disseminated.

The sustainability of the 'Nile Water Initiative,

Africa supraregional' was rated 'good'.

Through extensive training, the staff of the

water ministries in the member countries were

taught how to revise and adapt national water

policies as necessary. Water directives were

drawn up and 'best practice' examples com-

piled. The ecological sustainability of the

measure was ensured by incorporating the

recommendations on regulation into the na-

tional legislation.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 66

A good example of taking into account the objectives dimensions of sustainability:

In the 'Institutional Development of the Water Sector, Yemen' programme, the four objectives

dimensions (political, economic, social and ecological) are equitably represented. The economic

dimension is reflected in the commercialisation of water both as a prerequisite for economic de-

velopment and as a commodity. Through such aspects as pro-poor water policy and tariffs, con-

servation of resources for subsequent generations, and access to clean drinking water, the so-

cial dimension plays an important role for all concerned. The programme accounts for the eco-

logical target dimension in the areas of resource conservation, efficient use of water and curbing

the overuse of water resources. All these objectives dimensions are interlinked in the pro-

gramme's five components. The programme links together the various sub-sectors of the water

sector, promotes political dialogue and is therefore able to address and moderate rival objec-

tives dimensions. The programme's broad partner spectrum and its cooperation and alliances

with other donors and sectors promote support for sustainable development in the water sector

in all four objectives dimensions.

Factors determining failure with negative

effects on sustainability can also be formu-

lated for that criterion on the basis of indi-

vidual evaluations. Accordingly, a project

or programme is considered less sustain-

able if

the lead executing agency displays lit-

tle willingness to accept advice in con-

nection with higher-level political ob-

jectives;

implementation partners hesitate to

implement new laws and regulations,

e.g. by introducing break-even tariffs

and providing funds for the operation

and maintenance of WS&S;

inefficient overlapping of institutional

tasks is not accounted for;

the target group attaches no priority to

the measures, e.g. to the introduction

of a management system (software) as

opposed to the construction of infra-

structure (hardware) as a visible sign

of progress;

the conceptual approach lacks sys-

tematic capacity development compo-

nents, and too little attention is paid to

operating and self-administration struc-

tures;

retention of the value of investments is

systematically neglected;

heavy subsidisation of the water sector

has distorting effects;

ecological effects are not taken into

account.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 67

Statistical analysis of the relationship between

the duration of implementation and the sus-

tainability rating reveals strong interdepend-

ence – even more so than the dependence of

sustainability on the volume of employed funds

(see also Table 11). This result shows how

important the duration of advisory services

actually is, especially against the backdrop of

specific results levels in German TC (priority

area: capacity development, see also Section

2.2.2).

Table 11: Sustainability correlations

Characteristic

Correlation

value /

Rank correlation

Funding level (overall) – sustainability - 0.23

Funding level (per month of implementation) - sustainability + 0.04

Duration of implementation – sustainability - 0.74

3.5.2 Weighting of sustainability

Apart from relevance (see also Section 3.1.2),

the DAC criterion sustainability was the most

important rating element in the reviewed

evaluations. The average sustainability weight-

ing was 2.3, with the interim evaluations at-

taching less importance to the criterion (aver-

age weighting 2.1) and the final and ex-post

evaluations attaching more (2.7 and 2.5, re-

spectively). All in all, the weighting of sustain-

ability was increased in six evaluations and

decreased in one. In particular, two out of three

final evaluations and two out of four ex-post

evaluations upgraded the criterion (see also

Table 10.)

Main reasons for changes made in the weight-

ing of sustainability:

upgrading: Improvements achieved

through the work of the programme must

endure beyond the end of the programme

(Algeria);

upgrading: Decisive criterion with regard

to institutional strengthening and capacity

development projects (Ethiopia);

upgrading: Sustainability is the most im-

portant criterion in an ex-post evaluation

(India);

upgrading: The programme is able to

build on a number of ongoing previous

projects or programmes, and special im-

portance is attached to ecological sus-

tainability, particularly in water-deficient

countries (Yemen);

upgrading: Due to premature termination

of the project, special importance is at-

tached to the partner's problem-solving

capacities and to independent continua-

tion of the measure (Mexico).

The evaluators performing the cross-section

evaluation were unable to follow the reasons

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 68

for upgrading this criterion as put forward in the

final evaluation of the „Infrastructure Pro-

gramme Water Supply and Wastewater Dis-

posal, Kosovo'. It was argued there that some

actors, judging by the e-VAL results, harboured

doubts about the programme's sustainability.

This, then, appears to be an assessment – not

an argument for weighting a criterion.

The low weighting of sustainability in the

'Wastewater Disposal in Provincial Centres,

Viet Nam' programme was founded on the fact

that, due to delayed implementation during the

initial phase, the TC contribution had been

unable to render the planned outputs and,

hence, achieve the anticipated results. That

line of reasoning too is more of a justification

for inadequate achievement of objectives (ef-

fectiveness).

In line with the GTZ concept of sustainable

development, there can be scant justification

for downgrading the criterion sustainability,

particularly since the evaluated projects and

programmes are very complex, and their prior-

ity areas contain results constellations that

focus on establishing durable capacities (see

also Section 2.2.2). Consequently, the sustain-

ability criterion should, in the opinion of the

evaluators, be perceived as particularly impor-

tant in the course of implementation and drawn

upon as a crucial factor of performance as-

sessment when all interventions have ceased.

3.6 Synopsis

3.6.1 Weighting of DAC criteria

As explained above, different degrees of im-

portance are attached to the various DAC crite-

ria in the evaluations (in terms of upgrading or

downgrading from a standard weighting of 2.0).

The criteria 'relevance' and 'sustainability' were

considered to be especially important (average

weighting 2.3), as was 'effectiveness' (2.2)

(see Table 12).

In addition to absolute averages, the frequency

of upgrading and downgrading also illuminates

the importance of a given criterion. In this

sense, too, 'relevance' and 'sustainability' are

the most strongly emphasised criteria. The

actual achievement of objectives ('effective-

ness') is next in importance, while the contribu-

tion to overarching development results (i.e.

the results beyond the 'attribution gap') and the

cost efficiency of service delivery are viewed

as less important factors for the performance

rating. The rationale for each of the ratings is

explained in the preceding sections.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 69

Table 12: Average weighting of DAC criteria

DAC criterion Average weight-

ing

Number of up-

grades (weighting

factor 3)

Number of

downgrades

(weighting factor 1)

Relevance 2.3 5 0

Effectiveness 2.2 3 0

Impact 2.1 2 1

Efficiency 2.0 2 1

Sustainability 2.3 6 1

Differences in the weighting of individual DAC

criteria are also apparent between the various

types of evaluation, i.e. depending on the time

of evaluation, as illustrated below in Table 13).

Finally, in comparison with interim evaluations,

the focus in ex-post evaluations is seen to lean

more toward the registration of indirect results

and sustainability. That, however, is fully in line

with the corresponding terms of reference.

Table 13: Weighting of DAC criteria according to time (= type) of evaluation

DAC crite-

rion / Type

of

evaluation

Relevance Effectiveness Impact Efficiency Sustainability

Interim

evaluations 2.6 2.3 1.9 2.1 2.1

Final

evaluations 2.0 2.3 2.3 2.3 2.7

Ex-post

evaluations 2.0 2.0 2.3 1.5 2.5

The overview of all fifteen evaluations shows

how vital it is, with regard to the success of the

measures, that the various projects and pro-

grammes are in conformance both with the

sector strategies of BMZ and with the respec-

tive national policies and poverty reduction

strategies, and that they take into account key

guidelines on regional cooperation and integra-

tion, capacity development and IWRM. At the

same time, it is also important to systematically

pursue the concept of sustainable develop-

ment in all planning and to reconcile the objec-

tives dimensions of the project by way of a

multi-level approach. This requires not only a

systematic approach to capacity development,

but also closer attention to the duration of ad-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 70

visory services in the sense of efficient 'follow-

up assistance'.

3.6.2 Overall rating of evaluated projects

and programmes

With an average rating of 2.7, the evaluation

portfolio is situated in the 'satisfactory' (= 'suc-

cessful') range of assessment. Altogether,

there is a preponderance of positive results in

the evaluated projects/programmes. However,

the collectively satisfactory result was slightly

below expectations. Most of the individual pro-

jects/programmes (80%) were classified as

'successful', and only three pro-

jects/programmes (20%) were deemed 'unsuc-

cessful' (see also Table 14 and Table and Ta-

ble 15).

With regard to the projects and programmes

with an 'unsuccessful' overall rating of 4 ('un-

satisfactory'), the principle held true in each

case that a project or programme can only be

classified as 'successful' in terms of develop-

ment policy if its direct results ('effectiveness'),

indirect results ('impact') and sustainability are

all rated at least 'satisfactory'(3):

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 71

'Improvement of Water Supply in the Tigray

Region, Ethiopia' – The weighted overall rat-

ing of this project stands at 3.2 and is therefore

still within the 'satisfactory', 'successful' range.

Its sustainability, however, was rated 'unsatis-

factory' (level 4). The overall rating was there-

fore downgraded to within the 'unsuccessful'

range.

'Rural Water Supply in Maradi, Niger' – For

this project, the calculatory weighted rating

would be 3.6, or somewhere between 'suc-

cessful' and 'unsuccessful'. Since, however,

both the achievement of the objectives 'effec-

tiveness' and 'sustainability' were respectively

rated as 'unsatisfactory' and 'inadequate', the

overall rating was lowered to 'unsatisfactory'

('unsuccessful') in this case also;

'Upgrading of Municipal Services, Turkey' –

Despite a weighted overall rating of 3.2, this

project was classified as 'unsuccessful', be-

cause its impact was rated 'unsatisfactory'.

Table 14: Overall ratings of evaluated projects/programmes

Country Project/Programme Type of

evaluation

Rating Success

level Weight. Unweight. Total

Africa su-

praregional NWI

Interim

evaluations

2.0 2.0 2 Successful

Algeria PDEA 3.4 3.4 3 Successful

Yemen WSP 2.1 2.0 2 Successful

Jordan WMIA 2.0 2.0 2 Successful

Kenya WSRP-KEN 2.2 2.2 2 Successful

Peru PROAGUA 1.9 2.0 2 Successful

Zambia WSRP-ZMB 2.4 2.4 2 Successful

Viet Nam WDPC 2.5 2.6 3 Successful

Ghana EVORAP Final

evaluations

2.6 2.6 3 Successful

Kosovo WSWDP 2.5 2.4 2 Successful

Mexico GICA 2.8 2.8 3 Successful

Ethiopia IWSTR

Ex-post

evaluations

3.2 3.2 4 unsuccessful

India IGWDP 2.1 2.2 2 Successful

Niger PHV/MI 3.6 3.6 4 unsuccessful

Turkey QADI 3.2 3.2 4 unsuccessful

Total (average) 2.6 2.6 2.7 successful

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 72

The table also shows that the weightings of

individual DAC criteria do not have very much

influence on the final overall rating. The differ-

ence between the calculatory weighted and

unweighted overall ratings stays within the

decimal range and is, in principle, negligible.

Table 15: Distribution of overall ratings according to success level

Success level Rating Quantity Percentage

'Successful'

'very good' (1) 0 0%

'good' (2) 8 53%

'satisfactory' (3) 4 27%

'Unsuccessful'

'unsatisfactory' (4) 3 20%

'inadequate' (5) 0 0%

'useless' (6) 0 0%

Figure 10: Distribution of overall ratings

Statistical analysis on the basis of rank correla-

tions (see also Section 1.2.2) reveals a some-

what closer connection between the term of

the project/ programme and its overall rating

(Rho = - 0.32) than between the funding level

and the overall rating (Rho - 0.10). While the

level of significance of such correlations is by

no means statistically secure, it can neverthe-

less serve as a general indication of the impor-

tance of the duration of implementa-

tion/advisory services.

The following Table 16 contrasts the average

weightings of the DAC criteria with their aver-

age ratings. Clearly, the criteria 'relevance' and

'effectiveness' tend to be given 'good' ratings,

while 'sustainability' is much more likely to be

given a rating in the 'satisfactory' range. Within

the context of GTZ contract and cooperation

management according to the logic behind

AURA, 'effectiveness', i.e. the measures'

achievement of objectives, is considered par-

ticularly important (justification for the success

of the project/programme vis-à-vis the client

BMZ). In principle, then, this criterion should be

expected to get ratings well into the 'good'

range. This area, though, was afflicted by

some problems affecting the recording of re-

sults due to imprecise (and at times unrealistic)

indicators and, above all, a lack of fundamental

data for dealing with qualitative aspects. In the

opinion of the evaluators, the hypothesis that

more (direct and indirect) results were

0%

53%27%

20%

very good

good

satisfactory

not satisfactory

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 73

achieved by a given project or programme than

were ultimately recorded should be taken as a

basis for improved monitoring. Ultimately, the

significance of objectives achievement in terms

of its contribution to the indirect results ('im-

pact') and 'sustainability' of the measures must

also be taken into account (see also Table 6).

Good effectiveness does not necessarily lead

to good impact and good sustainability, but the

correlations are definitely relevant. 'Effective-

ness' and 'impact' were rated equal in some

40% of the evaluations. No project or pro-

gramme with only a 'satisfactory' rating for the

achievement of objectives was able to secure

a 'good' rating for 'impact' and 'sustainability'.

Only in two (2) projects/programmes (Yemen

and Niger) was 'impact' rated higher than 'ef-

fectiveness'.

Table 16: Comparison of average weightings and ratings of DAC criteria

DAC criterion Average weight-

ing

Average rating

Relevance 2.3 2.0

Effectiveness 2.2 2.5

Impact 2.1 2.7

Efficiency 2.0 2.8

Sustainability 2.3 2.9

As illustrated below in Figure 11, the average

overall rating varies significantly as a function

of the time and/or type of evaluation. Six of the

eight projects/programmes for which interim

evaluations were performed received an over-

all rating of 2 ('good'). Conversely, three of the

four projects/programmes assessed in ex-post

evaluations were rated 'unsuccessful', i.e. 4.

As already mentioned, a poor rating for a key

criterion led to a downgraded overall rating.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 74

Figure 11: Average rating according to type of evaluation

A differentiated view of the average overall

rating and of the average ratings of DAC crite-

ria according to region (see also Table 17)

yields no substantive results, because the

scope of random sampling differs widely from

region to region (Africa 6, Maghreb/Middle

East 3, Asia 2, Southeast Europe 2 and Latin

America 2 projects/ programmes each). In that

connection, the four evaluated pro-

jects/programmes in Asia and Latin America

show conspicuously good 'relevance'; this be-

comes apparent in their relatively good 'sus-

tainability' ratings. The 'sustainability' ratings of

the African and Southeast European pro-

jects/programmes are poorer by comparison.

Table 17: Average overall ratings and DAC criterion ratings according to region

DAC criterion / region Overall

rating Relevance

Effective-

ness Impact Efficiency

Sustain-

ability

Sub-Saharan Africa 2.8 2.2 2.5 2.5 3.0 3.2

Maghreb / Middle East 2.3 2.3 2.3 2.3 2.7 2.7

Southeast Europe 3.0 2.5 2.5 3.5 2.5 3.0

Asia 2.5 1.0 2.5 3.0 3.0 2.5

Latin America 2.5 1.5 3.0 3.0 2.5 2.5

2,22,6

3,5

1

2

3

4

Mid-term Evaluations Final Evaluations Ex-Post Evaluations

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 75

4. Rating of cross-cutting devel-

opment themes

4.1 Poverty reduction and MDGs

For more than 30 years now, the water supply

and sanitation sector has been counted among

the most important areas of German develop-

ment cooperation. In line with the BMZ 'Water

Sector Strategy' adopted in 2006 and the BMZ

'Waste Management Sector Strategy' adopted

in 2008, the objectives here coincide with

those stated in the MDGs and the Millennium

Declaration. German development cooperation

in the water supply and sanitation sector pres-

ently comprises projects and programmes with

a total volume of some EUR 4 billion and is

geared to improving the living conditions of

more than 80 million people worldwide. The

terms of reference are formed by the national

development strategies devised by the partner

countries on their own, including sector strate-

gies and poverty-reduction strategies. In ac-

cordance with the sector strategies, the Ger-

man contribution stands for sustainable, re-

source-conserving, target group-oriented ap-

proaches. The most important target group in

the water supply and sanitation sector is the

poor population9.

Target group: 'The key target group for German development cooperation in the water sector is the poor and extremely poor population, which currently has little or no access to safe and/or adequate drinking water, sanitation and/or water for farming. The urban slums and rural regions are especially important in this

context.'10

9 Waste Management Sector Strategy, BMZ, 2008.

10 Water Sector Strategy, BMZ, 2006.

It must be noted at this point that the majority

of the projects and programmes evaluated in

2008 were planned and commenced prior to

the conclusion of the Millennium Declaration

and the adoption of the cited sector strategies.

They are therefore based on prior approaches

to German development cooperation. Never-

theless, most of the projects and programmes

do relate to poverty, the sector strategies and

the MDGs, though perhaps to different extents

and at times abstractly. All evaluation reports

contain more or less detailed descriptions of

the extent to which the evaluated measures

help(ed) achieve the MDGs. In most cases,

that impact is situated at the overarching level

at the end of a long results chain. Only in a

very few cases is an 'anticipated' contribution

documented.

All evaluated projects and programmes are

positioned in the water sector and designed to

potentially achieve MDG 7 ('Ensure environ-

mental sustainability'):

improved management of water resources

constitutes a direct contribution towards

achieving MDG 7A ('Integrate the princi-

ples of sustainable development into

country policies and programmes and re-

verse the loss of environmental re-

sources');

improved access to WS&S constitutes a

direct contribution towards achieving

MDG 7C ('Halve, by 2015, the proportion

of the population without sustainable ac-

cess to safe drinking water and basic

sanitation'). The relevant indicators are (i)

the proportion of the population with ac-

cess to improved drinking water supply

and (ii) the proportion of the population

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 76

with access to improved sanitation facili-

ties.

It should be noted, however, that this global

assessment is of only very limited validity with

regard to the 'Upgrading of Municipal Services,

Turkey' project, because its impact was rated

'unsatisfactory' (level 4).

An analysis of the objectives systems of the

evaluated projects and programmes with re-

gard to MDGs shows that the improvement of

water supply (WS) is mainstreamed in seven of

the projects/ programmes and that wastewater

management/sanitation services (WM/S) are

anchored in the objectives systems of four of

those seven. The improvement of WS and

WM/S is mainstreamed both in the formulation

of objectives and in one indicator of three (3)

projects/ programmes. Only a fraction of the

projects/programmes have an objectives sys-

tem that takes into account the achievement of

MDG 7C, in particular the improvement of

WM/S. Hence, in relation to the entirety of the

evaluated development measures, the con-

crete contribution toward increasing the pro-

portion of the population with access to im-

proved sanitary facilities by 2015 is compara-

tively small.

The evaluated projects/programmes also con-

tribute to the achievement of additional MDGs

(see also Table 18):

10 projects/programmes (66%) contribute

to the achievement of MDG 1 ('Eradicate

extreme poverty & hunger'), for example

by creating jobs and potentials for eco-

nomic activities and promoting irrigated

agriculture;

11 projects/programmes (73%) contribute

to the achievement of MDG 4 ('Reduce

child mortality'), for example by reducing

water-induced diseases, in particular diar-

rhoea, and by improving hygiene in baby

care;

8 projects/programmes (53%) contribute

to the achievement of MDG 5 ('Improve

maternal health'), for example by reducing

water-induced diseases, improving hy-

giene at child birth, increasing the avail-

ability of hygienically safe drinking water,

and creating adequate sanitation services;

5 projects/programmes also contribute to

MDG 2 ('Achieve universal primary educa-

tion'), 5 projects/programmes to MDG 3

('Promote gender equality and empower

women'),

1 project/programme to MDG 6 ('Combat

HIV/AIDS, malaria and other diseases')

and 1 project/ programme to MDG 8 ('De-

velop a global partnership for develop-

ment').

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 77

Table 18: Overview of contributions by evaluated projects/ programmes to the achievement of MDGs

Country Project/

Programme

Type of

evaluation

W

S

W

M/

S

Contribution toward achievement of Mil-

lennium Development Goal (MDG) num-

ber:

7 1 2 3 4 5 6 8

Africa

supra-

regional

NIW

Interim

evaluations

X X X

Algeria PDEA X X X

Yemen WSP Z I Z I X X X X

Jordan WMIA X X

Kenya WSRP-KEN Z I Z I X X X X X X

Peru PROAGUA Z I Z X X X X

Zambia WSRP-ZMB Z I Z I X X X X X X

Viet

Nam WDPC X X X X

Ghana EVORAP Final

evaluations

Z I X X X X X

Kosovo WSWDP X X X X X

Mexico GICA X X X

Ethiopia IWSTR

Ex-post

evaluations

Z X X X

India IGWDP X X

Niger PHV/MI Z I X X X X X

Turkey QADI X

Total (average) 15 10 5 5 11 8 1 1

X=contribution to MDG, Z=covered by an objective in the objectives system, I=covered by an

indicator in the objectives system

4.1.1 Conceptual problems

Frequently, poor settlements have only low-

priority access to the services of utility compa-

nies. A corresponding context was established

in approximately 50% (7) of the evaluated pro-

jects/programmes. Ten of the evaluated pro-

jects/programmes therefore aim to improve the

operational efficiency of water utilities (WUs)

while creating the requisite institutional frame-

work conditions for privatisation and/or com-

mercialisation. Drinking water supply is the key

area. In these projects, poverty reduction can

only be achieved indirectly by way of long re-

sults chains. Only 4 of the 15 pro-

jects/programmes take into account a 'waste-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 78

water component' like that called for by the

BMZ sector strategy and which is predomi-

nantly geared to improving the living conditions

of the poor population.

Consideration of poor people's needs is impor-

tant in all phases of development measures,

especially in connection with:

the conceptual design of the measure;

the selection of water supply systems and

types of equipment, e.g. water kiosks;

the selection of sanitation systems and

technologies, e.g. EcoSan;

the design of tariff systems;

health and hygiene education;

public awareness-raising work at the sec-

tor institutions;

the structuring of capacity development.

In 12 of the 15 evaluated projects/ pro-

grammes, the poverty relevance of projects

and programmes is not evident in the objec-

tives system, i.e. in the objectives and indica-

tors, but is nevertheless given a positive rating

with regard to the assessed relevance of the

project or programme. In many of the pro-

jects/programmes, a result is projected on the

assumption that the measure contributes to

poverty reduction. One example of this is the

argument that improved WS&S always helps

reduce poverty. Likewise, the argument that

measures taken in rural settings automatically

help reduce poverty is inconsistent.

Demonstrating the plausibility of indirect results

of projects and programmes is a major chal-

lenge with regard to poverty reduction. Poverty

indicators derived from a baseline study, and

their systematic monitoring during both the

implementation phase and the follow-up assis-

tance phase of the project/programme, as well

as the formation of 'control groups' before im-

plementation commences, provide information

on results actually achieved.

In fact, little effort has been invested at the

project/programme level in this area until now.

In many places, people are afraid of the 'ex-

penditures' that such investigations can in-

volve. In this respect, training measures and

the dissemination of good approaches from

comparable projects and programmes should

enhance the determination of results, particu-

larly with regard to poverty reduction. Again,

the formulation of indicators is the point of

departure.

'Best Practice'

The programmes 'Water Sector Reform Programme, Kenya', 'Drinking Water and Sanitary Pro-

gramme, Peru' and 'Water Sector Reform Programme, Zambia' are characterised by direct pov-

erty relevance, not only in their objectives systems but also in their consideration of the poor

population for the design of tariff systems. Other projects and programmes (e.g. 'Improvement

of Water Supply in the Eastern and Volta, Ghana') are characterised by good poverty-reducing

effects despite their not being embedded in the objectives systems.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 79

4.1.2 Mainstreaming in development strategies

The poverty reduction strategies provide the frame of reference for improving the living conditions of

the poor population. Analysis of the reviewed evaluation reports shows that most of the projects and

programmes are successfully embedded within the development strategies of the partner countries, as

evidenced by their 'good' relevance ratings. This mainstreaming is particularly emphasised (see also

Section 3.1.1) in 5 projects/programmes (Yemen, Ghana, Kenya, Peru and Zambia). Direct integration

into national poverty reduction strategies receives a particularly positive rating. The practical imple-

mentation of corresponding policies and, hence, the complementary orientation of short-term poverty

reduction measures with long-term sustainability, however, has proven difficult.

4.1.3 Poverty marker in the offers and its distribution

To enable the evaluation of contributions to poverty reduction, each and every project and programme

of German technical cooperation must be categorised according to different poverty markers. The first

question is whether the project/programme in question is one with a direct results chain or one with an

overarching approach at the macro and sector levels. Projects and programmes with direct results,

that is, ones with a definable, directly accessible (via a short results chain) circle of beneficiaries, are

for example regional measures devoted to basic health, water supply, basic education and advice to

organisations providing direct services for a target group. Typical examples of projects and pro-

grammes with overarching results include government advisory service projects, promotion of nation-

wide systems (e.g. a sector information system), and advice on organisation and management at the

macro/sector level.

The following 'poverty markers' are used for classifying the projects and programmes in GTZ offers

submitted to the BMZ11

:

direct poverty reduction (target group-oriented)

o SHA (Self-help oriented poverty reduction)

SUA (Other direct poverty reduction esp. basic social services)

comprehensive poverty reduction (at macro/sectoral level, long results chains)

o MSA (projects situated at macro/sectoral level)

general development approach

o EPA (no direct or comprehensive poverty orientation)

Five of the evaluated projects/programmes (33%) contribute directly to poverty reduction (i.e. in a tar-

get group-oriented manner; marker SHA or SUA). Seven of the evaluated projects/programmes (47%)

make comprehensive contributions to poverty reduction via long results chains (marker MSA). Two

11 Erfolgreiche Armutsbekämpfung in der Arbeit der GTZ – Eine Handreichung, GTZ, 2008.

( Successful poverty reduction in the work of GTZ – A guiding framework, GTZ, 2008)

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 80

projects/programmes (13%) have, at most, indirect poverty relevance (EPA). For the 'Infrastructure

Programme - Water Supply and Wastewater Disposal, Kosovo', neither the GTZ offer nor the cor-

responding evaluation report contained or recommended any poverty marker.

The following Table 19 surveys the markers attached to each project/programme. It should be noted in

this connection that the analysis of DAC markers in the reviewed evaluation reports is inhomogeneous

and very fragmentary. Only three of the reports (Niger, Turkey and Viet Nam) include any reference to

the poverty markers contained in the GTZ offers. The relevant category assignments were confirmed.

Consequently, the poverty markers shown in the Table correspond to those in the GTZ offers. It was

not recommended that they be adjusted in connection with the evaluation.

Table 19: Poverty markers of the evaluated projects/programmes

As the overview shows, only one of the ongo-

ing programmes (Peru) that were assessed

within the scope of an interim evaluation has a

direct target group-oriented approach to pov-

Country Project/

Programme

Type of

evaluation

Poverty markers

SHA SUA MSA EPA

un-

speci-

fied

Africa

supra-

regional

NWI

Interim

evaluations

X

Algeria PDEA X

Yemen WSP X

Jordan WMIA X

Kenya WSRP-KEN X

Peru PROAGUA X

Zambia WSRP-ZMB X

Viet Nam WDPC X

Ghana EVORAP Final

evaluations

X

Kosovo WSWDP

Mexico GICA X X

Ethiopia IWSTR

Ex-post

evaluations

X

India IGWDP X

Niger PHV/MI X

Turkey QADI X

Total (average) 3 2 7 2 1

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 81

erty reduction (marker SHA). Six of the eight

ongoing projects/programmes are situated at

the macro/sector level and only contribute to

poverty reduction via long results chains

(marker MSA). By contrast, three of the four

completed projects/programmes that were

assessed within the scope of ex-post evalua-

tion (Ethiopia, India and Niger) were geared to

immediate, target group-oriented poverty re-

duction. All three began far ahead of the Mil-

lennium Declaration and the current sector

strategies: Ethiopia in 1996, India in 1989 and

Niger in 1995.

A prime example in this context is the 'Im-

provement of Water Supply - Eastern and

Volta Regions Assistance Programme,

Ghana'. This programme has the poverty

marker SUA. By strengthening the local gov-

ernment structures, it has direct poverty rele-

vance while indirectly contributing to the solu-

tion of structural problems. This cooperative

programme is part of Ghana's national poverty

reduction strategy.

Projects and programmes with a strong orien-

tation towards sector reform (macro level)

through their focus on sector-policy advisory

services, regulation and financing mechanisms

carry the marker MSA, because they only con-

tribute indirectly to poverty reduction via meso-

level adjustments at the intermediary organisa-

tions, e.g. utility companies. Especially in the

case of projects and programmes in which

poverty relevance is clearly embedded within

the objectives system (Kenya and Zambia), the

positive effect at the micro level is consistent.

In the opinion of the evaluators, the assign-

ment of markers is not transparent in all of the

projects/programmes. MSA markers in various

other projects/programmes (Algeria, Mexico)

are less plausible, because the results from the

intermediary organisations down to the target

group are less distinct in those projects. Such

projects/programmes actually should have

been assigned the marker EPA, since they

have a more general development-policy ap-

proach. In the case of the 'Integrated Water

Management Programme, Algeria', for ex-

ample, no reference at all is made to the solu-

tion of poverty issues specific to the partner

country and to the target groups. That, of

course, led to a poor relevance rating (see also

Section 3.1.1).

There appears to be a certain degree of fun-

damental uncertainty with regard to the as-

signment of poverty markers. The argument

that all projects and programmes in the water

sector contribute either directly or indirectly to

poverty reduction is inconsistent and would

imply that the poverty marker EPA is superflu-

ous. Hence, importance should be attached to

poverty analysis at the planning stage, and the

actual relevance to poverty should be given a

close, critical examination.

4.1.4 Target group differentiation and

poverty

analysis

Roughly half (7 out of 15) of the evaluated

projects and programmes exhibit objectives-

level target group differentiation in their objec-

tives systems. Six of the projects/programmes

also have such differentiation integrated into

their indicators. A basic distinction is drawn

between four target groups: urban populations,

provincial populations, rural populations and

poor populations:

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 82

three programmes (Kenya, Peru, Zambia)

are geared towards the poor urban popu-

lation;

one programme (Ghana) is geared solely

towards the provincial population;

two projects/programmes (Ethiopia,

Yemen) are geared towards the urban

and rural sections of the population.

Based on offers submitted to the BMZ and on

the evaluation reports, approximately half of

the evaluated projects and programmes have

no recognisably clear differentiation of target

groups. These projects and programmes are

generally geared towards the improvement of

sector institutions, WS&S companies or certain

regions.

As mentioned above, 12 of the evaluated 15

projects/programmes have direct or indirect

poverty relevance by reason of their corre-

sponding poverty markers. However, only in

the objectives systems of three programmes

(Kenya, Peru and Zambia) is there any clear-

cut pro-poor orientation. Thus, as analysed by

the evaluators, most of the evaluated projects

and programmes lack a clear poverty orienta-

tion, which is characterised as a key task in

most applicable sector strategies and con-

cepts.

It is surprising in this context that only a single

poverty analysis was performed (Yemen) and

that the absence of a poverty analysis only led

to downgrading of relevance in five of the

evaluations. In two programmes (Ethiopia and

Ghana), the analyses of the evaluation mis-

sions even perceive a risk that poor sections of

the population could be excluded the results of

the development measure (lack of acceptance

at target-group level for charge-carrying WS

facilities in place of traditional supply options).

In the 'Improvement of Water Supply in the

Tigray Region, Ethiopia' project, the target

group (provincial and rural population) is

clearly identified in the objectives system, but

only technical and institutional studies were

performed in preparation for the measure, i.e.

no socio-economic investigations for determin-

ing a willingness to pay. The requisite tariff

study was not performed until the end of the

measure.

Most particularly for projects and programmes

in least developed countries (LDCs), poverty

relevance should be integrated into the respec-

tive measures' concept and substantiated by

socio-economic studies, because it is there

that especially strong poverty reduction results

can be achieved.

4.1.5 Conclusions

As analysed by the evaluators, the rare exis-

tence of a definite pro-poor orientation in most

of the evaluated projects and programmes is

not necessarily attributable to a false concept.

It must be assumed that relevant impacts on

poverty can be demonstrated for nearly all

development measures. This orientation could,

however, be more plausibly represented if the

target groups of the projects and programmes

were more closely analysed and defined in the

objectives system. For the purposes of the

projects and programmes under review, a pov-

erty analysis would make a crucial contribution

to presenting the corresponding results poten-

tial and to establishing corresponding monitor-

ing activities. Especially in LDCs, poverty ori-

entation presupposes a strong results constel-

lation. Unfortunately, none of the evaluation

reports explicitly deal with the contribution of

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 83

development measures to broad-based eco-

nomic growth (participation of poor segments

of society in economic growth – pro-poor

Growth) in the respective partner countries,

because that area lies outside the terms of

reference prescribed for the evaluations.

Nevertheless, at least some of the projects and

programmes display either direct or 'semi-

direct' relevance to the target group. This con-

stitutes an essential prerequisite for a partici-

patory approach to the involvement of poorer

sections of the population in economic and

political processes. Some examples of this can

be found in the 'Drinking Water and Sanitary

Programme, Peru', in which the existing bond

between WUs and target groups/customers is

being strengthened by the institutionalisation of

'socio-economic environment management'. In

the rural, traditionally religious and conserva-

tive parts of Yemen, the 'Institutional Devel-

opment of the Water Sector, Yemen' pro-

gramme relies on the work of women's groups

within the village communities to establish local

water user committees and involve women in

downstream investment planning (construction

of cisterns).

The evaluation reports under review also pro-

vide no answer to the question of how well the

implemented projects and programmes were

actually able to sustainably contribute to a

more equitable distribution of water resources.

That area, too, was not included in the terms of

reference for the conducted evaluations. Until

now, the evaluators' analysis has done just as

little to document the relevant results at the

project/programme level as it has to provide

concrete data on the number of people who,

ultimately, benefitted from the measures in the

sense of poverty impacts.

4.2 Gender equality

4.2.1 Gender markers of the evalu-

ated projects and programmes

German development policy sees gender

equality as part of the policy dimension of de-

velopment and therefore attaches just as much

importance to it as to respecting human rights

and to promoting good governance and de-

mocracy. The mainstreaming of gender as-

pects is a key characteristic of quality at the

GTZ and derives from the concept of sustain-

able development. In addition to this, the GTZ

acts in pursuance of the gender mainstreaming

strategy of the German Government. The

overall goal is for women and men to draw

equal benefit from the development-policy

contribution of TC and be equally able to ac-

tively participate in shaping this contribution.

The relevant indicator is that, from 2006 on,

GTZ's final and ex-post evaluations increas-

ingly confirm that women and men are making

equal contributions to, and deriving equal

benefit from, TC projects and programmes.12

The gender (G) markers of DC projects and

programmes are intended to facilitate the as-

sessment of measures according to how much

they contribute to gender equality and to

strengthening the rights of women. They pro-

vide important information on anticipated re-

sults and help align development processes

with internationally agreed development objec-

tives of gender equality. The corresponding

12 GTZ Corporate Strategy on Gender Mainstreaming,

2006.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 84

markers employed within the OECD-DAC sys-

tem are13

:

marker G0: The development measure

does not have the potential to contribute

to gender equality and/or no gender-

specific results can be inferred from it.

The mere fact that men and women make

equal use of a development measure

does not justify allocation to the category

G-0, which may only be assigned in ex-

ceptional, well-founded cases;

marker G1: The development measure

has inferable positive effects on gender

equality, although gender equality is not a

main objective of the development meas-

ure. It is a significant secondary objective

of the project/programme (at the results

level) but is not solely decisive for its im-

plementation. The approach and proce-

dure for fostering gender equality should

be clearly mainstreamed in the project

design. The mere avoidance of negative

effects on the gender-equality situation

does not justify allocation of the marker

G1;

marker G2: Gender equality is a principal

objective of the project/programme and is

clearly expressed both in the project ob-

jective and in the entire hierarchy of ob-

jectives. Gender equality is at the core of

the project design, and direct results are

intended in this area.

13 BMZ/GTZ/KfW: Impacts on Gender Equality in Devel-

opment Cooperation Interventions: Gender Markers in

Technical and Financial Cooperation, 2006.

Positive gender-related results rarely occur all

by themselves, but must be actively integrated

into the project approaches. The available

ways and means of contributing toward greater

gender equality in connection with water pro-

jects and programmes differ according to pro-

ject approach and region. The evaluated pro-

jects and programmes devoted to the thematic

priority area 'water' are analysed below with

respect to their gender equality orientation.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 85

Table 20: Gender markers of the evaluated projects/programmes

Country Project/

Programme

Type of evalua-

tion

Gender markers

G0 G1 G2 unspecified

Africa supra-

regional NWI

Interim

evaluations

X

Algeria PDEA X

Yemen WSP X

Jordan WMIA X

Kenya WSRP-KEN X

Peru PROAGUA X

Zambia WSRP-ZMB X

Viet Nam WDPC X (X)

Ghana EVORAP Final

evaluations

X

Kosovo WSWDP X

Mexico GICA X

Ethiopia IWSTR

Ex-post evalua-

tions

X (FP)

India IGWDP X

Niger PHV/MI X (FP)

Turkey QADI X (FU)

Total (average) 4 11 0 0

Three projects/programmes (Ethiopia, Niger

and Turkey) were assigned the erstwhile 'F

markers' FP and FU, with FP corresponding to

G-1 and FU corresponding to G-0.14

All three

G-0 markers are consistent, because the pro-

jects/programmes (Africa supraregional, Kos-

ovo and Turkey) are, by definition, not oriented

towards gender equality. The 'Infrastructure

Programme Water Supply and Wastewater

14 In December 2000, the previous 'F categories' were

replaced with 'G markers' according to the DAC sys-

tem.

Disposal, Kosovo' is the only one of the fif-

teen evaluated projects/ programmes in which

a gender analysis was performed at the begin-

ning of the measure. In connection with the

'Nile Water Initiative - Planning and Man-

agement of Water Resources in the Nile

Basin', gender equality was written into the

national water policies.

The gender aspect was anchored as an indica-

tor in the objectives systems of two pro-

grammes (Yemen and Zambia). Accordingly,

both programmes are classified as G-1. This

marker is consistent, because gender equality

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 86

is, by definition, a secondary objective of both

programmes. In five projects/programmes

(Ghana, India, Kenya, Mexico and Peru) gen-

der equality is integrated into the design of the

respective measure but is neither substanti-

ated by a corresponding objective nor by an

indicator. However, according to statements in

the evaluation reports, these pro-

jects/programmes did include gender-specific

activities with discernible results at the target-

group level.

In the 'Improvement of Water Supply in the

Eastern and Volta Regions, Ghana' pro-

gramme (marker G-1), however, the concep-

tual design was not gender-responsive, nor

was a gender analysis performed. Gender

mainstreaming, though, is known to be in-

cluded in the Ghanaian sector policy. The re-

spective evaluation mission assumes that this

specification is being put into practice by ac-

tively involving women in the implementation

process.

In the 'Indo-German Watershed Develop-

ment Programme, India' (marker G-1), a

gender analysis was performed not at the be-

ginning of the measure, but in the course of its

implementation (1995). The measure has ef-

fected visible, albeit not quantifiable, gender-

specific changes, because the needs of

women were taken into account. Women also

benefit equitably from the results of the pro-

gramme. Behaviour toward women in the sec-

tor environment has changed recognisably,

and women's participation in public decision-

making processes is increasing. They have

gained better access to resources and are now

playing a more active role in the community.

Analysis of the evaluation reports also showed

that it may not have been consistent to assign

the marker G-1 to four of the pro-

jects/programmes (Algeria, Ethiopia, Jordan

and Viet Nam). Gender equality is neither a

secondary objective nor the principle objective

in those projects/programmes:

'Integrated Water Management Programme,

Algeria': In the original version of the offer,

gender equality was accounted for with an

indicator. The corresponding component, how-

ever, was discontinued, and no gender analy-

sis was performed. The underlying IWRM ap-

proach, which provides for the promotion of

women, was weakened between phase 1 and

phase 2. The mere promotion of women's

groups, coupled with the argument that women

play an important role with regard to the sensi-

tisation and participation of female villagers,

does not justify allocation to the category G-1.

'Improvement of Water Supply in the Tigray

Region, Ethiopia': No gender analysis was

performed, and no gender-responsive themes

were reflected in the reporting. The explanation

for assigning marker G-1 was that all WS&S

projects generally support women, because

they are the ones responsible for matters of

water supply and health. Hence, improving

WS&S, it is said, reduces their time expendi-

ture, whereas the contention that is also gen-

erates additional leisure time was already fre-

quently disproven. The assumption that all

measures taken to improve WS&S contribute

to gender equality does not justify assigning

marker G-1.

'Efficient Water Resources Management in

Irrigated Agriculture in the Jordan Valley

and Highland Areas, Jordan': Basically, the

fact that gender aspects are an important part

of Jordanian policy and that the decentralisa-

tion process in the water sector opens up new

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 87

possibilities for women to take part in the deci-

sion-making process without restrictions based

on status does not in itself justify assignment of

marker G-1. Nor, in principle, does the project-

engendered, anticipated potential for women to

become more closely integrated in water user

groups and in public life justify the assignment

of marker G-1.

According to the submitted reports, five of the

evaluated projects/programmes (Jordan,

Kenya, Kosovo, Niger and Zambia) make

highly aggregated contributions to the

achievement of MDG 3 ('gender equality').

Particularly in the 'Water Sector Reform Pro-

gramme, Zambia', this result is understand-

able, because gender equality is main-

streamed at the objectives level. By contrast,

the 'Institutional Development of the Water

Sector, Yemen' programme, which also has

gender aspects integrated within its objectives

system, is not assumed to have an effect on

MDG 3. The contribution made by the 'Infra-

structure Programme Water Supply and

Wastewater Disposal, Kosovo', with the

marker G-0, to achieving MDG 3 is inconsis-

tent. Again, it was argued that all projects and

programmes contributing to improved WS&S

also contribute indirectly to gender equality,

because women normally take care of their

families' health and hygiene matters. The

'WMIA Jordan' project does contribute indi-

rectly to achieving MDG 3, because many

women are employed as seasonal workers.

There is, however, a conspicuous lack of con-

crete data for documenting any genuinely gen-

der-related results in any of the pro-

jects/programmes (for example: surveys con-

cerned with easier access to water, improved

household hygiene, etc.). Only in the evalua-

tion of the 'Water Sector Reform Pro-

gramme, Zambia' was a gender-responsive

survey made of some 850 households. That

survey identified and recorded such definitely

positive effects of improved water supply on

such aspects as workload, leisure time and

health.

4.2.2 Conclusions

Many projects and programmes, sometimes in

cooperation with parallel measures, have the

potential to promote gender equality and to

implement activities identified in gender analy-

ses. In projects and programmes leading to the

incorporation of gender equality as a cross-

cutting theme in national water policies and

development strategies, however, practical

implementation often proves difficult, and no

relevant approach is implemented at the tar-

get-group level. The line of reasoning that all

projects in the water sector contribute to gen-

der equality and, hence, to the achievement of

MDG 3 is inconsistent. Although gender analy-

ses should, in principle, form part of the project

or programme‟s preparation process, corre-

sponding investigations were only performed at

the beginning of one project (Kosovo) and

during one programme (India).

Unfortunately, reliable data to document gen-

der-specific aspects and results of the pro-

jects/programmes are not available. Positive

effects and results constellations have

been'anticipated' rather than recorded and

documented until now. Only one evaluation

report, i.e. the ex-post evaluation of 'Rural

Water Supply in Maradi, Niger' programme

shows that, by the end of the programme, the

share of women in the newly established and

operational water user committees for super-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 88

vising the operation of the newly built and re-

habilitated well systems had reached 50%. By

the time of the ex-post evaluation, however,

that same percentage could only be found in

the roughly 10% of committees that were still

active.

In addition to conceptual mainstreaming of the

cross-cutting theme 'gender equality' in a given

project or programme, crucial importance is

also attached to the incorporation of appropri-

ate objectives and indicators in the objectives

system of the respective project or programme.

Then, when formulating the indicators, it is

likewise crucial to precisely formulate/define

the measurable appraisal factors thereby ena-

bling the straightforward recording of direct

results with regard to gender equality. Gender

analysis, which can also be performed after

implementation has begun – that is, during an

ongoing project or programme – can provide

information on the results dimensions and

serve as a basis for subsequent monitoring.

The most important recommendations drawn

from the evaluation reports on the cross-cutting

theme gender are:

gender analyses should always be per-

formed (even after the measure has

commenced);

indicators enabling straightforward regis-

tration of direct and indirect gender equal-

ity results need to be formulated;

reform programmes should actively pro-

mote the integration of gender equality

into national water policies as a cross-

cutting theme;

targeted awareness-raising and training

measures should be offered to women in

order to strengthen their role and, hence,

to initiate changes in the traditional gen-

der role model. The targeted assignment

of women to management-level positions

should be promoted, and initiatives and

forms of organisation geared to women

should be created and strengthened.

4.3 Effects on the partners' capac-

ity for action

4.3.1 Capacity development within

the context of the evaluated

projects and programmes

Capacity development is a holistic process

through which people, organisations and socie-

ties mobilise, maintain, adjust and expand their

capacities for sustainable development. Sup-

port for capacity development by external part-

ners is a central instrument of development

cooperation with which people, organisations

and societies are empowered to develop and

expand their capacities for action and man-

agement. Basically, the capacity development

process requires the support of the involved

actors and presumes their assumption of

'ownership', i.e. high levels of identification with

and commitment to the targeted changes.

Ownership may, however, not develop until the

reform process has already set in; external

partners can foster the development of owner-

ship and assume temporary co-responsibility

during the process.

GTZ's specific approach to supporting capacity

development derives from its concept of sus-

tainable development. Development is under-

stood as a permanent process of seeking,

negotiating and learning on the part of all

stakeholders that defies detailed advance

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 89

planning. Its basic principles are a holistic ap-

proach and process/value orientation. This

approach cannot be implemented via financial

transfers, but depends on the joint responsibil-

ity of the partner(s) and GTZ in the sense of

mutual accountability15

.

According to its definition, capacity develop-

ment takes place at four levels: at the individ-

ual level, the institutional level and the net-

working and system development levels. This

breakdown displays certain parallels with the

intervention levels, but it also provides an addi-

tional analysis model for the strategic orienta-

tion of projects and programmes. The Paris

Declaration promotes the development of ca-

pacity development strategies at the national

and sector levels, and GTZ has participated in

the drafting of corresponding strategies in

many different countries. German TC attaches

special importance to verifying the results of

capacity development. Hence, monitoring and

evaluation of capacity development measures

enjoy priority at GTZ. The monitoring systems

of projects and programmes should do more to

record changes in the capacities (for action

and management) of individuals and organisa-

tions, and developments in social contexts, via

appropriate indicators. All this must be ac-

counted for both in offers for projects and pro-

grammes and in operational planning.

The evaluation reports neither contain a pre-

cise definition of the capacity development

interventions that were conceived and imple-

mented in the respective projects and pro-

15 Das Verständnis der GTZ von Capacity Development -

Ein Orientierungs- und Handlungsrahmen für die GTZ,

2006 ('GTZ‟s Understanding of Capacity Development:

A Guiding Framework for Corporate Action

grammes, nor do they describe the corre-

sponding results. If at all, the respective texts

merely contain fragmentary assessments and

reflections. It must also be noted in this con-

nection that the use of the term 'capacity de-

velopment' in the evaluation reports (as well as

in the GTZ offers) does not follow any gener-

ally applicable definition. In most cases, 'ca-

pacity building' is equated with capacity devel-

opment, but such terms as 'training', 'institu-

tional capacity building' and 'institutional

strengthening' are also used synonymously. All

in all, in the simultaneous absence of an ap-

propriate concept or strategy, it is very difficult

to identify and categorise the respective inter-

ventions at the level of the individual projects

and programmes. At the same time, it is not

practicable within the scope of cross-section

evaluation to identify factors of success and

failure that are relevant to capacity develop-

ment, because they were not described.

The following Table 21 summarises the dimen-

sions and results levels of all fifteen evaluated

projects/programmes. It must be noted in this

connection that the information provided was

'filtered out' of the evaluation reports by the

authors of the cross-section evaluation. The

interventions and levels indicated are therefore

those that were actually identified, not those

that were 'targeted' or expected. The docu-

mentation includes no consistent depiction of

the respective interventions and results con-

stellations.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 90

Table 21: Levels of capacity development in the evaluated projects/programmes

Country Project/

Programme Training OE

Network-

ing

System

devel.

Results

level

CD

concept

Africa

supra-

regional

NWI X X macro

Algeria PDEA X X micro-meso

Yemen WSP X X X X multi-level X

Jordan WMIA X X micro

Kenya WSRP-

KEN X X X X multi-level

Peru PROAGUA X X X X meso-

macro

X

Zambia WSRP-

ZMB X X X X multi-level

Viet Nam WDPC X meso

Ghana EVORAP X X meso

Kosovo WSWDP X X meso

Mexico GICA X X meso

Ethiopia IWSTR X X meso

India IGWDP X X X micro X

Niger PHV/MI X X micro

Turkey QADI X meso

The overview shows that the explicit priority

area of actually implemented capacity devel-

opment interventions is situated within the

individual dimension in terms of personnel

training. Such measures are often combined

with the provision of advisory services for insti-

tutional or organisational development of

meso-level executing agencies and intermedi-

ary organisations. Often, concrete assistance

is provided for process optimisation at utility

companies. Only a minor share of the pro-

jects/programmes include measures geared to

system development and network formation. It

is also apparent that, in analogy to the focusing

of results levels (see also Section 2.2.2), there

is a heavy concentration of capacity develop-

ment measures at the meso level. Only about

one third of the projects/programmes intervene

in the sense of capacity development at the

macro level. What is more surprising is that

only three programmes (India, Yemen and

Peru) have a documented, monitored capacity

development strategy or concept.

Comparison of the conceptual results levels of

the evaluated projects and programmes shown

in Figure 2 with the 'actual' intervention levels

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 91

summarised in Table 21 reveals discrepancies

in some of the projects/programmes in the

sense that the realities of implementation devi-

ate from the concept or planning. To the extent

that such discrepancies are noticed, the plan-

ning is rarely modified. For example, PDEA

Algeria intervenes only at the micro and meso

levels, despite the fact that a significant macro-

level contribution was anchored in the pro-

gramme concept. By way of contrast,

EVORAP Ghana unsuccessfully attempted to

implement a multi-level approach as designed

and with due attention to the micro and macro

levels. According to the evaluators' analysis,

most of the programme's work is done at the

meso level. The WMIA Jordan project, on the

other hand, provides most of its outputs with a

micro-level focus, even though the original

planning called for decidedly more attention to

the meso and macro levels.

The most important capacity development

interventions of selected projects and pro-

grammes are compiled below to convey a

general idea of their spectrum. (The PDEA

Algeria and IWSTR Ethiopia programmes were

omitted, because the respective evaluation

reports shed no light on interventions under-

taken for capacity development purposes):

The project entitled 'Nile Water Initiative -

Planning and Management of Water Re-

sources in the Nile Basin' (NWI) focuses

macro-level capacity development interven-

tions in the areas 'improving individual capaci-

ties' and 'system building' by providing task-

specific and strategic training measures for the

staff of national sector ministries (political

analysis and dialogue, communication, strate-

gic planning) and by exerting positive influence

on framework conditions through support for

national water policies (bearing in mind trans-

boundary cooperation and IWRM principles);

EVORAP Ghana: Training measures for WU

personnel (financial, commercial and technical

operations) plus organisational advice for the

development of utility companies;

IGWDP India: Training measures, information-

gathering tours, networking of intermediary

organisations (NGOs);

WSP Yemen: Human resources development

and training (HR component in the pro-

gramme), organisational advice for the estab-

lishment of municipal WUs and water authori-

ties, networking of sector institutions, and advi-

sory services to generate a water policy;

WMIA Jordan: Training measures for mem-

bers of water-user groups and advisory staff at

the lead executing agency, and advisory ser-

vices for establishing water user groups and

committees;

WSRP Kenya: Training measures for staff of

regional water authorities, WUs and the higher-

level sector authority, organisational advice for

water authorities and the trust fund for the

establishment of financing mechanisms and for

the introduction of appropriate technologies

(water kiosks, sanitation systems), regional

networking and system development (regula-

tion, pro-poor policy development and invest-

ment planning);

WSWDP Kosovo: Training measures for WU

staff (financial, commercial and technical op-

erations) and assistance to WUs for introduc-

ing technical improvements in technical opera-

tions;

GICA Mexico: Training measures for staff of

water and regulatory authorities and for per-

sonnel of non-governmental intermediaries,

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 92

and organisational advice for staff of water and

regulation authorities;

PHV/MI Niger: Training measures (and

awareness-raising measures) for village water

user committees, building contractors and staff

of regional water authorities, advisory services

for the establishment of village water user

structures, and task-specific assistance to

regional water authorities;

PROAGUA Peru: Training measures for per-

sonnel of WUs, advisory services for the estab-

lishment and development of a countrywide

human resource development system for the

water sector, organisational advice for utility

companies, establishment of association struc-

tures for WUs and sanitary utility companies,

and establishment of an exchange system;

WSRP Zambia: Training measures for per-

sonnel of regional water authorities and WUs,

organisational advice for water authorities and

the trust fund for establishing and developing

financing mechanisms and introducing appro-

priate technologies (water kiosks, sanitation

systems), regional networking and system

development (regulation, pro-poor policy de-

velopment and investment planning);

QADI Turkey: Project and investment planning

training measures for staff at local government

units;

WDPC Viet Nam: Training measures on tech-

nical operations for staff of national sanitation

utilities.

The evaluation reports contain little information

about the actual results of the implemented

capacity development interventions. Measures

are referred to as positive, but without citing or

explaining why they were regarded as suc-

cessful, one such case being Peru, where a

national programme for human resource de-

velopment based on the capacity development

concept was successfully established. The

respective progress reports would have been

useful here as a source of information. The

evaluation report did show, however, that the

extensive training measures at the utility com-

pany level were particularly successful in pro-

fessionalising the operations. In addition, train-

ing measures for decision-makers in national

policy dialogue are of decisive importance.

Especially the larger sector programmes with a

systematic, well-balanced multi-level approach

(Yemen, Kenya, Zambia) stand as good ex-

amples of the very complex results constella-

tion of capacity development interventions.

At the same time, it is evident from the reports

of the evaluators who performed the individual

evaluations that the time allowed for establish-

ing or decentralising capacities and capabilities

for action, or for processes of change, was

usually too short. Time and again, it is recom-

mended in this connection that advisory ser-

vice cycles be lengthened in the sense of

process support. The recommendations made

by the authors of the individual evaluations

concentrate on the need for systematic expan-

sion and focusing of capacity development

interventions in order to fill the existing 'capac-

ity gaps' in the partner countries.

4.3.2 Conclusions

In principle, capacity development was found

to be taking place at all levels in the evaluated

projects and programmes, and there was a

very definite setting of priorities in the area of

training measures and organisational consul-

tancy. Adequate consideration of the areas

'networking' and 'system development' – the

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 93

areas with the greatest results options – within

the scope of a 'multi-level approach' is, how-

ever, only evident in the evaluation reports of a

minor share of the projects/programmes. In the

more complex projects and programmes, the

evaluators also noticed relevant difficulties with

the interlinkage of technical, organisational and

policy advisory services (main dimensions:

steering of inputs and professional training).

It was surprising to note the apparent lack of a

systematic approach and the basically inade-

quate conceptual base in the greater part of

the evaluated projects and programmes. With-

out a capacity development strategy that also

the implementing partner can consider trans-

parent, a given project or programme can

hardly produce system-oriented results. In this

respect and in many cases, the national water

policies and sector strategies prescribe general

guidelines. In the view of the evaluators, each

and every development measure should for-

mulate an appropriately oriented development

strategy and routinely examine it with regard to

relevance and implementation. In this respect,

the introduction of 'Capacity WORKS' as a

toolbox for corresponding analytical and plan-

ning processes should yield an effective im-

provement.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 94

5. GTZ's concept of sustainable

development

5.1 GTZ's concept of sustainable

development within the context

of the evaluated projects

and programmes

The three principal characteristics of the work

performed by GTZ in accordance with its con-

cept of sustainable development are:

a holistic approach: combining the eco-

nomic, social and ecological aspects (ob-

jectives dimensions) with sectoral, organ-

isational and policy advice at the micro,

meso and macro levels;

a process-oriented approach: help to-

wards self-help, creating transparency

with regard to the interests of different

stakeholders and promoting interaction

between the state, civil society and the

private sector;

a value-oriented approach: promoting

democracy, the rule of law, human rights,

gender equality, good governance and a

social and ecological market economy.

Fourteen of the evaluation reports adopted a

largely stereotypical, positive view of these

principles, substantiating the context of the

respective project or programme in a very ab-

stract manner with regard to the objectives

dimension and all three approaches (see also

the examples cited below: Zambia and Tur-

key). Only in the rarest of cases is there any

recognisably concrete relevance to the actual

interventions, and then only by way of very

long results chains. In the opinion of the au-

thors of this cross-section evaluation, the rele-

vant parts of the reports are, in their present

form, not very informative. No in-depth analysis

is possible on that basis.

Only in the case of the 'Improvement of Wa-

ter Supply in the Tigray Region, Ethiopia'

project did the evaluators deny its claim of

having a sustainable approach to development

because it dealt almost exclusively with eco-

nomic aspects of WU operations and was

practically non-existent at the target-group

level. It gave no consideration to the interests

and needs of the target groups and did not

monitor development in the supply situation. At

the same time, important factors like ground-

water availability and the economic viability of

poor households went unaccounted for in the

plans for expanding the water supply.

Table 22 below represents an attempt to realis-

tically record how well the characteristics of

sustainable development were actually ac-

counted for in the corresponding parts of the

individual evaluations. Twelve projects and

programmes are seen to be pursuing a holistic

approach, but only eight of the measures have

a process-oriented concept. Simultaneously,

most of the projects and programmes (11) are

acknowledged to have an explicit orientation

towards values, and the same number of pro-

jects and programmes do a well-balanced job

of combining the economic, social and ecologi-

cal objectives dimensions. A failure to give

balanced consideration to the objectives di-

mensions was found in four of the pro-

jects/programmes (Africa supraregional, Ethio-

pia, Niger and Turkey).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 95

Table 22: Consideration of the characteristics of sustainable development in the evaluated pro-

jects/programmes

Country Project/

Programme

Holistic

approach

Process

orientation

Value

Orienta-

tion

Objectives dimensions

Africa

supra-

regional

NWI X X economic, ecological

Algeria PDEA X X economic, social, ecological

Yemen WSP X X X economic, social, ecological

Jordan WMIA X X X economic, social, ecological

Kenya WSRP-KEN X X X economic, social, ecological

Peru PROAGUA X X X economic, social, ecological

Zambia WSRP-ZMB X X X economic, social, ecological

Viet

Nam WDPC X economic, social, ecological

Ghana EVORAP X X economic, social, ecological

Kosovo WSWDP X economic, social, ecological

Mexico GICA X X X economic, social, ecological

Ethiopia IWSTR economic

India IGWDP X X X economic, social, ecological

Niger PHV/MI X economic, social

Turkey QADI X economic, social

A holistic approach is established for the 'Wa-

ter Sector Reform Programme, Zambia', for

example, where the social, economic and eco-

logical objectives dimensions are combined by

the programme's focus on the supply of drink-

ing water and sanitation at socially acceptable

cost with incorporation of the IWRM approach.

At the same time, all the advisory services

provided at the sectoral, institutional and politi-

cal levels are based on a systematic multi-level

approach. Key areas of macro-level advice are

water policy and donor harmonisation. At the

meso level, the programme intervenes via

organisational advice (priority area WUs) and

the development of financing and sector infor-

mation systems. This is supplemented by sup-

port for the establishment and strengthening of

water user groups. With regard to networking,

the programme fosters exchange between the

numerous sector institutions and civil society

organisations and the private sector. In the

sense of good governance, the establishment

of a regulatory authority led to more transpar-

ency and quality improvements within the sec-

tor (monitoring of standards, tariffs, etc.). The

programme's value orientation is expressed in

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 96

terms of regard for human rights in the sense

of creating access to safe drinking water, fos-

tering democratisation through a participatory

approach and intervening at micro-level to

identify locations for new water kiosks, and

emphasising a social market economy by in-

troducing a socially equitable tariff structure.

A holistic approach aligned with the IWRM

concept was ascertained for the 'Drinking

Water and Sanitary Programme, Peru'. The

ecological (wastewater management and

treatment), economic (better economic per-

formance capabilities on the part of WUs) and

sociopolitical dimensions (sociopolitical envi-

ronment management of WUs) all stand in a

well-balanced relationship to one another,

even though the ecological dimension is lag-

ging behind because of minor impacts on sani-

tation so far. The programme pursued a multi-

level approach that is to contribute to sustain-

able development, especially through interac-

tion between the macro and meso levels.

Thus, participation in the design of the sector

policy and regulatory framework is geared both

to fostering the sustainability of progress made

by the WUs and to disseminating the learning

effects achieved at the WUs. The intended

coordination of training and advisory services

for the water sector also reaches beyond the

WUs participating in the programme to benefit

the entire sector. If the programme is brought

to a successful conclusion, the strengthened

performance capacities of the WUs and their

sustainably improved supply services can be

expected to contribute to sustainable develop-

ment. The development measure is based on a

process- and value-oriented approach. Particu-

larly at the local level, one of the programme's

key priorities is to improve dialogue between

the WUs and the social groups in the local

area. In addition, the interests of WU personnel

were appropriately considered, and importance

was attached to improving the internal com-

munication.

5.2 Conclusions

It is apparent that the objectives systems under

review, some of them very complex, with their

many indirect and highly aggregated results

constellations, basically cater to all objectives

dimensions and approaches of GTZ's sustain-

able development concept. Thanks to this 'an-

ticipated results constellation', the larger sector

programmes in particular, with their very ex-

plicit multi-level approach, are practically al-

ways in line with the concept. The extent to

which these relationships really can be attrib-

uted to actual project interventions should at

least be discussed within the scope of the

evaluations in order to promote learning effects

and further development.

Certainly, the good relevance ratings of the

evaluated projects and programmes indicate

that the approaches to sustainable develop-

ment were adequately considered in the con-

ceptual design. Conversely, the rather medio-

cre success of the projects and programmes

with regard to overarching results (impact) and

sustainability lead to the conclusion that the

actual implementation of these approaches still

leaves room for improvement. In the opinion of

the evaluators, the underlying contradictions

should be addressed somewhat more candidly.

Regarding the rather unbalanced consideration

given to the social dimension, greater attention

to poverty studies and/or target-group analyses

at the planning stage could yield improvement.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 97

6. Sectoral assessment

6.1 Cooperation and task-sharing

among German interface or-

ganisations

Table 23 below surveys the cooperation be-

tween and coordination of German DC inter-

face organisations in all fifteen evaluated pro-

jects and programmes. A basic distinction is

drawn between 'cooperation', which normally

takes place within the scope of a cooperation

agreement and involves common goals,

strategies and areas of responsibility (tasks),

and 'coordination', in which both sides keep

each other informed about interventions and

planning.

As the table shows, more than half of the pro-

jects and programmes involve cooperation with

KfW, including formalised cooperation in seven

'cooperative projects/ programmes'. The Ger-

man Development Service (DED) is involved in

four projects/programmes via cooperation

agreements, while CIM (Centre for Interna-

tional Migration and Development), the Federal

Institute for Geosciences and Natural Re-

sources (BGR) and Inwent - Capacity Building

International participate by way of cooperation

and coordination.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 98

Table 23: Cooperation between the evaluated projects/programmes

and German DC interface organisations

Country Project/

Programme CP KfW DED CIM BGR Inwent

Africa

supra-

regional

NIW

Algeria PDEA discon-

tinued16

phase I

only

Yemen WSP X X X X X X

Jordan WMIA (X) (X) (X) (X) (X)

Kenya WSRP-KEN X X X (X)

Peru PROAGUA X (X) (X)

Zambia WSRP-ZMB X X X X

Viet

Nam WDPC

X X X X

Ghana EVORAP X X

Kosovo WSWDP (X)

Mexico GICA (X)

Ethiopia IWSTR

India IGWDP X

Niger PHV/MI X X

Turkey QADI

X Cooperation (X) Coordination of activities

16 During phase I the programme was established as a cooperative programme with KfW. The results of components 2 and 3

were arranged directly with KfW, and objectives achievement was dependent on FC. The cooperation was discontinued

when German FC was withdrawn from Algeria.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 99

The 'Water Sector Reform Programme,

Zambia' stands as an example of complex

relationships between German DC organisa-

tions. Working on the basis of a formalised

cooperation agreement, GTZ, KfW, DED and

CIM try to establish synergies in their interven-

tions and, above all, to avoid the delivery of

overlapping services. DED and CIM focus

primarily on providing advisory services to

WUs and local water authorities (operation,

monitoring), while KfW promotes the develop-

ment of infrastructure. Simultaneously, utility

employees receive targeted training through

Inwent measures. In a different programme

component, BGR provides assistance for the

management of groundwater resources. With-

out such cooperation, the objectives could

never be achieved.

In projects and programmes involving coopera-

tive relationships between German interface

organisations, joint achievement of objectives

calls for a high level of coordination and joint

planning. In most of the projects/programmes,

the partners in cooperation meet those re-

quirements. It is important in this connection to

unequivocally clarify who is responsible for

lead management. Experience drawn from the

'Rural Water Supply in Maradi, Niger' project

shows that vague responsibilities and the lack

of a clearly assigned control mandate for the

GTZ or the KfW can be very disadvantageous

in terms of objectives achievement and, above

all, efficiency.

In view of the complex cooperative relationship

between the German interface organisations

involved in the 'Institutional Development of

the Water Sector, Yemen' programme, and

considering the fact that three German consult-

ing firms were also involved in the programme,

the evaluators had reservations about the effi-

ciency of such a structure. The question arises

as to whether services rendered by the BGR

could perhaps be provided at lower cost by

local or German consultancy firms.

Especially for non-formalised cooperation, the

evaluation reports identify problems concern-

ing the coordination of interface organisations.

Usually, 'coordination' hardly extends beyond

simple 'information', i.e. no real synergetic

interlinkage takes place between interventions

in these projects/ programmes. Hence, coop-

eration with the KfW, for example, is at the

same level as with other donor institutions,

namely that of non-committal exchanges of

information that do not enable concerted action

in the interest of German DC regulations and

directives.

Despite the heterogeneity of the assessments

contained in the evaluation reports, progres-

sively formalised cooperation has improved the

image of 'joined-up development cooperation'.

Nevertheless, the 'coexistent' activities of

German DC actors persist in many places.

There remains a lack of clear-cut steering

mechanisms with which the actors would be

able to coordinate their measures even in the

absence of formal contractual relations.

6.2 Embedment of development

measures in the activities of the

international donor community

The fact that the evaluated projects and pro-

grammes generally received good relevance

ratings indicates that their underlying concep-

tual designs are highly integrated into the na-

tional sector strategies and synergize with the

interventions of the international donor com-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 100

munity. Maximum embedment in concerted

assistance measures is a decisive success

factor with regard to relevance. At the same

time, this is an area that, basically, receives

very positive assessment for all evaluated

development measures.

As analysed by the authors of the cross-

section evaluation, however, the following dis-

tinction must be drawn: for a large share of the

projects and programmes, the causal relation-

ship between the strategic orientation of the

respective project or programme and the

common objectives and results dimensions of

the donor community interventions of a given

country is largely theoretical. That alone, how-

ever, does not constitute effective embedment.

It also becomes apparent that, basically, coor-

dination only takes place within projects and

programmes characterized by a strong macro-

level presence. In reality, projects and pro-

grammes that principally act at the meso or

micro level – including most of those dealt with

in the present evaluation sequence – fail to be

involved in donor coordination and harmonisa-

tion, even though this would be easily feasible.

It was also noticed that there is no ongoing

cooperation between the 'Management of

Water Catchment Areas for the Río Lerma

and Toluca Valley and Río Balsas, Mexico'

project and other activities of the international

donor community, since these have different

regional and/or thematic priority areas. Accord-

ingly, the project has no harmonised sector-

wide approach.

An indicator for genuine cooperation within the

international community should be the exis-

tence of joint planning and interventions. This

holds true for three (3) of the evaluated pro-

jects/programmes (Yemen, Kenya and Zam-

bia):

'Institutional Development of the Water Sec-

tor, Yemen': Both the presence and the profile

of the international donor community within the

sector are high. Within the scope of its own

advisory activities, the programme works to

update and implement the sector policy and

strategy and to cooperate closely with the KfW

in the development of programme-oriented

joint financing (SWAp). The 'Water Sector

Support Programme' (WSSP) is a preliminary

form of SWAp. The field testing of innovative

methods and approaches, as called for, inter

alia, by the sector strategy was incorporated

into the programme by way of the 'milestones

concept' introduced in cooperation with the

KfW. Its further development is being facilitated

by SWAp and other tools. The fact that the

programme is very much present at all relevant

formal and informal decision-making and im-

plementing levels – hence yielding tangible

synergies – was given a particularly positive

rating.

'Water Sector Reform Programme, Kenya':

The international donor community is focusing

its cooperation on the development of a SWAp

for the country. All donor activities in the Ken-

yan water sector are being coordinated by a

steering group headed by the KfW in collabora-

tion with Swedish and French DC. Within that

scope, the GTZ supported, among other

things, the operationalisation of a SWAp com-

mittee, which then set up the rules and princi-

ples of cooperation for the sector. All this is

now anchored within the framework of the

cooperation agreement entered into by the

donor organisations. Formalised cooperation

between the GTZ, Danish and Swedish DC

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 101

and UNICEF is being managed by a joint

committee that initiates joint budget planning

and operations planning on the basis of the

sector strategies, among other things.

6.3 Implementation of the multi-

level approach in advisory pro-

jects and programmes

GTZ pursues a holistic approach. To achieve

this, social, economic and ecological aspects

are combined in the system of objectives. Ad-

visory support of a sectoral, political and or-

ganisational nature is provided at the macro

level, the meso level and the target group-

oriented micro level. The GTZ‟s work is based

on a values concept comprising the promotion

of democracy, the rule of law, good govern-

ance, human rights and a social and ecological

market economy. The GTZ‟s process-oriented

approach is geared to providing help towards

self-help, the transparent representation of

stakeholder interests, and mediation between

the state, civil society and the private sector. In

so doing, GTZ's goal is to embed its develop-

ment measures in the activities of international

donors and to apportion the tasks among all

implementing organisations. This attaches

importance to cooperative projects and pro-

grammes marked by complementary technical

and financial cooperation. The IWRM approach

plays a special role in the water sector and

should be integrated into all development

measures.

The assessment of the underlying approaches

is handled differently from case to case in the

reviewed evaluation reports. Assessment of

the technical and methodological approach is

not uniformly systematic. While some evalua-

tors attach much importance to a detailed

analysis of the methodology and investigate all

the various approaches taken, other reports

make do with a more comprehensive, abstract

assessment. In most cases, the strengths and

weaknesses of a measure's conceptual design

are reflected in the effectiveness rating.

As already mentioned in Sections 2.2.2 and

4.3.1, only three programmes (Yemen, Kenya

and Zambia) were found to have a complete,

conceptually well-balanced, multi-level ap-

proach mainstreamed at all impact levels.

In the project entitled 'Nile Water Initiative -

Planning and Management of Water Re-

sources in the Nile Basin', the design of the

methodological approach was of limited flexibil-

ity, because the project had to be closely

geared to the existing organisational structures

and donor landscape. All in all, the project

aimed to harmonise the water policies of the

member states and, hence, to effect change at

the policy-making level. In this project, the

multi-level approach was implemented within

the regional context, that is, the macro level is

occupied mainly by international committees

and working groups.

In the 'Improvement of Water Supply in the

Tigray Region, Ethiopia' project, the concept

of working first in eight pilot towns and then in

19 replication towns was rated 'good', and the

combination of competence building with the

direct GTZ contribution was emphasised as

particularly positive thanks to the effective

combination of theory and practice. The envis-

aged multi-level approach was, in principle, not

implemented, because the project focused

more on advising the Water Resource Devel-

opment Bureau than on addressing the re-

gion's population. On this point, the evaluators

recommend a stronger micro-level orientation.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 102

At its core, the 'Integrated Water Manage-

ment Programme, Algeria' is a policy advi-

sory programme with an IWRM approach

geared to improving governmental perform-

ance at the national, regional and local level.

Its priority aim is to foster capacity develop-

ment among the staff of the sector institutions.

There are no plans to aim the programme

more toward a sector-wide approach (SWAp).

The programme is active at the national, re-

gional and local level (multi-level approach).

Close cooperation with the population facili-

tates exchanges between the administration

and the water users themselves. The working-

group approach was given an emphatically

positive rating as an important instrument of

institutional learning via cooperation with the

partner.

The 'special evaluation issues' chapter of the

final evaluation report on the 'Improvement of

Water Supply in the Eastern and Volta Re-

gions, Ghana' programme deals with issues

concerning the measure's basic concept. The

priority area of this development measure is

the provision of drinking water in small rural

towns by way of locally managed 'water and

sanitation development boards' established

within the scope of the programme. The pro-

gramme implemented a holistic approach to

the provision of technical, organisational and

policy advisory services. The multi-level ap-

proach includes advising and supporting the

sector institutions at the national and regional

level and assisting the district and municipal

levels. The interlinkage of the objectives di-

mensions was analysed in detail. The process-

and value-oriented approach also comes to

bear in this programme. Its local-level help

towards self-help and efforts to foster the de-

centralisation process were seen in a particu-

larly positive light, as were its close coopera-

tion and interfacing with the water programme

in Benin. The instruments and advisory ap-

proaches developed there for the areas of

decentralisation, operation and maintenance of

provincial water supply systems and knowl-

edge management were further developed and

modified for Ghana. The fact that this pro-

gramme invested little effort in local-level cam-

paigns and public awareness activities was

rated negatively, as was its lack of adequate

wastewater management aspects.

By contrast, the 'Rural Water Supply in Ma-

radi, Niger' project consciously dispensed with

the concept of a multi-level approach. Its prior-

ity was to work at the micro level and build a

strong relationship with the target group. The

holistic, participatory approach in the sense of

'integrated water resources management' in-

volving a systematic balancing of economic,

social and ecological objectives was not of

primary interest. Cooperation between the KfW

and the GTZ within the scope of this coopera-

tive project was given a negative rating. Nei-

ther was the conceptual design (technical

models, operation and maintenance concepts)

jointly adjusted, nor were the objectives and

indicators standardised, and no systematic,

collective planning of operations and work took

place. Finally, due not least to protracted per-

sonal conflicts between the participating ex-

perts, it was not possible to establish an effi-

cient, direct communication structure and cul-

ture between the FC and TC projects.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 103

6.4 Utilisation of GTZ's knowledge

management methods

The evaluation reports under review are not

very informative with regard to GTZ knowledge

management. The majority of the reports make

no mention of it at all. Most of the few evalua-

tions that do broach the subject (Africa su-

praregional, Algeria, Peru) confirm that the

staff of the respective project or programme

took part in the regional sector networks and

staff conferences, where they presented the

results of their projects/programmes to the

respective working groups. Participation in the

sector networks is generally described as a

worthwhile instrument of exchange and infor-

mation management. On the other hand, all

reports qualify that assessment to the extent

that an actual exchange of technical informa-

tion in the sense of best-practice dissemination

to the benefit of the projects and programmes

rarely takes place. The single positive example

of relevance is the dissemination of strategic

planning and monitoring instruments devel-

oped within the scope of the 'Drinking Water

and Sanitary Programme, Peru' by the sector

network and the staff conference.

Indeed, the subject of knowledge management

is given very scant treatment within the pro-

jects and programmes. On this point, the

evaluators see major potential for improve-

ment, but their reports offer no concrete ap-

proaches or proposals.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 104

7. Contract and cooperation man-

agement

7.1 Modes of delivery

Primarily in connection with efficiency ratings

for the reviewed projects and programmes, the

mode of delivery played an important role in

the relevant evaluation reports. As a specific

issue, the modes of delivery employed in all

fifteen evaluated projects and programmes are

significant in that twelve of the pro-

jects/programmes involved more or less exten-

sive inputs from consulting firms acting on

behalf of the GTZ. In one case (Niger), GTZ

International Services (GTZ-IS) delivered ser-

vices as a KfW contractor.

Nevertheless, the reports make little reference

to the respective mode of delivery. Often, the

service delivery structures and/or the involved

organisations/companies are not even named.

The following table provides an overview of

contractors and subcontractors involved in the

individual projects and programmes. It should

be noted in this connection that the information

provided was obtained by way of supplemen-

tary research, because the documentation in

the reports was incomplete.

Table 24: Contractors and subcontractors involved in the evaluated

projects/programmes

Country Project/

Programme Contractor / Subcontractor

Africa su-

praregional NWI GTZ

Algeria PDEA GTZ / AHT

Yemen WSP GTZ / GOPA, GFA, Dorsch Consult

Jordan WMIA GTZ

Kenya WSRP-KEN GTZ / GFA

Peru PROAGUA GTZ / RODECO, AKUT

Zambia WSRP-ZMB GTZ / RODECO

Viet Nam WDPC GTZ / GFA

Ghana EVORAP GTZ / RODECO

Kosovo WSWDP GTZ / RODECO

Mexico GICA GTZ / RODECO

Ethiopia IWSTR GTZ / GFA, RODECO

India IGWDP GTZ / RODECO

Niger PHV/MI GTZ (TZ) – GTZ-IS (FZ)

Turkey QADI GTZ

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 105

In their reports, the evaluators were very re-

served in their assessment of service delivery,

particularly in those pertaining to the interim

evaluations. For example, the interim evalua-

tion of the 'Integrated Water Management

Programme, Algeria' noted that, due to the

large share of external experts, the evaluator

assignments were not optimally adjusted to the

needs of the programme. Problems were per-

ceived primarily in terms of inadequate qualifi-

cations (language proficiency) and difficult

incorporation into the programme structures

(flow of information, steering, consultation). In

the ex-post evaluation report on the 'Im-

provement of Water Supply in the Tigray

Region, Ethiopia' project, service delivery by

the consulting firm commissioned for the first

phase of implementation was described as

inadequate, but no concrete reasons were

stated.

Apparently, the 'Drinking Water and Sanitary

Programme, Peru' yielded good experience

concerning the integration of qualified national

consultancy firms for the area of long-term

organisational consultancy. Particularly with

respect to process orientation and sustainabil-

ity in the support of institutional change proc-

esses, continuous national-evaluator assign-

ments are regarded as more advantageous

than the seconding of external evaluators.

In the interim evaluation report on the 'Institu-

tional Development of the Water Sector,

Yemen' programme, the award of a consider-

able contract volume to outside consultants

was not always regarded as appropriate and

efficient due to the high attendant fixed costs

and commitment of funds. While the pro-

gramme did attempt to be economically effi-

cient by awarding block contracts for self-

contained tasks and components, hence avoid-

ing fragmentary, cost-intensive micro-

management, this approach harbours the risk

that the requisite networking with German DC

institutions may not take place on a sufficient

scale and that the knowledge of procedures

could remain inadequate.

The network of consultancy services from the

TC and FC portfolios is regarded as being in

general need of improvement. For most parts,

the joint planning of interventions and the har-

monisation of procedures are still problematic.

Moreover, delayed outflow of funds due to the

formulated joint achievement of objectives has

a negative influence on both the effectiveness

and the efficiency of the TC measures.

The 'Rural Water Supply in Maradi, Niger'

project constitutes a special case in this con-

text. Here, the FC consultancy services were

contracted directly from GTZ IS. The TC advi-

sory services were rendered by the GTZ's own

personnel. Due to substantial 'static' between

the FC and TC personnel, this project never

reached the point of appropriate joint action.

This was primarily due to steering problems

that could not even be rectified within the GTZ.

As a result, major conceptual deficits rendered

the measure unsuccessful, obviously due to

the steering problem.

7.2 Steering and cooperation with

partner institutions

The greater share of the evaluated projects

and programmes intervene in a very complex

partner structure. In many places, institutional

responsibilities for sub-sectors and/or compo-

nents are spread over a number of different

sector authorities. The reviewed evaluations do

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 106

not explicitly deal with the resulting steering

challenges in individual cases. Apart from

three projects/programmes operating within the

SWAp context, clear-cut steering mechanisms

are the exception. The challenge for TC actors

is to achieve expedient, institutionalised net-

working between the participating partner insti-

tutions. Often, this requires 'negotiations' con-

cerning strategic orientation.

Relevant weaknesses result from the fact that

most of the projects and programmes have a

strongly established focus on the meso level.

Their influence on the evolution of steering

structures remains small, and numerous

evaluators perceive it as a constraint. Here, the

observed weaknesses in the implementation of

systematic capacity development interventions

impede progress precisely at the upper results

level (meso to macro level).

The basic question to be identified from the

evaluation reports is 'Who should actually be

doing the steering?'. Here, the huge demands

being placed on the partner-structure person-

nel cannot be met at present by the corre-

sponding counterparts. Hence, one of the main

future tasks should be to promote steering

competencies and capacities by means of an

explicit multi-level approach. As recommended

by many of the evaluators, some important

steps in that direction should be taken at the

planning stage. A more systematic and critical

analysis of the strengths and weaknesses of

the respective partner structures should be

taken as a point of departure for mainstream-

ing capacity development approaches into the

project concept from the very beginning. In

addition, the partner structure should only have

to contend with a scope of tasks that it can

actually 'absorb'.

The factor 'confidence building', which is a

major topic in the evaluation reports, entails

maximum possible closeness to the partner,

also in the physical sense. A number of pro-

jects and programmes still have some potential

for further optimisation in that respect.

What is referred to as the institutionalisation of

monitoring, including the documentation of

results beyond quantitative parameters, pro-

vides a further basis for effective steering of

complex projects and programmes. Some

good examples of useful approaches to moni-

toring were described in the preceding sec-

tions. Nevertheless, most of the reviewed pro-

jects and programmes remain open to criticism

with regard to inadequate or poorly sustained

monitoring systems. In that respect, the infor-

mational content of the reviewed evaluations is

limited. Monitoring serves not only purposes of

project steering and accounting, but also and

above all else for initiating learning processes

from which, ultimately, strategic and concep-

tual adjustments can emerge. Results, lessons

learned and best practices should be recorded

and evaluated together with the partners. Part-

ners who receive too little or no such informa-

tion cannot fully discharge their steering man-

dates.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 107

8. Summary of conclusions and

lessons learned

8.1 Summarising conclusions

As part of the 'independent evaluations' in-

strument, the performed cross-section analysis

of the evaluations in the thematic priority area

'water' should make a valuable contribution to

discharging GTZ's accountability obligations. It

does not amount to an overview of a selected

group of projects and programmes considered

representative of the GTZ water portfolio, but

comprises exemplary evaluations which in their

entirety cover a very broad sector-specific

spectrum. Nevertheless, the compiled results

can provide information of interest for steering

and learning.

The analysed evaluation reports are not al-

ways fully comparable with each other, and

they had different methodological and content-

related priority areas. Each evaluation report

constitutes a discrete, very subjective assess-

ment and analysis of the respective context.

Nevertheless, some general conclusions can

be drawn and taken as a basis for further dis-

cussion.

Apart from a summary of performance as-

sessments, this synthesis report offers some

important points of reference particularly with

regard to the methodological orientation of the

evaluated projects and programmes and their

attention to the principles and concepts of

German DC: poverty relevance, consideration

of the principles of sustainable development

and systematic furtherance of performance

capabilities within the partner structure. These

are major cross-cutting themes into which the

evaluation can yield new insight.

Still, the ability of a cross-section evaluation to

describe and evaluate the concrete scope and

content of service delivery in the evaluated

projects and programmes remains limited.

Very little pertinent information is contained in

the individual evaluation reports. The synthesis

report therefore confines itself to compiling

some typical, readily understandable examples

of successful and unsuccessful implementa-

tion.

Analysis of the fifteen evaluations yields a

'successful' broad-brush picture of the imple-

mented projects and programmes. The analy-

sis shows good relevance of the development

measures to be the main success factor. At the

same time, it becomes obvious that the docu-

mentation of results achieved in the selected

group of projects and programmes is still too

poorly developed. There is reason to suspect

that the TC projects and programmes actually

achieve many more results than can be identi-

fied and recorded with the existing range of

instruments. This could be one reason for the

merely 'satisfactory' assessment of objectives

achievement. Another problem is the still rather

weak sustainability of the achieved results. On

this point, the evaluators see a crucial connec-

tion between the measures' strong meso-level

focus and the existing weaknesses in the

structuring of systematic capacity develop-

ment. In the presented sub-portfolio, system-

building and networking interventions are still

too marginal.

Most of the evaluated projects and pro-

grammes lack the kind of clear orientation

towards poverty reduction that is called for by

the BMZ sector strategies and the Millennium

Declaration, among others. This is not neces-

sarily due to false conceptual design of the

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 108

projects and programmes, and it does not

mean that results with regard to poverty reduc-

tion are not achieved. The projects and pro-

grammes could, however, more plausibly por-

tray this orientation by more closely analysing

the target groups and profiling them in the

system of objectives. A measure's contribution

to poverty reduction and to the achievement of

MDGs can be more plausibly substantiated

through the formulation and systematic moni-

toring of target group-oriented, pro-poor indica-

tors. A poverty analysis would help illuminate

the existing opportunities for achieving direct

results – opportunities that should be made

use of by targeted activities.

Important lessons learned, as a synthesis of

conclusions drawn in the individual evalua-

tions, were compiled in the respective parts of

the DAC criteria analysis (see Sections 3.1 –

3.6) to form 'success factors' and 'factors of

failure'. The following section recapitulates the

most important conclusions and lessons

learned from the evaluation reports.

8.2 Lessons learned from the

individual evaluations

8.2.1 Relevance of the development

measures

With regard to the 'relevance' of development

measures, it proves to be of decisive impor-

tance that the respective conceptual design

and formulation of objectives achieve a maxi-

mum degree of concordance with the policies

and strategies of the respective partner coun-

tries and partner institutions. At the same time,

the thematic priority areas of the BMZ in the

water sector should also be covered by the

conceptual design, and systematic orientation

towards sector-specific policies, country

strategies and position papers (basic develop-

ment-policy orientation of the German Gov-

ernment) should be established. It is also very

important, especially for technical planning,

that regard be paid to international standards

and conventions.

The 'relevance' rating ultimately reflects the

amount of priority attached to the respective

measures both by the partners and by the

target group. In that connection, one very es-

sential criterion is whether the target groups

and partner institutions participated in the

planning of the relevant project or programme

and whether their needs were systematically

incorporated into the concept. Experience

drawn from the present sequence of evalua-

tions shows that very many development

measures are still being designed without a

proper target-group analysis or differentiation.

This does not only impair the relevant orienta-

tion of the projects and programmes, but also

fails to build a basis for identifying and re-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 109

cording the dimensions of the results at the

target-group level.

Concerning the requisite poverty relevance of

water sector projects and programmes, poverty

analyses need to be performed at the planning

stage and taken as a basis to establish poverty

reduction as part of the formulation of objec-

tives in accordance with the poverty reduction

strategies. Projects with no – or no verifiable –

poverty orientation and no underlying poverty

analysis rarely achieve good relevance. In

equal measure, consideration of the gender

dimension and environmental aspect in the

project design, on the basis of appropriate

investigations and/or analyses, is a crucial

success factor.

With regard to water resources management, a

recognisable attempt to embed that area in

existing or planned regional cooperation has

proven to be a critical success factor for the

relevance of the respective projects and pro-

grammes. Integrated and cooperative ele-

ments should be 'evident', and transboundary

initiatives should give holistic consideration to

the region in question.

The nature of a multi-level approach is also

considered to be very important. The extent to

which it can actually be implemented as

planned is decisive. Precisely the implementa-

tion of interventions at the macro and micro

levels is a critical success factor. The present

evaluation reports show that implementation at

just those levels is difficult and that too few

interventions are planned for the target-group

level. This can lead to conflicting objectives in

connection with impacts on poverty.

8.2.2 Effectiveness of the develop-

ment measures

The basic prerequisite for the successful

achievement of the objectives of development

measures is a shared understanding of objec-

tives and cooperation between the partners

and the experts entrusted with implementation.

It is important that both the partner experts

(counterparts) and the respective target group

understand and accept the measure's benefits

and that they exercise their rights and carry out

their duties.

It is also important in this connection for TC

and FC in cooperative projects to be optimally

interlinked, so that synergy effects can be gen-

erated and the results of TC measures can be

amplified by FC investments, and vice-versa.

Experience to date shows that this is best

achieved in projects and programmes with

formalised cooperative relationships, whereas

it is also important that the coordination of

interventions by DC actors be accompanied by

the harmonised use of resources. As a rule,

this requires the transparent (also for the part-

ner) and generally accepted awarding of a

steering mandate to one of the institutions

included in the cooperative structure.

In view of the dynamic political and institutional

settings within which the projects and pro-

grammes are implemented, it has proven es-

sential with regard to effectiveness that guide-

lines governing the cooperative implementation

and best-practice dissemination be formulated

together with the partner structure and formally

recognised. This includes adapting and further

developing the technical concept within the

course of the implementation phase, if this may

become necessary.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 110

With regard to the creation of durable capaci-

ties and structures at the implementing-partner

level, the formulation of transparent capacity

development strategies geared to system-

oriented results has emerged as an essential

success factor. In this respect, the national

water policies and sector strategies often de-

fine the general guidelines. The scope, quality

and coherence of the offered package of train-

ing and capacity building must be made trans-

parent to the partners. It is also important to

enable better coordination of supply and de-

mand by way of systematic needs assess-

ments.

Simultaneously, the duration of the advisory

services and, above all, due allowance for

follow-up assistance phases count as crucial

factors for successful achievement of objec-

tives in TC projects and programmes.

8.2.3 Impact of the development

measures

The present evaluations show that the rating of

overarching development results in the major-

ity of the projects and programmes only ap-

pears as an approximation based on assump-

tions and abstractions via lengthy results

chains. It was determined in that connection

that the aggregated results to be achieved by

the respective project or programme are often

unrealistic: excessive expectations (level to

which the objectives are to be achieved, very

long results chains). In formulating systems of

objectives, it is therefore important to plausibly

substantiate the measure's contribution to

overarching results and to formulate the over-

arching results with no room for interpretation.

Overarching results were successfully docu-

mented in projects and programmes in which

not only qualitative assumptions but also quan-

titative statements (perhaps based on a base-

line study) could be made. In that respect, it is

regarded as important that a contribution be

made to poverty reduction and/or to MDGs and

that the target groups confirm in surveys that

there has been a qualitative improvement in

their situation and/or their living conditions.

Of decisive importance in that same context

are such structural and systemic results as the

decentralisation of WS&S and/or the estab-

lishment of successful NGOs at the target-

group level and the further pursuance of sector

reform beyond the level of objectives achieve-

ment (positive effects in other areas of the

broader sectoral and regional context). It is

also important for role-model functions to

emerge and for the communicated approach

and concepts to become common property.

8.2.4 Efficiency of the development

measures

The crucial success factor for the efficiency of

the evaluated projects and programmes was to

embed DC interventions within the sector in-

terventions as a whole. Importance was at-

tached both to coordination between the do-

nors themselves and/or the German interface

organisations and to complementary appor-

tionment of working areas. Systematically util-

ising relevant potentials and synergies at the

various levels is most promising when the cor-

responding cooperation arrangements are

formalised. Most notably, the harmonisation

and coordination of resource utilisation be-

tween the German DC organisations has fre-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 111

quently remained inadequate or poorly man-

aged.

Beyond this, the personnel concept must be

convincing, i.e. the services of internal and

external short-term and long-term experts must

be coordinated with each other, and synergies

must be fostered through well-developed

knowledge management. The complexity of

the personnel concept must have no negative

effects on the cost-effect ratio. Where neces-

sary, preference should be given to alternate

modes of delivery (such as calling in qualified

local consultants or NGOs).

Another critical success factor that came to

light is the initiation, in the course of implemen-

tation and after having been approved by the

BMZ, of any necessary adjustments to the set

objectives or conceptual design, a typical case

being the non-provision of investment re-

sources in cooperative projects (outgoing

funds problems in FC).

8.2.5 Sustainability of the develop-

ment measures

Sustainability is seen as an important criterion

for performance assessment, also in the

course of implementation. It is decisive that all

objectives dimensions (economic, social, politi-

cal and ecological) be incorporated into the

conceptual design of the respective measures

in a timely and well-balanced manner, and that

structural risks be addressed in further-

reaching interventions.

The design of systematic capacity develop-

ment approaches of a structure- or system-

building nature lays the foundation for sustain-

able achievement of objectives. This includes

establishing participatory structures at the tar-

get-group level (micro level), hence enabling

the target groups to participate in dialogue

concerning the use of water resources. On this

point, acceptance on the part of the lead exe-

cuting agency must either already exist or be

brought about. Indeed, with regard to the train-

ing of partner personnel, there appears to be a

'critical mass', that is, training a sufficient num-

ber of personnel in the sector counts as a suc-

cess factor.

One other crucial success factor for develop-

ment measures is the lead executing agency's

willingness to accept advice in connection with

higher political objectives (macro level) and to

implement new laws and regulations governing

such things as break-even tariffs.

Durably effective operational structures for

major supply systems in poor settings can only

be built up slowly and with an accordingly large

volume of advisory inputs. The targeted supply

standards can be raised within a relatively

short time through investments and/or techni-

cal interventions. Institutional advances in de-

velopment alone rarely allow sustainable op-

eration as soon as the technical goals have

been achieved. In many of the evaluated pro-

jects and programmes, this problem poses

risks for the sustainability of achieved objec-

tives. In this same connection, time require-

ments and limited influence on handed-down

modes of sociocultural behaviour in the use of

water resources are of utmost importance.

Prevailing framework conditions make it virtu-

ally impossible to synchronise the achievement

of objectives in terms of technical supply ca-

pacities, establishment of operating structures

and changes in the behaviour of the target

group. Nevertheless, such synchronisation is

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 112

frequently adopted as a basis for the achieve-

ment of objectives.

The reviewed evaluation reports call attention

to the fact that too little time is allowed for es-

tablishing or decentralising capacities, per-

formance capabilities and/or change proc-

esses. It proves successful in this respect to

lengthen the advisory cycles in order to provide

process support and to include follow-up assis-

tance phases in the planning (particularly for

FC involvement).

Sustainable achievement of objectives also

relies on institutionalised knowledge and

methods at the implementation structure and

target group levels. It is therefore important to

promptly document lessons learned from the

projects and programmes and to integrate

them into the partner organisation's knowledge

management. In this connection, examples of

best practice can be compiled and dissemi-

nated.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 113

9. Recommendations

9.1 General recommendations

The results of this cross-section evaluation of

independent evaluations in the thematic area

'water' show that evaluation as an instrument

can make a relevant contribution to the con-

ceptual design, steering and quality assurance

of TC projects and programmes. Of decisive

importance in this respect is that evaluations

and their results be taken into account together

with project progress reviews for the steering

of ongoing projects and programmes within the

GTZ and for planning new ones. It is recom-

mended that more space be devoted to the

recording and analysis of relevant cross-cutting

development themes in the evaluations. Cru-

cial lessons learned, particularly with regard to

concepts and methods, could be drawn from

this 'qualitative' part of the evaluation reports, if

it were more strongly emphasised and system-

atically processed than was the case in the

current evaluation sequence.

Since independent evaluations are only per-

formed in two thematic areas each year, there

will be no comparable evaluation sequence in

the water portfolio within the near future. It is

therefore recommended that key issues from

the evaluation be integrated into routine project

progress reviews. In the opinion of the evalua-

tors, closer attention to reflections on the DAC

criteria impact, efficiency and sustainability, in

particular, together with a critical examination

of the cross-cutting development themes cov-

ered by the PPRs, would greatly improve the

steering and planning bases. It is also recom-

mended that additional ex-post evaluations be

performed for projects and programmes of

strategic importance in order to intensify the

lessons learned. Especially the more recently

designed, complex sector projects and pro-

grammes should be able to yield key insights

into the achievement of objectives and results

and about successful methods and ap-

proaches.

The results of the evaluations also reveal that

the understanding of cross-cutting develop-

ment themes, their importance within the Ger-

man DC context and their underlying concepts

and guiding principles is still very heterogene-

ous. These cross-cutting themes should be

given more attention in 'day-to-day implemen-

tation' and, to some extent, also in the planning

of measures. The GTZ‟s knowledge manage-

ment tools can make a valuable contribution,

e.g. by addressing cross-cutting themes more

pointedly in sector networks and at staff con-

ferences.

The key performance monitoring instrument for

all projects and programmes is substantiation

of their results orientation in line with the re-

spective conceptual design. On the basis of

the reviewed evaluations, it became evident

that determination of results in the 'day-to-day

implementation' of projects and programmes

has been something of an exception to the

rule. Moreover, the results of projects and pro-

grammes are also still seen to be playing a

very subordinate role in the established form of

reporting. Against that backdrop, it would be

advisable to bring the instrument of project

progress reporting more closely in line with the

requirements of results-based performance

monitoring. Simultaneously, project progress

reporting should include a portrayal and dis-

cussion of methods, approaches and models

and perhaps disseminate them as best prac-

tices. In the view of the evaluators, the TC

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 114

profile could be further sharpened by incorpo-

rating successful approaches (e.g. the capacity

development strategy, best advisory practices,

identification of lessons learned) into reports

for clients.

On the basis of findings from the evaluation

reports on which this cross-section evaluation

is based, and from the corresponding offers

submitted to the BMZ by the GTZ, the system

behind and coherence of assigning DAC

markers is seen to be in need of improvement.

The appropriate assignment of markers is es-

sential for classifying and evaluating the re-

spective projects and programmes. It is there-

fore recommended that the assignment of DAC

markers be more systematically aligned with

the objectives systems formulated for the pro-

jects/ programmes. In order to reduce the ex-

isting discrepancies, the respective relevance

should be checked and reassessed as neces-

sary in connection with evaluations and ad-

justments to plans.

Regarding capacity development, it is recom-

mended that a more uniform definition be ob-

served for purposes of routine implementation

and evaluations.

The success factors identified in Section 3.1.1

for the relevance of a project or programme

should, as far as possible, be given full consid-

eration when adjusting ongoing pro-

jects/programmes and in the conceptual de-

sign of new ones. In the evaluators‟ opinion,

these factors could also be retroactively inte-

grated when adjusting ongoing pro-

jects/programmes.

9.2 Recommendations for steering

ongoing

projects and programmes

For ongoing projects and programmes, too, the

extent of agreement between the formulated

objectives systems and the respective underly-

ing classification of projects and programmes

according to the DAC markers should be sys-

tematically investigated. This could be done in

connection with PPRs. The decisive factor

should be a current assessment of the realistic

(achievable) results of the respective pro-

ject/programme, which usually deviates from

the assessment made when the offer was writ-

ten. Classification of the projects and pro-

grammes according to the DAC criteria and the

adjustment of indicators for recording the re-

sults should then be based on that assess-

ment.

Critical reflections on the actual results levels

and objectives dimensions of a given project or

programme are recommended. If any discrep-

ancies are noted between the underlying con-

ceptual designs and the 'realities of implemen-

tation', appropriate adjustments should be

made. It is recommended to attach special

priority to the consistent implementation of a

multi-level approach and to alignment with the

principle of sustainable development.

It is also recommended that the existing sys-

tem of indicators be simplified and supple-

mented with measurable results factors. Some

examples of comparatively uncomplicated

approaches were already given in the preced-

ing sections. Here, too, the monitoring systems

may need to be adjusted and possibly stan-

dardised accordingly. Next, in the opinion of

the evaluators, it is of decisive importance that

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 115

the project/programme reporting include refer-

ence to the results factors. It is recommended

that GTZ's sectoral structure request or moni-

tor appropriate reporting. Internal resources

(personnel, available best practices) should be

used to provide advice on adjusting and stan-

dardising monitoring systems. Here, the GTZ

knowledge management system may serve as

a basis.

As far as the capacity development strategies

of projects and programmes are concerned, it

is recommended that transparent capacity

development concepts be elaborated and har-

monised, and adjusted to the available re-

sources of the respective projects and pro-

grammes. Such concepts should be consis-

tently aligned with the existing/adjusted objec-

tives definitions and, if necessary, focus more

on system-building and networking interven-

tions in order to meet the requirements of ca-

pacity building in the partner structures. In this

area, too, it is recommended to make use of

internal technical expertise for delivering advi-

sory services.

In addition, all existing means should be em-

ployed to achieve further formalisation of exist-

ing cooperative relationships (German and

international DC) within the context of the pro-

jects and programmes, as this provides a very

valuable basis for enhancing both the pros-

pects of success (mainly in terms of efficiency)

and the results of the measures. The goal

should be to achieve more real 'cooperation'

beyond the signing of corresponding agree-

ments.

9.3 Recommendations for planning

new projects and programmes

The analysis and/or differentiation of target

groups and of the poverty, gender and envi-

ronmental dimensions should be systematically

taken into account before actually designing

and planning projects and programmes. In the

opinion of the evaluators, this calls for specifi-

cations to be made by the sectoral structure

and for adjustment of the planning steps. It is

also recommended in that context to ask more

insistently for the relevant studies. In this re-

spect, the recommendations from the individ-

ual evaluations can be summarised as follows:

Poverty analyses should identify target

groups and correspondingly target group-

appropriate activities that contribute to

poverty reduction;

Especially in the world's poorest coun-

tries, the main focus should be on poverty

reduction;

Poor sections of the population should be

more closely integrated into political proc-

esses;

The decentralisation of outputs should not

focus merely on economic aspects, com-

mercialisation and privatisation, but also

on sociocultural aspects and pro-poor

outputs;

Tariff studies should pay particular atten-

tion to socioeconomic aspects and record

both the people's willingness to pay and

the affordability of the services provided;

The needs of poor people should be ac-

counted for in tariff systems;

The minimum quantity of water available

to poor people should be assured by such

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 116

means as cross subsidies, even if that

would have negative implications for the

revenue-to-cost ratio of the respective

supply structure;

In order to create a knowledge base,

socio-economic monitoring should be in-

tegrated into the measure as an institu-

tionalised mechanism for measuring the

contribution to poverty reduction and to

the achievement of the MDGs;

Cooperation with national poverty reduc-

tion programmes should be further

strengthened;

Such pro-poor concepts as water kiosks

and EcoSan should receive more atten-

tion in the 'choice of technologies'.

Regarding the consideration of wastewater

management and sanitation services, together

with declining interventions in rural areas, it is

the evaluators' opinion that the attention being

paid to the activity areas 'wastewater man-

agement' and/or 'sanitation' and to rural areas

per se should be re-examined for new projects

and programmes. Both of those priorities are of

major significance for the orientation of Ger-

man DC.

Systematic incorporation of a multi-level ap-

proach into the conceptual design of new pro-

jects and programmes also appears important.

In this respect, it is recommended that lessons

learned from the reviewed evaluation se-

quence be utilised and that more attention be

paid to the 'deconcentration' of service delivery

at the meso level. The goal here should be to

pay more attention to the micro level in line

with pursuing a holistic approach and with

achieving results at the target group level. At

the same time, the realistic macro-level results

should also be determined.

In the formulation of objectives systems for

new projects and programmes, attention

should be paid to simplification (small number

of indicators) and to the relevance to results

(by recording measurable factors). It is rec-

ommended in this context to formulate only

such indicators as are actually connected with

the planned outputs. Agreements with partners

should also and simultaneously help generate

a common understanding and shared respon-

sibility.

A transparent capacity development strategy

should be formulated at the planning stage and

agreed on with the partners. In so doing, any

foreseeable need for follow-up assistance

should be taken into account and the availabil-

ity of required resources established (planning

security). In the opinion of the evaluators, the

areas 'networking' and 'system development'

should be more closely incorporated into the

capacity development service package of pro-

jects and programmes in order to achieve

higher levels of 'effectiveness' and 'impact'.

Harmonisation and coordination with German

and international DC actors should also take

place during the planning phase in the interest

of achieving complementarity and synergies.

To the extent possible, concrete agreements

and perhaps jointly conceived design and

planning should be targeted. Important areas

like defining the scopes of responsibility, steer-

ing and resource provision should be agreed in

appropriately formalised cooperation. This

could be founded on concrete service com-

mitments that the partner institutions also find

transparent.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 117

To the extent that cooperative projects or pro-

grammes are planned, the contributions and

results that the partners in cooperation can

actually provide should be realistically as-

sessed at the planning stage. It is recom-

mended that this include an assessment of

existing risks (such as delayed outflow of FC

funds, TC personnel assignments, and contri-

butions from DED, CIM, BGR, etc.). Such risks

can, if necessary, be offset by adjustments at

some later date. It is recommended to seek

dialogue with the BMZ in this respect.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 118

Annex 1

Terms of reference for the synthesis report on water

I. Background and object of evaluation

The GTZ extended and optimised its evaluation system in 2005, the major innovation being the intro-

duction of independent evaluations on behalf of the GTZ. The Evaluation Unit commissions independ-

ent research institutes and consulting firms to conduct these evaluations in order to review the GTZ's

work from an external perspective. These institutes and firms in turn contract the services of one inter-

national and one national evaluator from the partner country in each case. Each year, the GTZ com-

missions a total of approx. 30 independent evaluations (interim, final and ex-post evaluations) in two

thematic priority areas. In 2008, the priority areas were 'decentralisation' and 'water'. Uniform evalua-

tion criteria and a uniform assessment grid enable comparison of the evaluations.

Each year, after the Evaluation Unit has examined and accepted the evaluation reports submitted by

the evaluators, it arranges for synthesis reports (cross-section evaluations) to be performed in each

priority area. These serve to promote institutional learning at the GTZ and to report to the German

Federal Ministry for Economic Cooperation and Development (BMZ). The results contribute to knowl-

edge management at the GTZ and can also be used for PR activities.

The object of the present synthesis report is:

16 evaluation reports from 2008 on the thematic priority area 'water':

8 interim evaluations (Africa supraregional, Algeria, Yemen, Jordan, Kenya, Peru, Zambia,

Viet Nam)

3 final evaluations (Ghana, Kosovo, Mexico)

5 ex-post evaluations (Ethiopia, India, Niger, Tanzania, Turkey)

II. Objectives of the synthesis report

The evaluation is intended to:

summarise the results of the 16 individual evaluations

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 119

identify recurring strengths and weaknesses or rather factors determining success and failure;

clarify whether identified deficits, if any, are the result of omission or of neglected documenta-

tion

identify cross-project lessons learned and recommendations

III. Tasks

To evaluate and process the 16 evaluation reports from 2008 in the thematic priority area 'water' and

to subject them to quantitative and qualitative comparison.

The questions to be answered result from the following report structure:

Contents

Executive summary

1. Introduction

1.1 Background, objective and object of the synthesis report

1.2 For the reader's guidance

2. Objectives and activity areas

Each of the evaluated projects and programmes is to be briefly summarised.

Key questions of relevance:

o Which are the main objectives pursued by the projects/programmes?

o Over the course of time, can trends be established in the objectives systems of the

evaluated projects and programmes?

3. Rating according to DAC criteria (See definition and key questions for DAC criteria in Annex 3

[GTZ Guidelines on Evaluating the Success of Projects and Programmes])

3.1 Relevance

3.2 Effectiveness

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 120

3.3 Impact

3.4 Efficiency

3.5 Sustainability

In each of these areas:

Rating (average and distribution) and weighting (average and distribution) of the criterion in the

reports

Reasons for weightings that deviate from the standard (factor 2)?

Were those reasons convincing?

Reasons for good and poor ratings (frequency!)?

Can 'patterns' be identified from them, i.e. factors determining success or failure?

For effectiveness and impact: At which levels are the results situated (institutional change, per-

capita water/sewage connections, water quality, wastewater disposal, legal framework, etc.)?

Are there any correlations between the ratings of individual DAC criteria?

Conclusions

3.6 Synopsis of 3.1 to 3.5

Overall rating (average and distribution) in the reports

In how many cases did (poor) ratings of individual criteria lead to downgrading of the overall rating?

The ratings assigned in the individual evaluations are binding for the analysis (sections 3.1 to 3.6). In

other words, the ratings must not be altered within the framework of this synthesis report.

4. Rating of cross-cutting development themes

4.1 Poverty reduction and MDGs

4.1.1 Conceptual problems

4.1.2 Mainstreaming in development strategies (in the field)

4.1.3 Poverty marker in the offers and its distribution

Poverty markers of development measures (distribution)?

To what extent do/did the development measures comply with these markers?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 121

4.1.4 Target-group differentiation and poverty analysis

­ Was the project design differentiated according to target groups, and was a poverty analysis

performed? Were poverty analyses used for the project design?

In how many development measures is/was an 'adequate' poverty impact achieved?

­ Did the project promote the participation of poor people in economic and political processes?

­ Does the project have positive results in terms of poverty reduction?

­ Does the project help to remedy structural problems identified by the national poverty reduction

strategy?

­ Were questions of equitable distribution (e.g. access to land?) addressed, and if so, how? (was

not part of the ToR)

­ How many people have benefited from the poverty impact and/or will do so in the future?

What are the reasons for achieving and not achieving poverty impacts? Can 'patterns' be identi-

fied, i.e. factors determining success or failure? Are there findings that are specific to given re-

gions?

Conclusions

4.2 Gender equality

What is the distribution of gender markers for the development measures?

To what extent do/did the development measures comply with these markers?

­ G-1 and G-2: Was the project design gender-responsive, and was a gender analysis (gender-

based analysis of the initial situation) available?

­ Was the G-0 marker awarded on the basis of a gender analysis? (Note: This became obligatory

for development measures at the end of 2005.)

­ Are there consistent examples of the mainstreaming of the gender aspect in 'water' projects and

programmes?

How many development measures have achieved an 'adequate' gender impact?

­ Do men and women make equal contributions to shaping the project/programme?

­ Do men and women derive equal benefit from the project/programme?

­ Does the project/programme help to reduce structural gender discrimination and thus to achieve

a positive change in the gender ratio?

­ Was one gender discriminated against or was this made up for by positive measures?

­ What is the ratio of direct to indirect results?

What are the reasons for achieving or not achieving gender impacts? Can 'patterns' be identi-

fied, i.e. factors determining success or failure? Are there findings that are specific to given re-

gions?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 122

Conclusions (also in terms of how gender-responsive aspects can be mainstreamed into 'water'

measures).

4.3 Effects on the partner's capacity to act (capacity development)

In how many development measures are/were (very) good / satisfactory / inadequate successes

achieved in terms of capacity development, and at which levels (individual, institutional, societal)?

­ Did the advisory services have an effect on the partner's capacity to act?

­ Did it prove possible for pilot approaches to be successfully multiplied or adjusted by the partner

structures?

­ Did the development measures have an effect on the design of national policies and laws, and

on the stakeholders' ability to shape such policies and laws? And on interaction between the

state, the economy and civil society?

What are the reasons for successful/unsuccessful capacity development? Can 'patterns' be

identified, i.e. factors determining success or failure? Are there findings that are specific to given

regions?

Conclusions

5. Concept of sustainable development

To what extent and how frequently did/do the development measures, i.e. the GTZ's working

method, reflect the concept of sustainable development (holistic, process-oriented and value-

oriented approach)?

­ Holistic approach: Linkage of economic, social and ecological objectives dimensions; of sec-

toral, organisational and policy advice; of the micro, meso and macro levels

­ Process-oriented approach: help for self-help: establishing transparency of actors' interests;

promoting interaction between the state, civil society and the private sector

­ Value-oriented approach: Promoting democracy, the rule of law, human rights; gender equality;

good governance; social and ecological market economy

Conclusions

6. Sectoral assessment

6.1. How do the German interface organisations work together and engage in

task-sharing ('joined-up DC')?

Additional explanations

Who is involved in the provision of water-sector advisory services? To what extent are planning and

implementation performed jointly? Do the combined efforts of the German interface organisations yield

the anticipated synergies in the sector? Can expedient task-sharing take place, or does obstructive

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 123

overlapping occur? How is interface management effected? How does cooperation rate? How does

the partner see this? How much value can be added by cooperative projects/programmes? Where has

FC investment actually provided added value to the development measure, and where would the de-

velopment measure have remained unsuccessful without it? Are the FC flanking measures comple-

mentary, or to what extent do their contents overlap? To what extent could the content of the FC flank-

ing measures also have been implemented by TC? Which concrete contribution has TC made towards

consolidating the sustainability of FC investments?

6.2. How are the development measures embedded in the activities of the international donor

community?

Additional explanations

How far has cooperation between donors progressed? Which coordination measures have been taken

regarding advisory content and task-sharing? Is a sector-wide approach in place? Which concrete

impacts is this having on the work of the GTZ? Does this lead to common strategies, steps of planning

or perhaps even cooperation in the implementation of advisory services? What is the donor commu-

nity's assessment of ongoing cooperation? What do the partners think of it?

6.3. To what extent has it been possible to implement the multi-level approach in the provision

of advisory services, i.e. to utilise synergies by providing the services simultaneously at

the international and transnational levels, at the national level (ministries) and at the re-

gional and local levels?

The multi-level approach is being increasingly included in the planning of TC projects. Its purpose is to

design national policies, strategies and laws in a solution-oriented and activity-oriented way and, con-

versely, to gain impetus for the reform or revision of national framework conditions via local or regional

implementation. 'Multi-level approach' also means greater harmonisation and coordination efforts. In

complex development measures, the various levels are sometimes split up among various German

implementing organisations and, primarily in Africa, numerous international development measures

which, ideally, should help develop the respective national water sector.

How is the multi-level approach being realised in the project, and which exchange mechanisms have

been introduced between the advisors at the different levels? How is utilisation of the results at the

respective advisory level ensured by the other levels? How can experience and findings from local

processes, pilot measures and partial solutions be transferred to the national level and mainstreamed

in a systemic overall context?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 124

6.4. How are GTZ's knowledge-management methods being applied to the updating and qualita-

tive improvement of the range of advisory services?

How did the switch to sector networks or staff conferences lead to the dissemination of advisory ser-

vice contents and instruments?

Which concrete ideas or incentives are available for further developing the range of knowledge man-

agement instruments in the water sector?

How, and in which concrete cases, have improved learning processes been introduced into the part-

ner structures?

7. Contract and cooperation management

Which 'modes of delivery' were appropriate and why?

Was TC the appropriate instrument? (Differences between anchor countries and poor coun-

tries?) Are preconditions for programme-based approaches (e.g. SWAp's) in place?

Which particular strengths and weaknesses of project steering occurred (frequently), and what

are your conclusions?

How was cooperation with the partner institutions rated? Which particular strengths and weak-

nesses occurred (frequently)? Can any recommendations be made?

How were the cooperative relationships and the quality of cooperation in DC/IC rated? (coop-

eration and delimitation between FC/TC, joined-up DC, donor coordination and the role of the

GTZ, requirements stemming from the Paris Declaration)

Which particular strengths and weaknesses occurred (frequently)? Do you have any recom-

mendations about how to deal with the weaknesses?

8. Summary of conclusions and lessons learned

9. Recommendations

(from the above sections 3. - 7.)

Total length of report: approx. 70 pages

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 125

Annex 2: (Standard) Terms of reference for individual evaluations

I. Background

Independent evaluations are conducted either during the term of a development measure, on its com-

pletion, or in the form of an ex-post evaluation between two and five years after the promotion has

ended. The evaluations are managed by the Evaluation Unit. As a rule, the unit commissions inde-

pendent research bodies, institutes affiliated to universities or consulting companies to perform the

evaluation. These in turn place an international evaluator and a local evaluator under contract, who

jointly evaluate the development measures. In addition to their own sector-specific expertise, the local

evaluators are also able in particular to add the perspective of the respective partner country.

Each year, a total of 30 independent evaluations are performed in various regions. This annual ran-

dom sampling addresses certain thematic priority areas in order to subsequently formulate compara-

tive and overarching key points. The 2008 priority areas are decentralisation and water. The evalua-

tion described here is part of the thematic priority area water.

II. Object of the evaluation

[Brief description of the context in the country]

[Brief description of the development measure including the prior history and any continuation within

the framework of a programme, etc.]

III. Objective of the evaluation

The evaluation aims to rate the success of the development measure. The assessment is based on

five blocks of questions.

a) Assessment according to international evaluation criteria

The success of the development measure is to be described and assessed on the basis of predefined

evaluation criteria (based on facts and figures): relevance, effectiveness, impact, efficiency and sus-

tainability (see also Annex 3: Guidelines on evaluating the success of projects/programmes).

b) Assessment of the development measure with regard to poverty reduction and millennium

development goals

The assessment of the development measure should include an appraisal of the extent to which it

verifiably contributes to poverty reduction and achieving the MDGs.

Key questions of relevance:

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 126

Was the conceptual design of the development measure differentiated according to target

group, and was a poverty analysis performed?

Did the development measure promote the involvement of poor people in economic and politi-

cal processes?

Does the development measure have positive impacts on poverty reduction?

Does the development measure help overcome structural problems identified in the national

poverty reduction strategy?

c) Assessment of the development measure with regard to the promotion of gender equality

The assessment of the development measure should include an appraisal of the extent to which it

verifiably contributes to the promotion of gender equality.

Key questions of relevance:

Is the conceptual design of the development measure gender-responsive, and was a gender

analysis performed?

Do men and women contribute equally to the conceptual design of the development measure?

Do men and women draw equal benefit from the development measure?

d) Assessment of the development measure with regard to the promotion of sustainable de-

velopment

The assessment of the development measure should include an appraisal of the extent to which it

verifiably contributes to the promotion of sustainable development.

Key questions of relevance:

Does the development measure take a clearly holistic approach (linking of economic, social

and ecological objectives dimensions; technical, organisational and policy advisory services;

micro, meso and macro levels)?

Does the development measure take a process-oriented approach (help for self-help; estab-

lishing transparency of actors‟ interests; promoting interaction between the state, civil society

and the private sector)?

Does the development measure take a value-oriented approach (promotion of democracy,

rule of law, human rights; good governance; social and ecological market economy)?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 127

e) Sector-specific performance assessment of the development measure

The performance assessment of the development measure based on the aforementioned criteria must

include, in particular, a factually substantiated assessment on the basis of specific thematic questions.

Additional explanations of the various questions should be included in italics.

These are:

BMZ sector strategy, regional and country strategies

1. To which extent are sector strategies, regional and country strategies helpful for combining

and innovatively structuring advisory services for the water sector?

Additional explanations

Do these papers provide new, content-specific impulses? If so, which? Are they helpful for partner

dialogue? If so, how? Do they perhaps limit content-related sectoral development? If so, with respect

to which themes? Do the sector strategies make statements about gender equality that enter into the

advisory services? 'Sanitation' for example: This theme is mainstreamed in most strategy papers as

well as in the sector strategy. Does this make it easier to broach the subject with the partners? 'Irriga-

tion', for example: This branch of advisory services is not (or no longer) included in many country

strategies. Does that force the advisory inputs to move in a less effective direction (considering that

agriculture accounts for the major share of all water consumed)? Or does this amount to a practical

thematic focus? Why?

2. Do the 'water' development measures contain the 'Integrated Water Resources Manage-

ment (IWRM)' approach called for by, among others, the BMZ water sector strategy? How is

this approach or strategy implemented in concrete terms?

Additional explanations

IWRM demands attention to various sectors that have to do with water. This serves to uncover con-

flicts of use and identify common strategies, the objective being to maximise the (social, economic,

ecological) benefits. This, in turn, requires the involvement of various user groups and the monitoring

of negotiation processes. In concrete water-sector terms, this means that not only the actors them-

selves, e.g. those concerned with drinking water supply, but also other water users, e.g. the agricul-

tural, energy and/or industrial sector(s), be involved in the dialogue. The same applies to those who

will be affected by the results of water use (downstream riparians / ecosystems / tourism / fishing

grounds / fisheries, etc.). Are poor people involved as well? Which distribution effects does the devel-

opment measure have? Which good and which less successful examples are to be found in the GTZ

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 128

'water' development measures, and how was IWRM actually implemented? In other words, were any

institutions established and supported, or any methods and procedures initiated and established that

provide anchorage for the IWRM negotiation processes at the national or regional level? Were women

involved in these IWRM negotiation processes? How were the needs of female water users accounted

for within the IWRM? How could other sectors and/or water users be included in water resources

planning? Where and how, if at all, did the development measures encounter limits? Does the devel-

opment measure also address sub-sectors of relevance to the country's water sector (e.g. water sup-

ply, wastewater management, rural, urban, water resources management, irrigation)? How is experi-

ence transferred between the sub-sectors, within the development measure itself, and within the

scope of support for the policy discussion in the partner country? Which synergy effects exist between

the contents and results of the advisory / extension services in the sub-sectors?

Cooperation and task-sharing

3. How are the development measures embedded in the activities of the international donor

community?

Additional explanations

How far has cooperation between donors progressed? Which coordination measures have been taken

regarding advisory content and task-sharing? Does a sector-wide approach exist? Which concrete

effects does this have on the work of the GTZ? Does this lead to common strategies, steps of planning

or perhaps even cooperation in the implementation of advisory services? How is the progressive co-

operation being rated by the donor community? How do the donors rate it?

4. How is cooperation and task-sharing being handled by the German interface

organisations (‘joined-up development cooperation’)?

Additional explanations

Who is involved in providing advisory services in the water sector? How much joint planning and im-

plementation is taking place? Do the anticipated synergies emerge through the combined efforts of the

German interface organisations? Can expedient task-sharing be performed or does obstructive over-

lapping occur? How is interface management effected? How is cooperation rated? How do the part-

ners rate it? How much value is added by cooperative projects/programmes? Is the personnel and

financing constellation in accordance with the objectives of the development measure? Where have

investments in FC created genuine added value in the development measure, and where would the

development measure have been unsuccessful without it? Are the FC flanking measures complemen-

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 129

tary, or to which extent does content-related overlapping take place? To which extent could the FC

flanking measures have been implemented by way of TC? Which concrete contribution has TC made

toward securing the sustainability of FC investments?

Characteristic features of GTZ:

Direct contribution, capacity development, multi-level approach

5. Have the characteristic features of GTZ advisory services (direct contribution and capacity

development) been successful or not?

Additional explanations

The work of GTZ is characterised by 'direct contribution' as a form of advisory services and by its fo-

cus on 'capacity development'. The latter means that processes of change are supported by appropri-

ately adopted and diverse measures in order that people, organisations and societies can mobilise,

maintain, adjust and expand their capacities for sustainable development. What were the most impor-

tant existing framework conditions for successful capacity development in the individual service pack-

ages of the development measure? Which approach was successful with regard to interaction be-

tween stakeholders and reconciliation of conflicting interests? Where are the blockages? Where and

how did the development measure visibly contribute (via capacity development) towards good govern-

ance? Did the development measure, within the framework of capacity development, build capacity for

action for achieving gender equality as a major prerequisite for good governance and sustainable de-

velopment?

6. To what extent has it been possible to implement the multi-level approach in the provision

of advisory services, i.e. to utilise synergies by providing the services simultaneously at

the international and transnational levels, at the national level (ministries) and at the re-

gional and local levels?

Additional explanations

The multi-level approach is being increasingly included in the planning of TC projects. Its purpose is to

design national policies, strategies and laws in a solution-oriented and activity-oriented way and, con-

versely, to gain impetus for the reform or revision of national framework conditions via local or regional

implementation. 'Multi-level approach' also means greater harmonisation and coordination efforts. In

complex development measures, the various levels are sometimes split up among various German

implementing organisations and, primarily in Africa, numerous international development measures

which, ideally, should help develop the respective national water sector.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 130

How is the multi-level approach being realised in the project, and which exchange mechanisms have

been introduced between the advisors at the different levels? How is utilisation of the results at the

respective advisory level ensured by the other levels? How can experience and findings from local

processes, pilot measures and partial solutions be transferred to the national level and mainstreamed

in a systemic overall context?

Results of GTZ development measures for water

7. What are the essential results of the evaluated German DC development measure in the

water sector?

Additional explanations

How can the results of the development measure in the water sector be summarised? At which levels

are the results produced (institutional change, per-capita water and sanitation connections, water qual-

ity, wastewater disposal, legal framework, etc.)? What can be said about the cost-benefit ratio for the

achievement of certain results (e.g. cost of each sustainable water and/or sewer connection, cost of

institution promotion, cost of regulation, etc.)? Which contributions are being made to the MDGs (MDG

7 targets water supply & wastewater management; MDGs 5+6 health; MDG 1extreme poverty and

hunger)? (See item 2.3: overarching development results / Millennium Development Goals)

Which income groups are (predominantly) being reached)? To what extent are poor people / the dis-

advantaged / women being reached? Transboundary water management: How can/could it be

achieved that the adopted approach to institutional strengthening of transboundary water management

has direct, positive effects on the people? How are such results documented and communicated?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 131

Supply and demand side of service packages

8. On which themes should German DC in the water sector focus its future advisory services?

Additional explanations

Have there been any major shifts in the contents of the requested or required advisory services?

Which water-sector themes are generating heightened demand for advisory services and support, now

and in the future? (Since actors presently operating in the water sector were interviewed during the

evaluation, this question can also be posed for concluded development measures.)

Successful methods/instruments and knowledge management

9. Which innovative and/or replicable methods and/or instruments are being recommended

for dissemination and/or for broader application?

Additional explanations

Which concrete methods and/or instruments are available for dissemination in the region or in GTZ

water-development measures? (e.g. use of certain workshops, training modules, OE methods, etc.)

10. How are GTZ knowledge-management methods being used for updating and

improving the quality of the advisory services?

Additional explanations

How and in which concrete cases did the switch to sector networks or staff conferences lead to the

dissemination of advisory services and/or instruments? Which concrete ideas or incentives are there

for the further development of knowledge management tools for the water sector?

IV IMPLEMENTATION

a) Responsibilities

The Evaluation Unit is responsible for planning and managing GTZ's annual independent evaluation

programme. The subcontractor is institution X, who places an international expert and a national

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 132

expert under contract for the purpose of this evaluation. The subcontractor is responsible for conduct-

ing the individual evaluation, for quality assurance and for reporting on the evaluation.

b) Inputs

The subcontractor shall provide the following inputs:

peruse and evaluate relevant documents

draw up an 'inception report' containing the e-Val results and the facts, figures and data from

the beginning of the project/programme (as found in the appraisal report, baseline surveys,

and benchmarked indicators). Write the draft version of the inception report in German for

consultation with the Evaluation Unit, but draw up the final version – to serve as a working ba-

sis for the team of evaluators – in the lingua franca of the country in question. Submit the in-

ception report at least three weeks prior to the evaluator‟s departure. In special, well-founded

cases, an 'inception phase' for the country of assignment which precedes the evaluation may

be planned.

draw up the terms of reference for the national expert in the partner country

select and engage the national expert

have the international expert attend a one-day preparatory seminar about fundamental as-

pects of evaluation at the GTZ

prior to performance of the evaluation, have the selected international expert participate in lo-

cal preparatory meetings with representatives of the Evaluation Unit, the responsible regional

group and the Planning and Development Unit (P+D)

prepare, organise and perform on-the-ground surveys

meet with the German embassy and the local GTZ office, if possible prior to commencement

of the mission

document the preliminary results of evaluation and recommendations for the final presentation

in the project country

present and discuss the preliminary results of evaluation with the partners and local GTZ par-

ticipants prior to the international expert‟s departure.

draw up the report (containing the results determined by, and the recommendations made by

the international and national experts) in German (or, alternatively, by arrangement with the

Evaluation Unit also in English) within five weeks of return

attend an evaluation meeting at the GTZ

assess the national expert

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 133

draw up the final version of the evaluation report (comprising not more than 45 pages of sum-

mary and chapters 1 through 6, plus front pages, table of contents, list of abbreviations and

annexes; executed in MS Office 2003 as a PDF file; see specimen format for reports) in Ger-

man (or, alternatively, by arrangement with the Evaluation Unit, also in English) within three

weeks of the evaluation meeting, and generate a brief report (summary and tabular overview

of the main report, including an overview of the facts and figures of relevance to the assess-

ment of the project/programme results) in German and in the lingua franca of the given coun-

try

provide quality assurance for the reports and their foreign-language versions (inception report,

evaluation report and brief reports)

provide quality assurance for the organisation and implementation of the evaluation, as well as

technical follow-up assistance for the evaluators.

The GTZ shall provide the following inputs:

make the necessary documents available

hold preparatory talks with the subcontractor/institution and the evaluator, and host (including

organisation) a one-day preparatory seminar for the evaluators

announce and introduce the evaluation to the relevant partners and important contacts in the

country of assignment

provide (on request) logistical support through the local GTZ office (reimbursement of costs by

the commissioned institution in accordance with the contractual agreement)

organise and hold the evaluation meeting with the involvement of the responsible sectoral and

regional divisions and, in the case of cooperative projects/programmes, the other development

cooperation organisation

approve the inception report and the evaluation report (incl. foreign-language executive sum-

mary).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 134

Annex 3: Guidelines on evaluating the success of projects/programmes

1 Preliminary remarks

This document is a guide to evaluating the success of projects/programmes implemented by GTZ. It

contains the central evaluation criteria, the key questions devised to examine the criteria, and an as-

sessment grid devised to obtain an overall rating.

The evaluation criteria, which were developed in a working group operating under the name

“Evaluierung aus einem Guss”117

and are binding for its members, are based on the five criteria

agreed within the OECD-DAC2:18

relevance, effectiveness, efficiency, overarching development-policy

results (impact) und sustainability.

Furthermore, key questions were devised which represent important points of reference for assess-

ment of the evaluation criteria.

The rating of a project/programme according to the individual evaluation criteria is given on the basis

of a six-point scale (exception: a four-point scale is used for the sustainability criterion).

Explanation of terms:

Development measure: The term “development measure” used in this document encompasses the

following terms: project, programme, program-based approaches, and development intervention.

Results: The term “results” used in international debate and in this document includes the outputs, the

use of outputs, the direct and indirect results and the highly aggregated results.

17 A group of the Evaluation Departments of BMZ (lead), GTZ, KfW, DED and InWEnt created to harmonise and coordinate

evaluation activities.

18 OECD-DAC: Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development

(OECD), Paris

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 135

These guidelines are also based on the rationale and concepts of the GTZ results model:19

The GTZ results model

Basis of the assessment

The evaluation of a development measure requires systematic examination of the underlying cause /

effect-hypotheses. This examination is carried out in the following stages:

The quality of the results chain of the development measure is investigated and as-sessed

(including rationale for the assessment).20

This includes, for example, examining the plausibil-

ity of the cause / effect-hypotheses and the level to which the objectives are to be achieved.

(formulation of objectives and indicators)

If the cause / effect-hypotheses and the objectives of the development measure are judged to

be consistent, they are taken as the basis for the evaluation of the pro-ject/programme.

19 See also “Guidelines on Offers”, GTZ November 2006 4 The objectives and indicators defined in the offers are binding and

are applicable here, not the information given in any other project documentation.

20 The objectives and indicators defined in the offers are binding and are applicable here, not the information given in any other

project documentation.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 136

If the cause / effect-hypotheses and the objectives of the development measure are judged

not to be consistent, the evaluators develop their own hypotheses, objectives and indicators

which appear to them to be more appropriate. These then form the basis for evaluating the

success of the project/programme. General remarks on assessment

All that is assessed is the development-policy effectiveness of the project/programme as defined in the

evaluation criteria.21

What is assessed is primarily the situation as it can be measured or established at the time of the

evaluation. Exceptions are explained in Section 2 in the descriptions of the relevant evaluation criteria.

Results that are difficult to estimate at the time of the evaluation must be plausibly inferred and de-

scribed using proxy indicators or by verifying central assumptions about the results chain.

2 The five evaluation criteria

2.1 Relevance Are we doing the right thing?

The extent to which the objectives of the development measure match the needs of the target groups,

the policies of the partner country and partner institutions, the global development goals and the Ger-

man Government‟s basic development-policy orientation.

Key questions for assessing relevance22

21 The overall success of the development measure is not to be equated with the quality of the work performed by GTZ and its

implementation partners. For example, good management on its own does not necessarily lead to overall success.

22 The relevance of the project/programme is assessed as of the time at which the conceptual approach was formulated or

adjusted. A concept adjustment that was not carried out, despite being necessary, is also relevant to the assessment, how-

ever.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 137

To what extent is the development measure suitable for addressing central development is-

sues for the partner country and target groups (differentiated according to gender, ethnic

group, parties to a conflict), and specifically poor population groups?

To what extent is the development measure in compliance with the (sector) policies and

strategies of the partner country (national plans, PRSP etc.) and the partner institutions?

To what extent is the development measure in compliance with international themes, stan-

dards, conventions?

Does the project or programme have a high or low priority for the responsible institutions in the

partner country or for civil society? How is this apparent?

Does the development measure comply with the client‟s basic development-policy orientation?

23

Which of the cross-cutting issues are relevant? How far are these taken into consideration in

the design of the project or programme?

To what extent is the development measure targeted at poverty reduction and the MDGs?

23 Currently these are: poverty reduction, fostering gender equality, participatory development and good governance, environ-

mental protection and conservation of natural resources, crisis prevention, combating drug abuse, rural development, and

conservation of tropical forests. See also BMZ homepage: http://www.bmz.de/en/index.html

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 138

To what extent does the project or programme fit into the country strategy, priority area and

programme definition, and the sector strategies?

To what extent is the project or programme in line with GTZ‟s concept of sustainable devel-

opment, i.e. to what extent does it contribute – through a holistic project approach, through

process orientation and through value orientation – to ensuring that the results of the pro-

ject/programme and the developments in the partner country can be sustainable?24

2.2 Effectiveness

Are we achieving the objectives of the development measure?

The extent to which the intended direct results (objectives) of the development measure are being

achieved (comparison of actual situation with targets).

Key questions for assessing effectiveness25

To what extent are the objectives of the development measure being achieved (comparison of

actual situation with targets on the basis of the defined indicators)?26

Are any limits critical to

success for achieving the objectives being undershot or overshot?

What are the decisive reasons why the objectives are or are not being achieved?

To what extent do the political partners (lead executing agency in the partner country and

BMZ) and the implementing organisations (national implementation partners and GTZ) have a

positive or negative influence on the achievement of objectives?

What unintended positive and negative direct results have occurred? How should these be

assessed in the overall context? What is the response to these?

24 See also GTZ homepage: http://www.gtz.de/en/top-themen/15534.htm

25 When evaluating ongoing projects or programmes, the appraisers assess the effectiveness in terms of the degree to which

the objectives have been achieved at the time of evaluation, i.e. whether the project/programme is within the objectives corri-

dor. In final and ex-post evaluations, the effectiveness is assessed on the basis of the degree to which the objectives were

achieved at the end of the project or programme.

26 The assessment is made on the basis of the objectives and indicators that are considered appropriate; see also the explana-

tions on the “Basis of the assessment” on Page 2 of this document.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 139

2.3 Overarching development results (impact)

Are we contributing to the achievement of overarching development results?

The extent to which the project or programme contributes to achieving the intended over-arching re-

sults and produces other indirect results.

Key questions for assessing overarching development results (impact)

What (positive and negative) changes at the level of indirect results can be observed in the

wider sectoral and regional environment of the development measure?

Which of these changes can be plausibly attributed to the project/programme at various levels

(population, sector, institutions and regulations)? For example: 27

How far are important barriers to development being structurally eliminated or reduced

through the project/programme?

How are the living conditions or development opportunities changing?

How is the project/programme strengthening the ability to solve problems of the target

groups(of the project/programme and others), intermediaries and institutions?

What contributions is the project/programme making towards achieving overarching

development goals (e.g. Millennium Development Goals and implementing the Millen-

nium Declaration, structural reduction of poverty - e.g. promotion of pro-poor growth,

pro-poor governance)?

What results are being achieved in terms of other cross-cutting issues: gender, envi-

ronmental protection and conservation of natural resources, participation and govern-

ance, crisis prevention and conflict sensitivity?

What contributions can the development measure realistically make to the achievement of

these indirect results (expected level to be achieved by the project/programme)?

27 Depending on the individual circumstances, the examples listed here for indirect results may also be direct results, in which

case they have to be taken into account under the “effectiveness” criterion. In order to decide whether a result is direct (“ef-

fectiveness” criterion) or indirect (“impact” criterion), the question has to be answered whether achievement of the result can be

attributed causally to the development measure (direct result) or whether the development measure is making plausibly explica-

ble contributions to the achievement of the result (indirect result).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 140

How is the actual contribution of the development measure to the achievement of these indi-

rect results assessed?

What are the decisive reasons why indirect results are or are not being achieved?

To what extent is the effectiveness of the development measure positively or negatively af-

fected by other policy fields, strategies or interests (German ministries, bilateral and multilat-

eral development partners)? What consequences has the project/programme drawn from this?

What broad impact is being achieved: e.g. suitable as a model, structural changes, replication

of approaches and scaling-up?

2.4 Efficiency

Are the objectives being achieved cost-effectively?

A measure of the degree to which the resources invested in a development measure are appropriate

compared to the outputs and results achieved.

Key questions for assessing efficiency

What resources is the development measure using for the various forms of modes of delivery

(long-term experts, short-term experts, procurements, operating and administration costs,

training courses, local subsidies and financing contracts, other contributions)?

To what extent is the structure of the development measure (e.g. project, programme struc-

ture, program-based approaches) appropriate for achieving the outputs and results cost-

effectively? What structure might make it possible to do this more efficiently?

To what extent are the objectives and outputs/activities of the project or programme coordi-

nated with or complementary to those of others, or designed for task-sharing (joined-up Ger-

man Development Cooperation, program-based approaches such as SWAp, basket funding or

budget support)?

To what extent is there adequate coordination between donors? In what ways is this encour-

aged or hindered?

To what extent is the composition of the modes of delivery - personnel concept and personnel

assignment, provision of materials and equipment, training, funding - cost-effective, i.e. how

far are internal resources being used efficiently?

What alternatives are there to the modes of delivery employed, and which (if any) would be

more efficient?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 141

To what extent have the outputs been produced cost-efficiently (cost-benefit ratio)? What al-

ternatives are there, and which (if any) would be more efficient?

To what extent are the direct and indirect results achieved efficiently in terms of microeco-

nomic, macroeconomic and sector-specific standards?

Are the outputs and results achieved at the correct time and within a reasonable period?

2.5 Sustainability

Are the positive results durable?

A measure of the probability that the positive results of the development measure will continue beyond

the end of assistance.

Key questions for assessing sustainability

What period is appropriate/realistic for the continuation of the results, and which critical mini-mum

requirements for success in this period (expectations of sustainability) are appropriate for the pro-

ject or programme?

What approaches, instruments, methods or concepts are lastingly used, institutionalised or further

developed by the target groups, partner institutions or other actors? How is this done?

How will the results for the target groups, partner institutions and partner country continue be-yond

the end of assistance? How far can the (direct and indirect) results of the project/programme be

maintained, or is an improvement or deterioration to be expected?

To what extent are the requirements for sustainability met?

To what extent are (organisational, personnel, financial, economic) resources and capaci-

ties available in the partner country (in the longer term) for maintaining the results

achieved? To what extent is the required ability in place to adapt to changing frame-work

conditions and solve associated problems?

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 142

How should the four dimensions of sustainability28

(economic, political, social and ecologi-

cal) and their risks and opportunities be assessed?

How do these dimensions of sustainability interact? Under the specific conditions, to what

extent is the result balanced, stable and capable of modification in the longer term?

What are the key risk factors for longer-term sustainability of the results? How is the evolution of

these factors assessed?

Operationalisation of the four dimensions of sustainable development to assess sustainability:

Economic: how are the achieved outputs and results secured at the microeconomic or

macroeconomic level? How stable or capable of modification are they in terms of the

pace of economic growth (local, regional, national, global)?

Political: how is the project or programme contributing to a fair and peaceful balance

of interests? To what extent are changes in the political culture and changes in behav-

iour, attitudes and awareness apparent in the target groups and partner institutions?

To what extent and in what way is ownership on the part of target groups and institu-

tions necessary and existent? To what extent is the legal framework in place that is

necessary for changes? To what extent is the political will for change identifiable?

Social: how is the project or programme contributing to greater equality of opportunity,

social justice, improved access to social services and resources?

Ecological: in what way is long-term ecological viability ensured? What risks are there

of lasting negative environmental effects emerging in the long term?

3. Assessment grid

3.1 Assessment of the individual evaluation criteria

28 In specialist debate it is common to distinguish between three dimensions: economic, social and ecological sustainability. To

provide a more differentiated description, in this document social sustainability is broken down into political sustainability (socie-

tal level) and social sustainability in the narrower sense (individual level).

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 143

A six-point scale is used to rate the project or programme according to four of the criteria, namely

“relevance”, “effectiveness”, “overarching development results” (impact) and “efficiency”. The scale is

as follows:

1 very good rating, significantly better than expected

2 good rating, fully in line with expectations, no significant defects

3 satisfactory rating, falling short of expectations but with positive results dominant

4 unsatisfactory rating, significantly below expectations, and negative results dominate despite

identifiable positive results

5 clearly inadequate rating: despite several positive partial results, the negative results clearly

dominate

6 the project/programme is useless, or the situation has deteriorated on balance

A rating of 1-3 indicates a positive assessment, a rating of 4-6 an assessment that is not positive.

The “sustainability” criterion is rated on the following four-point scale:

1 (very good sustainability)

The overall success of the project/programme (positive to date) will continue unchanged or

even increase with a high degree of probability.

2 (good sustainability)

With a high degree of probability, the overall success of the project/programme (positive to

date) will only minimally decrease but will overall remain significantly positive (normal situation

– to be expected).

3 (satisfactory sustainability)

The overall success of the project/programme (positive to date) will decrease significantly but

remain positive with a high degree of probability. This also applies if the sustainability of a pro-

ject/programme is assessed as inadequate up to the point of evaluation, but will evolve posi-

tively with a high degree of probability, so that the project/programme will achieve positive

overall success.

4 (inadequate sustainability)

The overall success of the project/programme is inadequate at the time of evaluation and

there is a high degree of probability that it will not improve. This also applies where the sus-

tainability has previously been rated positively but there is a high degree of probability that it

will decline seriously and no longer meet the requirements for level 3.

Overall rating

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 144

The overall rating is calculated from the assessment and a weighting of the five individual criteria

which is determined for each specific project. The rating is given on a six-point scale:

1 very good rating, significantly better than expected

2 good rating, fully in line with expectations, no significant defects

3 satisfactory rating, falling short of expectations but with positive results

dominant

4 unsatisfactory rating, significantly below expectations, and negative results domi-

nate despite identifiable positive results

5 clearly inadequate rating: despite several positive partial results, the negative re-

sults clearly dominate

6 the project/programme is useless, or the situation has deteriorated on balance.

An overall rating of 1-3 shows that a project or programme was successful, a rating of 4-6 shows it

was unsuccessful. However, projects/programmes can only be rated as “successful” if the direct

results (effectiveness), indirect results (impact) and sustainability are rated at least “satisfactory”

(3). 29

A decision is made and justified for each criterion whether it is “particularly important”

(weighting 3), “important” (weighting 2) or “less important” (weighting 1) in the specific con-text of

the project/programme. In the absence of a special reason which makes it more or less important,

a criterion is held to be “important” (weighting 2). Ratings and weightings are always expressed in

whole numbers, rounded up or down in accordance with the usual rules of mathematics. 30

29 13 In exceptional cases a rating of 4 for sustainability is acceptable for a “successful” project/programme if this has been

planned from the start and was inevitable for the project/programme and still appears acceptable at the time of evaluation be-

cause of the great development significance of the project/programme. In such instances the weighting for the “sustainability”

criterion must be “1”.

30 The Evaluation Unit has developed an Excel macro for calculating the overall rating; only the values for the rating and weight-

ing of the individual criteria need to be entered in the macro, which then calculates the overall rating.

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 145

lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll

Annex 4: Rating of individual criteria and overall rating of 15 evaluated projects/programmes

Country Project /

programme Total Relevance

Effect-

iveness Impact Efficiency

Sustain-

ability Total

Africa

suprareg.

NWI 2 1 2 2 3 2 good

Algeria PDEA 3 4 3 3 4 3 satisfactory

Yemen WSP 2 2 2 1 2 3 good

Jordan WMIA 2 1 2 3 2 2 good

Kenya WSRP-KEN 2 2 2 2 2 3 good

Peru PROAGUA 2 1 2 3 2 2 good

Zambia WSRP-ZMB 2 2 2 3 2 3 good

Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 146

Viet Nam WDPC 3 1 3 3 3 3 satisfactory

Interim evaluations 2.3 1.8 2.3 2.5 2.5 2.7 good

Ghana EVORAP 3 3 2 2 3 3 satisfactory

Kosovo WSWDP 2 2 2 3 2 3 good

Mexico GICA 3 2 3 3 3 3 satisfactory

Final evaluations 2.7 2.3 2.3 2.7 2.7 3.0 satisfactory

Ethiopia IWSTR 4 2 3 3 4 4 unsatisfactory

India IGWDP 2 1 2 3 3 2 good

Niger PHV/MI 4 3 4 3 4 4 unsatisfactory

Turkey QADI 4 3 3 4 3 3 unsatisfactory

Ex-post evaluations 3.5 2.5 3.0 3.3 3.5 3.3 satisfactory

All evaluations 2.7 2.0 2.5 2.7 2.8 2.8 satisfactory

Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH – German Technical Cooperation – Dag-Hammarskjöld-Weg 1-5 65760 Eschborn/Germany T +49 61 96 79-1408 F +49 61 96 79-801408 E evaluierung@gtz.de I www.gtz.de