Deckblatt und Impressum für Publikationen (englisch ... · Jan Sass / Annika Schöpe Eschborn,...
Transcript of Deckblatt und Impressum für Publikationen (englisch ... · Jan Sass / Annika Schöpe Eschborn,...
Evaluation Unit
Cross-section evaluation
of independent evaluations in 2008
in the thematic priority area Water
Synthesis report
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water'
Published by: Deutsche Gesellschaft für
Technische Zusammenarbeit (GTZ) GmbH Evaluation Unit
Postfach 5180 65726 Eschborn
T +49 61 96 79-1408
F +49 61 96 79-801408 E [email protected]
Prepared on behalf of:
Federal Ministry for Economic Cooperation and Development
Internet: www.gtz.de
Produced by: GFA Consulting Group GmbH
Jan Sass / Annika Schöpe
Eschborn, Oktober 26th 2009
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' i
Contents
Summary 8
1. Introduction 15
1.1 Background, objectives and object of the synthesis report 15
1.2 For the reader's guidance 17
1.2.1 Overview of the evaluated projects and programmes 17
1.2.2 Methodological approach 22
1.2.3 Comparability of the evaluation reports 23
2. Objectives and activity areas 25
2.1 Brief description of the evaluated projects and programmes 25
2.1.1 Interim evaluations 25
2.1.2 Final evaluations 29
2.1.3 Ex-post evaluations 31
2.2 Analysis of objectives systems 32
2.2.1 Activity areas, priority areas and indicators 33
2.2.2 Intervention and results levels 37
3. Rating According to DAC Criteria 39
3.1 Relevance 40
3.1.1 Relevance rating 40
3.1.2 Weighting of relevance 46
3.2 Effectiveness 46
3.2.1 Effectiveness rating 46
3.2.2 Weighting of effectiveness 52
3.3 Overarching development results (impact) 52
3.3.1 Assessment of overarching development results (impact) 52
3.3.2 Weighting of overarching development results (impact) 56
3.4 Efficiency 56
3.4.1 Efficiency rating 56
3.4.2 Weighting of efficiency 62
3.5 Sustainability 63
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' ii
3.5.1 Sustainability rating 63
3.5.2 Weighting of sustainability 67
3.6 Synopsis 68
3.6.1 Weighting of DAC criteria 68
3.6.2 Overall rating of evaluated projects and programmes 70
4. Rating of cross-cutting development themes 75
4.1 Poverty reduction and MDGs 75
4.1.1 Conceptual problems 77
4.1.2 Mainstreaming in development strategies 79
4.1.3 Poverty marker in the offers and its distribution 79
4.1.4 Target group differentiation and poverty analysis 81
4.1.5 Conclusions 82
4.2 Gender equality 83
4.2.1 Gender markers of the evaluated projects and programmes 83
4.2.2 Conclusions 87
4.3 Effects on the partners' capacity for action 88
4.3.1 Capacity development within the context of the evaluated projects and programmes 88
4.3.2 Conclusions 92
5. GTZ's concept of sustainable development 94
5.1 GTZ's concept of sustainable development within the context of the evaluated projects
and programmes 94
5.2 Conclusions 96
6. Sectoral assessment 97
6.1 Cooperation and task-sharing among German interface organisations 97
6.2 Embedment of development measures in the activities of the international donor
community 99
6.3 Implementation of the multi-level approach in advisory projects and programmes 101
6.4 Utilisation of GTZ's knowledge management methods 103
7. Contract and cooperation management 104
7.1 Modes of delivery 104
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' iii
7.2 Steering and cooperation with partner institutions 105
8. Summary of conclusions and lessons learned 107
8.1 Summarising conclusions 107
8.2 Lessons learned from the individual evaluations 108
8.2.1 Relevance of the development measures 108
8.2.2 Effectiveness of the development measures 109
8.2.3 Impact of the development measures 110
8.2.4 Efficiency of the development measures 110
8.2.5 Sustainability of the development measures 111
9. Recommendations 113
9.1 General recommendations 113
9.2 Recommendations for steering ongoing projects and programmes 114
9.3 Recommendations for planning new projects and programmes 115
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' iv
Annexes
Annex 1: Terms of reference for the synthesis report on water
Annex 2: (Standard) terms of reference for individual evaluations
Annex 3: GTZ Guidelines on Evaluating the Success of Projects and Programmes
Annex 4: Rating of individual criteria and overall rating of 15 evaluated projects
List of illustrations and tables
Figure 1: Distribution of activity areas in the evaluated projects/programmes 33
Figure 2: Proportions of urban and rural priorities in the
evaluated projects/programmes 34
Figure 3: Trends in the thematic orientation of objectives systems 35
Figure 4: Direct results levels of the evaluated projects/programmes 38
Figure 5: Distribution of relevance ratings 41
Figure 6: Distribution of effectiveness ratings 48
Figure 7: Distribution of overarching development results rating 54
Figure 8: Distribution of efficiency ratings 58
Figure 9: Distribution of sustainability ratings 64
Figure 10: Distribution of overall ratings 72
Figure 11: Average rating according to type of evaluation 74
Table 1: Overview of evaluated projects/programmes 18
Table 2: Assessment scale for the DAC criteria relevance, effectiveness,
impact and efficiency, and for the overall rating 39
Table 3: Assessment scale for the DAC criterion sustainability 40
Table 4: Relevance ratings of evaluated projects/programmes 41
Table 5: Effectiveness ratings of evaluated projects/programmes 47
Table 6: Effectiveness correlations 52
Table 7: Rating of overarching development results of
the evaluated projects/programmes 54
Table 8: Efficiency ratings of the evaluated projects/programmes 58
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' v
Table 9: Comparison of term, level of funding and efficiency ratings 59
Table 10: Sustainability ratings of the evaluated projects/programmes 64
Table 11: Sustainability correlations 67
Table 12: Average weighting of DAC criteria 69
Table 13: Weighting of DAC criteria according to time (= type) of evaluation 69
Table 14: Overall ratings of evaluated projects/programmes 71
Table 15: Distribution of overall ratings according to success level 72
Table 16: Comparison of average weightings and ratings of DAC criteria 73
Table 17: Average overall ratings and DAC criterion ratings according to region 74
Table 18: Overview of contributions by evaluated projects/programmes to the
achievement of the MDGs 76
Table 19: Poverty markers of the evaluated projects/programmes 80
Table 20: Gender markers of the evaluated projects/programmes 85
Table 21: Levels of capacity development in the evaluated projects/programmes 90
Table 22: Consideration of the characteristics of sustainable development
in the evaluated projects/programmes 95
Table 23: Cooperation between the evaluated projects/programmes
and German DC interface organisations 98
Table 24: Contractors and subcontractors involved in the evaluated
projects/programmes 104
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' vi
List of abbreviations
AURA Development-policy Framework for Contracts and Cooperation
BGR Federal Institute for Geosciences and Natural Resources
BMZ German Federal Ministry for Economic Cooperation and
Development
CIM Centre for International Migration and Development
DC development cooperation
DED German Development Service
EcoSan ecological sanitation
EVORAP Eastern and Volta Region Assistance Programme, Ghana
FC financial cooperation
GICA Gestión Integrada de Cuencas y Acuíferos, Mexico
(Management of Water Catchment Areas for the Río Lerma in the Toluca
Valley and for the Río Balsas, Mexico)
GIS geographical information system
GTZ Deutsche Gesellschaft für Technische Zusammenarbeit GmbH
GTZ-IS GTZ International Services
IGWDP Indo-German Watershed Development Programme, India
IM irrigation management
InWEnt Capacity Building International, Germany
IWRM integrated water resources management
IWSTR Improvement of Water Supply in the Tigray Region, Ethiopia
KfW Kreditanstalt für Wiederaufbau / KfW Entwicklungsbank
LDC least developed country
MDGs millennium development goals
NGO non-governmental organisation
NWI Nile Water Initiative -
Planning and Management of Water Resources in the Nile Basin,
Africa supraregional
OECD-DAC Organisation for Economic Cooperation and Development
- Development Assistance Committee
PDEA Programme d‟Eau en Algérie
(Integrated Water Management Programme, Algeria)
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' vii
PHV/MI Programme Hydraulique Villageoise Maradi, Niger
(Rural Water Supply in Maradi, Niger)
PPR project progress review
PROAGUA Programa de Agua y Alcantarillado, Peru
(Drinking Water and Sanitary Programme, Peru)
SPR sector policy and regulation
SWAp Sector Wide Approach
TC technical cooperation
ToR terms of reference
UMS Upgrading of Municipal Services, Turkey
WDPC Wastewater and Solid Waste Management in Provincial Centres, Viet
Nam
WM wastewater management
WM/S wastewater management/sanitation services
WMIA Water Management in Irrigated Agriculture, Jordan
WRM water resources management
WS (drinking) water supply
WS&S water supply and sanitation
WSP Water Sector Programme, Yemen
WSRP (KEN) Water Sector Reform Programme in Kenya
WSRP (ZMB) Water Sector Reform Programme in Zambia
WS/WM water supply/wastewater management
WSWDP Water Supply and Wastewater Disposal Project (Kosovo)
WU water utility
ZOPP objectives-oriented project planning
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 8
SUMMARY
In the overall review of all evaluations per-
formed in the thematic priority area 'water', it
proves decisive for the success of these
measures that the respective project or pro-
gramme be in harmony both with the sector
strategies of the German Federal Ministry for
Economic Cooperation and Development
(BMZ) and with the respective national policies
and poverty reduction strategies, and that the
projects/programmes follow the key guidelines
on regional cooperation and integration, capac-
ity development and integrated water re-
sources management (IWRM). It is of particu-
lar importance to systematically pursue the
concept of sustainable development from the
planning stage onward, at the same time har-
monising the objectives dimensions within the
project/programme by way of a multi-level ap-
proach. In addition to a systematic approach to
capacity development, this also calls for closer
attention to the duration of advisory services
with an eye to efficient follow-up assistance.
The present cross-section evaluation of fifteen
projects and programmes which were evalu-
ated in the thematic priority area „water‟ in
2008 contains an analysis of the evaluation
reports on:
eight interim evaluations (Africa suprare-
gional, Algeria, Yemen, Jordan, Kenya,
Peru, Zambia and Viet Nam);
three final evaluations (Ghana, Kosovo
and Mexico);
four ex-post evaluations (Ethiopia, India,
Niger and Turkey).
The selection of evaluated projects and pro-
grammes constitutes a non-representative
random sample taken from the overall portfolio
of the Deutsche Gesellschaft für Technische
Zusammenarbeit (GTZ) GmbH.
The submitted synthesis report offers some
important points of reference apart from a
summary of performance assessments, espe-
cially with regard to the methodological orienta-
tion of the evaluated projects/programmes, and
consideration of the principles and concepts of
German development cooperation. Poverty
reduction, adherence to the principles of sus-
tainable development and systematic devel-
opment of capacity for action within the partner
structure are all major cross-cutting themes for
which this evaluation can provide insights and
findings.
The sector-specific priority areas of the fifteen
evaluated projects and programmes cover a
broad, diverse range, from sector reform and
policy advisory services to municipal water
supply and wastewater disposal, rural water
supply and water resources management, up
to and including irrigated agriculture. Eight of
the evaluated projects/programmes fall into the
category of programme building, and six of
them are cooperative projects/ programmes as
defined by German development cooperation.
Most of the correspondingly defined objectives
include two or more partial objectives and/or
sub-sectors of the water sector (interventions,
e.g. in water supply and wastewater manage-
ment, water resources management, consid-
eration of urban and rural sub-sectors, combi-
nation of policy advisory services and process
optimisation). The underlying objectives sys-
tems are therefore very complex and charac-
terised by a large number of indicators and, to
a certain extent, long results chains.
The main focus of the underlying definitions of
objectives lies on the quantitative improvement
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 9
of municipal drinking water supply. Only a mi-
nor share of the projects/programmes focus on
the rural sub-sector and the areas of wastewa-
ter management and sanitation services.
On the basis of their design, all evaluated pro-
jects/programmes generally contribute to the
achievement of MDG 7 ('Ensure environmental
sustainability'). In addition, the evaluated pro-
jects contribute to the achievement of the fol-
lowing MDGs:
10 projects/programmes (66%) contribute
to the achievement of MDG 1 ('Eradicate
extreme poverty and hunger'), for exam-
ple by creating jobs and potentials for
economic activities and promoting irri-
gated agriculture;
11 projects/programmes (73%) contribute
to the achievement of MDG 4 ('Reduce
child mortality'), for example by reducing
the incidence of water-induced diseases,
in particular diarrhoea, and by improving
hygiene in overall infant care;
8 projects/programmes (53%) contribute
to the achievement of MDG 5 ('Improve
maternal health'), for example by reducing
the incidence of water-induced diseases,
improving natal hygiene, increasing the
availability of hygienically safe drinking
water and providing for adequate sanita-
tion services;
5 projects/programmes also contribute to
the achievement of MDG 2 ('Achieve uni-
versal primary education'), 5 projects con-
tribute to MDG 3 ('Promote gender equal-
ity and empower women'), 1 project con-
tributes to MDG 6 ('Combat HIV/AIDS,
malaria and other diseases') and 1 project
contributes to MDG 8 ('Develop a global
partnership for development').
The projects/programmes evaluated within the
scope of the independent evaluations were
rated on the basis of a six-point scale for the
DAC criteria1 'relevance', 'effectiveness', 'im-
pact' and 'efficiency', plus an overall rating:
'very good' (1): very good results, signifi-
cantly better than expected
'good' (2): good results, fully in line with
expectations, with no significant defects
'satisfactory' (3): satisfactory results; fal-
ling short of expectations but with positive
results dominant
'unsatisfactory' (4): unsatisfactory results;
significantly below expectations, and
negative results dominate despite identifi-
able positive results
'inadequate' (5): clearly inadequate re-
sults; despite several positive partial re-
sults, the negative results clearly domi-
nate
'useless' (6): the project/programme is
useless, or the situation has, if anything,
deteriorated.
The 'sustainability' criterion was rated accord-
ing to the following four-point scale:
'very good' (1): With a high degree of
probability, the overall success of the pro-
ject/programme (positive to date) will con-
tinue unchanged or even increase.
'good' (2): With a high degree of probabil-
ity, the overall success of the pro-
ject/programme (positive to date) will de-
crease only minimally and remain signifi-
1 OECD-DAC:
Organisation for Economic Cooperation and Develop-
ment – Development Assistance Committee
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 10
cantly positive overall (normal situation –
to be expected)
'satisfactory' (3): With a high degree of
probability, the overall success of the pro-
ject/programme (positive to date) will de-
crease significantly but remain positive.
'inadequate' (4): The overall success of
the project/programme was inadequate at
the time of evaluation, and there is a high
degree of probability that it will not im-
prove.
In the above sense, ratings 1 - 3 are indicative
of a successful assessment.
Overall, 80 % of the evaluated projects/ pro-
grammes were rated as 'successful' in terms of
implementation. Only three development
measures (Ethiopia, Niger and Turkey) were
regarded as 'unsuccessful' in the sense of the
underlying performance assessment. In those
projects, the calculated average ratings were in
the lower satisfactory range - due to either
inadequate sustainability (Ethiopia and Niger)
or, in the case of Turkey, unsatisfactory contri-
butions to the achievement of overarching
results.
The average overall rating of all 15 develop-
ment measures covered by the cross-section
evaluation, at 2.7, lies within the 'satisfactory'
range.
Success level Rating Number Percentage
'Successful'
'very good' (level 1) 0 0%
'good' (level 2) 8 53%
'satisfactory' (level 3) 4 27%
'Unsuccessful'
'unsatisfactory' (level 4) 3 20%
'inadequate' (level 5) 0 0%
'useless' (level 6) 0 0%
In the assessment of projects/programmes
according to DAC criteria, „relevance‟ received
the best ratings with an average of 2.0, includ-
ing 11 individual ratings in the 'very good' (5
projects/programmes) and 'good' (6 pro-
jects/programmes) range. The 'effectiveness'
ratings of the evaluated development meas-
ures reached an average of 2.5. Again, most of
the individual success ratings were 'good' (9
projects/programmes). By contrast, the aver-
age ratings for the criteria 'impact' (2.7), 'effi-
ciency' (2.8) and 'sustainability' (2.9) were
more plainly situated at and around success
level 3, where most of the individual ratings
were 'satisfactory' and 'unsatisfactory'.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 11
Assessed DAC criterion Average rating
Relevance – 'Are we doing the right thing?' 2.0
Effectiveness – 'Are we achieving the objectives of the devel-
opment measure?' 2.5
Impact – 'Are we contributing to the achievement of overarching
development results?' 2.7
Efficiency – 'Are the objectives being achieved cost-effectively?' 2.8
Sustainability – 'Are the positive results durable?' 2.9
In the analysis of the objectives systems of the
evaluated projects/programmes, it becomes
clear that, above all, only a small number of
development measures include 'poverty rele-
vance' in their formulation of objectives and/or
include a 'poverty dimension' via appropriate
indicators: only two programmes (Kenya and
Zambia) have a recognisably formulated pov-
erty relevance in their definition of objectives.
Four programmes (Yemen, Kenya, Peru and
Zambia) have indicators for dealing with the
poverty dimension. This stands in contradiction
to the underlying service packages offered to
the BMZ, in which twelve projects/programmes
include poverty relevance, albeit with different
weightings, by way of the corresponding DAC
marker. Only a single programme (Africa su-
praregional) was classed EPA (general devel-
opment approach), while two pro-
grammes/projects (Kosovo, Turkey) neither
have a poverty marker in their respective offer,
nor a relevant recommendation included in
their evaluation report.
A further inconsistency arises from the fact that
ten of the evaluated projects/programmes
carry the DAC marker G1 (gender equality as a
secondary objective), but only three of those
ten development measures (Yemen, Kenya
and Zambia) have a corresponding indicator
for the 'gender dimension'.
With regard to their ecological impacts, all
projects/programmes were rated as UR1 or
better (India, Jordan and Mexico: UR2), while
only four (4) have indicators related to the 'en-
vironmental dimension' (India, Yemen, Jordan
and Kenya).
All in all, the assessment of all 15 individual
evaluations showed that the ratings of the de-
velopment measures on the basis of DAC
markers had, for the most part, not been per-
formed as specified and that corresponding
objectives and indicators had often not been
formulated with regard to these markers. On
most of the projects/ programmes reliable pov-
erty, gender or environmental impact studies
are not available. Instead, the respective re-
sults are presumed, i.e. aggregated up via long
results chains.
The various evaluations attached different
levels of importance to the DAC criteria (as
measured against the upgrading or downgrad-
ing of the standard weighting 2.0). Particular
importance was attached to the criteria 'rele-
vance' and 'sustainability' (average weighting
2.3) and to 'effectiveness' (2.2).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 12
DAC criterion Average weight-
ing
Number of upgrades
(weighting factor 3)
Number of down-
grades (weighting
factor 1)
Relevance 2.3 5 0
Effectiveness 2.2 3 0
Impact 2.1 2 1
Efficiency 2.0 2 1
Sustainability 2.3 6 1
In line with the impressions gained by the
evaluators who performed the individual
evaluations, the important success factors for
the 'relevance' of development measures in-
clude allowance for regional cooperation (par-
ticularly in the field of integrated water re-
sources management, IWRM), conformance of
the respective development measures with the
BMZ sector strategy and/or with the Millennium
Development Goals (MDGs), and systematic
attention both to a 'multi-level approach' and to
the principles of capacity development in the
design of the measures.
Regarding the relevance rating of the pro-
jects/programmes, the fact that only a small
portion of the measures were directly intended
to improve the living conditions of poorer popu-
lation groups, and that most of the pro-
jects/programmes had performed no poverty,
gender or environmental impact analyses,
clearly had a negative effect on the rating.
The evaluated projects/programmes mainly
achieve their direct results at the technical
improvement level and in terms of supply stan-
dards, but less so at the institutional level and
in connection with statutory change processes
and the creation of sustainable capacities.
There is little correlation between the em-
ployed volume of funding and the achievement
of objectives. The duration of advisory services
appears to be a major success factor for effec-
tiveness. The qualitative aspects of achieved
objectives (e.g. altered behaviour, satisfaction,
etc.) are still not collected consistently. As a
consequence, numerous results are presumed
without appropriate evidence.
In only three of the evaluated pro-
jects/programmes (Ghana, Peru and Zambia)
were any concrete links established between
the outcomes and the intended, or even unin-
tended, indirect results (impact). In those
cases institutionalised monitoring systems
were used to acquire the corresponding data at
an early stage and at regular intervals. In most
of the evaluations and projects/ programmes,
the overarching results were assessed on the
basis of assumptions and abstractions via long
results chains, without the corresponding in-
puts having been registered or investigated.
One success factor of prime importance for the
efficiency of development measures is the
embedment of TC interventions in sector inter-
ventions per se. Here, all evaluations identified
a definite need for improvement. Often, unsat-
isfactory efficiency is mainly attributable to
inadequate coordination and/or consensus-
building between DC organisations regarding
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 13
the use of resources, in addition to steering
and monitoring problems.
Sustainability is considered to be an extremely
important criterion, also in the course of im-
plementation, and correspondingly influences
the overall rating. A correlation was estab-
lished between the duration of the measure
and, hence, the duration of advisory services
and the sustainability rating of the respective
measure. The main success factors for sus-
tainability are the design of systematic capacity
development approaches and the considera-
tion of follow-up assistance phases - the latter
primarily in the case of cooperation between
TC interventions and financial cooperation
(FC) investments. As a rule, the basic chal-
lenge here is to flank and track the relatively
short implementation phases required for the
construction work with adequate measures to
create and/or strengthen appropriate opera-
tional and administrative structures.
The methodological approach of the evaluated
projects/programmes was given a positive
overall rating. Obviously, it was not possible in
some of the projects/programmes to implement
a systematic multi-level approach against the
backdrop of very complex partner structures.
The results constellation of the evaluated pro-
jects and programmes is focused at the meso
level in the area of personal capacity building
and organisational advisory services. System-
building and networking results are achieved
mainly by the more complex sector pro-
grammes (Yemen, Kenya, Zambia), in which
the multi-level approach was mainstreamed
with well-balanced attention to the economic,
social and ecological objectives dimensions.
Capacity development approaches formed part
of all projects/programmes. In many cases,
however, the underlying strategies were still
too unsystematic and non-transparent, particu-
larly for the partner structures. The time peri-
ods allowed for establishing capacities are
sometimes too short to achieve sustainable,
structurally relevant results. Due not least to
pronounced concentration on the meso level,
TC work sets high standards for steering by
the partner institutions. The recommendations
in the individual evaluations concentrate mainly
on the need to systematically broaden or focus
the corresponding interventions in order to
foster system building and to initiate or support
change processes.
Some of the projects' and programmes' impor-
tant innovations are already being dissemi-
nated via GTZ‟s knowledge management.
Nevertheless, the use of best practices in
German TC is still poorly developed.
The mode of service delivery was assessed as
appropriate in most of the evaluations. All in
all, both the German interface organisations
and the German consulting sector are very
broadly involved in delivering services on the
basis of an increasingly shared formulation of
objectives. The conceptual design of the hu-
man resource inputs underlying the implemen-
tation is still viewed as suboptimal sometimes.
In many places the integration of all service
providers into the overall programme, the flow
of information, the steering and, to a certain
extent, even the qualifications of the assigned
personnel continue to cause difficulties.
Cooperation agreements between the DC ac-
tors improve the joint planning and steering of
interventions. By contrast, the recording of
results via adequate monitoring systems as a
basis for further-improved steering still consti-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 14
tutes a weak point of most pro-
jects/programmes.
Important recommendations from the individual
evaluations refer primarily to the conceptual
orientation of the projects/programmes:
Include poverty and target-group analyses
in the planning of projects/programmes;
Closer orientation of the projects/ pro-
grammes to pro-poor results and change
processes;
More systematic attention to capacity de-
velopment at all levels of intervention;
Closer attention to networking and system
development in the multi-level approach
to ensure sustainability;
Improved recording of results via monitor-
ing systems and the use of project pro-
gress reviews and evaluations for a more
systematic appraisal of results;
More intensive coordination and steering
of German DC contributions in the water
sector in the partner countries on the ba-
sis of cooperation agreements.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 15
1. Introduction
1.1 Background, objectives and
object of the synthesis report
GTZ extended and optimised its evaluation
system in 2005. The adjusted evaluation sys-
tem is now consistently oriented towards the
evaluation criteria and principles of the Devel-
opment Assistance Committee of the Organi-
sation for Economic Cooperation and Devel-
opment (OECD-DAC). It is based both on self-
evaluations and independent evaluations2.
Within the scope of independent evaluations,
GTZ itself now commissions independent third
parties to perform the evaluations. Implemen-
tation contracts are concluded with independ-
ent research institutions, consulting firms and
external evaluators. The commissioning party
is the GTZ Evaluation Unit, which is organised
independently of the operative departments
and answers directly to the Office of the Man-
aging Directors. The unit bears overall respon-
sibility for planning and steering GTZ‟s evalua-
tion programme.
Independent evaluations serve especially to
meet accountability obligations regarding the
use of allocated resources, to improve the
management of projects and programmes and
to promote learning, both within the pro-
ject/programme and throughout the company.
Observations presented in this synthesis report
on the thematic priority area 'water' are based
exclusively on independent evaluation reports
concerning the relevant projects and pro-
grammes.
2 Until 2007, independent evaluations were still commonly
referred to as 'external evaluations'.
The planning and implementation of evalua-
tions is subject to the evaluation principles of
the Development Assistance Committee of the
Organisation for Economic Cooperation and
Development (OECD-DAC3):
Usefulness: The choice of project and the
terms of reference (ToR) reflect the inter-
est in obtaining specific findings both on
the part of those responsible for a project
and of the decision-makers. The evalua-
tion findings are communicated to all
stakeholders and incorporated into knowl-
edge management.
Participation: The partner institutions in
the partner country are involved in every-
thing from the formulation of the terms of
reference to the conclusion of the evalua-
tion.
Credibility: The evaluation system and the
evaluation programme are transparent,
and the evaluation findings, both positive
and negative, are published. The Evalua-
tion Unit, the commissioned research in-
stitutes and/or consulting firms, and the
evaluators performing the assessment are
all independent.
On the basis of methodically verified data,
evaluations are intended to provide substanti-
ated information about key criteria of the pro-
ject/programme assessment. The underlying
evaluation criteria are in line with the OECD-
DAC recommendations and are agreed as
binding between the BMZ and the German
implementing organisations:
3 'Organisation for Economic Cooperation and Develop-
ment - Development Assistance Committee'.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 16
the relevance of the implemented devel-
opment measure – 'Are we doing the right
thing?'
the results achieved by the pro-
ject/programme (effectiveness) – 'Are we
achieving the objectives of the develop-
ment measure?'
the respective contribution to overarching
development results (impact) – 'Are we
contributing to the achievement of over-
arching development results?'
the efficiency of the employed resources –
'Are the objectives being achieved cost-
effectively?'
the sustainability – 'Are the positive re-
sults durable?'
In addition, the evaluations are intended to
provide information on specific corporate and
development-policy issues. This refers in par-
ticular to how the development measures con-
tribute toward poverty reduction, the achieve-
ment of MDGs, gender equality and sustain-
able development. Moreover, appropriate
technical/sectoral issues are also evaluated.
Each year, the GTZ commissions approxi-
mately 30 independent evaluations (interim,
final and ex-post evaluations) with thematic
priority areas (2008: 'decentralisation' and 'wa-
ter'). Based on the reports concerning individ-
ual evaluations, the Evaluation Unit orders the
implementation of a 'cross-section evaluation4'
for each thematic priority area, which then
serves as the point of departure for a corre-
sponding 'synthesis report'. The latter is in-
4 Until 2008, cross-section evaluations were referred to as
'meta-evaluations'.
tended to facilitate institutional learning within
the GTZ and reporting to the BMZ. The results
and findings are incorporated into the GTZ's
knowledge management and are made avail-
able for public relations work.
The object of this synthesis report is the cross-
section evaluation of fifteen global evaluations
concerning the thematic priority area 'water'
from the year 20085. This includes:
eight interim evaluations (Africa suprare-
gional, Algeria, Yemen, Jordan, Kenya,
Peru, Zambia and Viet Nam);
three final evaluations (Ghana, Kosovo
and Mexico);
four ex-post evaluations (Ethiopia, India,
Niger and Turkey).
The cross-section evaluation encompasses the
following aspects:
overview and summary of the results of the
individual evaluations based on a quantita-
tively and qualitatively comparative as-
sessment;
identification of recurring strengths and
weaknesses and/or factors of success and
failure;
identification of overarching lessons
learned and recommendations for imple-
menting comparable future
projects/programmes and for the strategic
5 Altogether, sixteen (16) independent evaluations were
performed in 2008 for the thematic priority area 'water'.
The report on the jointly implemented ex-post evalua-tion of the cooperative project 'Rural Water Supply
East-Kilimanjaro, Tanzania' had not yet been coordi-
nated between the GTZ and the KfW at the time of the cross-section evaluation and was therefore not in-
cluded in the assessment.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 17
orientation of the GTZ's sector portfolio
'water'6.
1.2 For the reader's guidance
1.2.1 Overview of the evaluated pro-
jects and programmes
Table 1 on the following page surveys the
evaluations and corresponding projects and
programmes covered by the cross-section
evaluation. For the sake of straightforward,
systematic representation of the results, the
projects/programmes listed in this synthesis
report are arranged according to the type of
evaluation and/or alphabetically according to
the country, independently of any specific,
technical/ sectoral orientation or other criteria.
While all projects/programmes have to do with
the thematic priority area 'water', the specific
objectives and range of activities of the various
development measures differ substantially in
terms of content. The projects/programmes
can be basically differentiated with regard to
set priorities as belonging to the sectors or
sub-sectors (drinking) water supply, wastewa-
ter management/sanitation services, water
resources management and sector pol-
icy/regulation. Six (6) of the pro-
jects/programmes can be attributed to the pri-
ority area 'water supply', two (2) pro-
jects/programmes to the area 'water resources
management', and one (1) project/programme
to the area "wastewater management'. All
other projects/programmes deal with more
6 Selected results of the cross-section evaluation repre-
sented in connection with the GTZ Staff Conference in
Hohenroda on 6 July 2009.
than one priority area, i.e. have combined ob-
jectives definitions:
water resources management and sector
policy/regulation (1 project/ programme);
water resources management and irriga-
tion (1 project/programme);
water resources management and water
supply (1 project/ programme);
water supply and wastewater manage-
ment/sanitation services (1
project/programme);
water supply, wastewater manage-
ment/sanitation services and water re-
sources management (2
projects/programmes).
In the light of their objectives and respective
target groups, four of the evaluated pro-
jects/programmes can be attributed to the rural
sub-sector, while seven of the pro-
jects/programmes are situated in the urban
sub-sector. The objectives of four projects/
programmes relate both to the rural sub-sector
and to the urban sub-sector (Africa suprare-
gional, Algeria, Yemen and Ethiopia).
The regional distribution of the evaluated pro-
jects/programmes places eight of the pro-
jects/programmes in Africa, four of which relate
to the area 'Sub-Saharan Africa'. Two pro-
jects/programmes each from the regions Latin
America, Middle East and Southeast
Europe/Turkey were evaluated. Finally, one
project/programme from the region 'Asia' was
included in the cross-section evaluation.
Table 1: Overview of evaluated projects/programmes
Development measure Country Term Funding
volume7
Priority areas
Interim evaluations
Nile Water Initiative (NWI)
Planning and Management of Water Resources in the
Nile Basin
Africa
(AFR suprare-
gional
01/2002 - 2/2011 120 months EUR 10.0 M WRM
(urban & rural)
Integrated Water Management
Programme (PDEA)
Algeria
(DZA) 10/2003 - 2/2011 99 months EUR 9.7 M
WRM & SPR
(urban & rural)
Water Sector Programme (WSP) Institutional Develop-
ment of the Water Sector-
Yemen
(YEM) 07/2006 - 6/2015 114 months EUR 13.5 M
WS/WM & WRM
(urban & rural)
Water Management in Irrigated
Agriculture (WMIA)
Jordan
(JOR) 10/2006 - 9/2015 111 months EUR 2.5 M
WRM & IM
(rural)
Water Sector Reform Programme, Kenya (WSRP-KEN) Kenya 10/2003 - 2/2013 123 months EUR 13.9 M WS/WM & SPR
7 Includes either the total cost of TC (German contribution) in final and ex-post evaluations, or the planned total TC funding volume (German contribution) up to the end of the respective ongoing phase
of implementation in interim evaluations. Both in representation and analysis, the employment of funds is only accounted for in the case of 'Rural Water Supply in Maradi, Niger', since the evaluation was performed jointly on the basis of the harmonised system of objectives, and the FC inputs were implemented by the GTZ.
Development measure Country Term Funding
volume7
Priority areas
(KEN) (urban)
Drinking Water and Sanitary Programme (PROAGUA) Peru
(PER) 01/2002 - 2/2014 156 months EUR 8.2 M
WS
(urban)
Water Sector Reform Programme, Zambia (WSRP-ZMB) Zambia
(ZMB) 01/2004 - 2/2012 108 months EUR 8.1 M
WS/WM
(urban)
Wastewater and Solid Waste Management in Provincial
Centres (WDPC)
Viet Nam
(VNM) 02/2005 - 1/2013 107 months EUR 8.8 M
WM
(urban)
Final evaluations
Improvement of Water Supply - Eastern and Volta Re-
gion Assistance Programme (EVORAP)
Ghana
(GHA) 07/1998 - 6/2008 126 months EUR 6.2 M
WS
(urban)
Infrastructure Programme -
Water Supply and Wastewater
Disposal (WSWDP)
Kosovo
(KOS) 07/2000 - 6/2009 108 months EUR 7.7 M
WS
(urban)
Management of Water Catchment
Areas for the Río Lerma (GICA)
Mexico
(MEX) 04/2002 - 1/2008 70 months EUR 2.4 M
WRM & WS
(rural)
Ex-post evaluations
Improvement of Water Supply, Tigray Region (IWSTR) Ethiopia 11/1996 -09/2004 95 months EUR 2.9 M WS
Development measure Country Term Funding
volume7
Priority areas
(ETH) (urban & rural)
Indo-German Watershed Development Programme
(IGWDP)
India
(IND) 04/1989 - 6/2005 183 months EUR 12.8 M
WRM
(rural)
Rural Water Supply in Maradi (PHV/MI) Niger
(NER) 02/1995 -12/2003 107 months EUR 20.1 M
WS
(rural)
Upgrading of Municipal Services (QADI) Turkey
(TUR) 11/2002 - 2/2006 50 months EUR 1.7 M
WS
(urban)
WM: wastewater management/sanitation services WS: (drinking) water supply
SPR: sector policy and regulation WRM: water resources management
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 21
Altogether, eight of the fifteen reviewed meas-
ures fall into the category of programme build-
ing and are based on various parallel and/or
previous measures. Six of the pro-
jects/programmes (Yemen, Ghana, Kenya,
Niger, Peru and Vietnam) are 'cooperative
projects' between the GTZ and the KfW in
which the achievement of objectives is based
on close, synergetic intermeshing of the TC
and FC instruments. In the case of the 'Water
Sector Reform Programme, Kenya', the Ger-
man Development Service (DED) is formally
involved in the cooperation agreement. In 'In-
stitutional Development in the Water Sector,
Yemen', not only the DED but also the Federal
Institute for Geosciences and Natural Re-
sources (BGR), Inwent - Capacity Building
International, Germany and the Centre for
International Migration and Development (CIM)
are involved in the implementation at various
working levels by way of cooperation agree-
ments.
Only three (3) of the evaluated projects/ pro-
grammes (Africa supraregional, Algeria and
Turkey) were, or are still being, implemented
without contracts being awarded to the Ger-
man consulting sector. In all other pro-
jects/programmes, consulting firms assume
responsibility for implementing sub-sectors or
components. Two of the evaluated pro-
grammes (India and Mexico) were imple-
mented completely by a consulting firm work-
ing on behalf of the GTZ. In the 'Rural Water
Supply in Maradi, Niger' project, the FC advi-
sory services were provided by GTZ Interna-
tional Services on the basis of a negotiated
contract.
The overall volume of German TC contribu-
tions for these fifteen development measures
evaluated in the course of 2008 amounts to
EUR 128.5 million. Hence, the average amount
of funds employed per project/programme
comes to approx. EUR 8.6 million. The lowest
volume of all (EUR 1.7 million) - in combination
with the shortest term (50 months) - was em-
ployed for the 'Upgrading of Municipal Ser-
vices, Turkey' project. The largest volume
(EUR 20.1 million) was used for the 'Rural
Water Supply in Maradi, Niger' project. The
longest term, however, was that of the „Inte-
grated Indo-German Watershed Development
Programme, India' (183 months).
Three of the projects/programmes were termi-
nated early, i.e. ahead of the terms originally
planned:
'Rural Water Supply in Maradi, Niger': The
third TC promotion phase (follow-up
phase) January 2004 - December 2004
was not carried out due to funding prob-
lems on the part of the GTZ
'Management of Water Catchment Areas
for the Río Lerma, Mexico': The third pro-
motion phase January 2008 - March 2010
was not implemented because the pro-
gramme does not correspond to the prior-
ity areas defined for bilateral cooperation
with Mexico
'Upgrading of Municipal Services, Turkey':
Planned duration January 2000 - Decem-
ber 2005, began late due to the lack of a
programme agreement, and the term was
abbreviated due to design deficits and in-
tervention by other projects/ programmes.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 22
1.2.2 Methodological approach
The fifteen evaluation reports in their accepted
form served as the basis for the cross-section
evaluation and for producing the synthesis
report. In addition, the respective offers which
the GTZ had submitted to the BMZ were used
as a source of supplementary information as
well as for verifying the information contained
in the evaluation reports (particularly with re-
gard to objectives systems, markers, project
terms and funding volumes). The cross-section
evaluation rests exclusively on those two
sources of data for documenting the specific
situation of each project/programme; no addi-
tional studies or research were carried out. It
must therefore be called to the reader's atten-
tion that the results of the cross-section
evaluation depend very heavily both on the
quality of the evaluation reports and on the
informational content of the GTZ offers (see
also the following Section 1.2.3).
The evaluation reports were subjected to com-
prehensive quantitative and qualitative analy-
sis. The quantitative analysis was based firstly
on the objectives systems (objectives and indi-
cators) established for the development meas-
ures and, secondly, on the evaluation of indi-
vidual projects and programmes according to
the DAC criteria (see also Section 1.1) based
on the 'GTZ Guidelines on Evaluating the Suc-
cess of Projects/Programmes' (see Annex 3).
In that context, core statements based on the
respective unweighted ratings of the DAC crite-
ria were elaborated for the synthesis report,
because the weightings in the individual
evaluation reports were regarded as too incon-
sistent for general evaluation purposes. The
practice of weighting the DAC criteria was
discontinued by the Evaluation Unit in 2009 on
the basis of the non-transparent explanations
given for upgradings and downgradings.
The quantitative findings described further on
are the result of statistical analyses that should
be viewed in the light of the small random
sample (15 projects/programmes) and of the
highly heterogeneous content of the projects/
programmes (in combination with considera-
tion of three different types of evaluation).
While the stated averages are based on the
respective arithmetic mean values drawn from
the individual evaluations, potential dependen-
cies were accounted for by way of calculated
rank correlations (Spearman)8.
The qualitative analysis is based on the reflec-
tions and assessments of the development-
policy cross-cutting themes and sectoral as-
sessments documented in the evaluation re-
ports and on the expressed recommendations.
The evaluators made a qualified, though still
subjective, selection and interpretation of the
core statements in the evaluation reports. At
the same time, the argumentation chains in the
evaluation reports were subjected to a critical
review. That, too, constitutes a subjective re-
flection which, in a specific case, logically
leaves room for debate and alternative inter-
pretation.
The reference material for the qualitative
analysis of the evaluation reports consisted of
8 A rank correlation coefficient is a non-parametric meas-
ure of correlation - that is, it assesses how well an arbi-
trary monotonic function describes the relationship be-tween two variables without making any assumptions
about the frequency distribution of those variables. The
rank correlation coefficient is robust with respect to outliers and can be applied to non-linear correlations. Spearman’s rank correlation coefficient:
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 23
the usual GTZ and BMZ background docu-
ments, in particular the 'BMZ Water Sector
Strategy' (2006), the 'Guidelines for Successful
Poverty Reduction in the Work of GTZ' (2007),
'Sustainable Development - GTZ‟s Concept'
(2006, see also Annex 5) and the 'GTZ‟s Un-
derstanding of Capacity Development: A Guid-
ing Framework for Corporate Action' (2007,
see also Annex 6).
In the portrayal of factors of success and fail-
ure for evaluating individual DAC criteria (see
also Section 3), several examples were bor-
rowed from the evaluated projects/ pro-
grammes and explained. This is in line with the
expectations of the sectoral structure of the
GTZ‟s 'Water' Section and is intended to make
the respective ratings more understandable
with the help of practical information and an
explanation of the relationships involved.
1.2.3 Comparability of the evaluation
reports
The uniform structure of the evaluation reports
as well as the uniform assessment criteria
based on the DAC criteria ensure the compa-
rability of the individual evaluation reports cov-
ered by the cross-section evaluation (see also
Sections 1.1 and 1.2.2). The corresponding
terms of reference (see also Annex 2) and the
'GTZ Guidelines on Evaluating the Success of
Projects and Programmes' (see also Annex 3)
contain appropriate, uniform key questions for
use by the evaluators of the individual projects.
At the same time, it is prescribed that, in ad-
vance of any evaluation, the results constella-
tion of the project or programme be estab-
lished and the methodological approach of the
evaluation mission be accordingly harmonised
with it (Inception Report). The Evaluation Unit
is not only responsible for training the evalua-
tors but also for the quality of the respective
reports. To that end, the unit organises pre-
liminary talks and evaluation meetings to be
attended by the evaluators and representatives
of the participating organisational units of the
sectoral and regional structure. The results of
each individual evaluation are also subject to
comment by the participating implementing
structure.
It should, however, be kept in mind that, de-
spite extensive standardisation, the compara-
bility of the resulting evaluation reports remains
limited. The evaluated development measures
are naturally very heterogeneous both in terms
of their content and design. Also, the evalua-
tions were performed by evaluators from dif-
ferent institutions armed with different empirical
data and standards of comparison. Despite the
effort to remain objective, their respective in-
terpretations and assessments are very sub-
jective.
The 'atmospheric' circumstances of the respec-
tive evaluations are also important in this con-
nection. Acceptance and comprehension prob-
lems on the part of project staff regarding the
instrument of independent evaluation, in com-
bination with inappropriate, insensitive actions
on the part of the evaluators can disrupt the
course of evaluation and produce unbalanced
results. Ultimately, the quality of the evaluation
results depends crucially on collective attention
to the questions raised. Not infrequently, 'at-
mospheric disturbances' between the evalua-
tion teams and the project teams culminate in
evaluation reports that are marred by incom-
pleteness or unsatisfactory quality. In the pre-
sent, concrete evaluation sequence, such
problems were noticed in the evaluations of the
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 24
'Integrated Water Management Programme,
Algeria' and 'Water Sector Reform Programme,
Zambia'. In those two cases, the local project
team perceived the behaviour of the evaluators
as being inappropriate and inadequately
geared to cooperation. In the case of the
evaluation in Algeria, the project team also
considered the evaluators' analysis and con-
clusions to be incorrect.
The assessment of the provided evaluation
reports also brought to light that, despite very
intensive quality assurance on the part of the
Evaluation Unit, it was not possible to rectify all
of the weak points of the individual evaluation
reports with regard to „technical‟ aspects of the
evaluation process. In some cases, the evalua-
tors tasked with performing the cross-section
evaluation would have come to different con-
clusions about the quantitative rating of the
individual projects and programmes than the
individual evaluators. It must be unequivocally
noted that not all of the ratings and weightings
are transparent and/or consistent with the in-
formation (and corresponding key questions)
contained in the reports. Regarding the qualita-
tive ratings, the evaluation reports' comparabil-
ity can be summarised in a single sentence:
'not every report makes statements about ar-
eas on which statements should be made.'
Relevant statements are missing in a signifi-
cant share of the reports, particularly in the
analysis of objectives systems and the as-
sessment of important cross-cutting themes. At
the same time, the conceptual principles con-
tained in the relevant guidelines and recom-
mendations on capacity development or sus-
tainable development, for example, are not
always given consistent consideration in the
assessment.
The only consistently identifiable information
about the planned and implemented use of
funds in the various projects and programmes
covered by the reviewed evaluation reports,
relates to the German TC inputs with reference
to the GTZ offers submitted to the BMZ for
those projects and programmes. As far as
actually implemented partner contributions and
FC funding are concerned, a large portion of
the information contained in the evaluation
reports is based on assumptions and esti-
mates. Concrete data are only stated for some
of the projects and programmes. Conse-
quently, no analyses of the utilisation of funds
by the implementation and cooperation part-
ners were performed within the scope of the
cross-section evaluation.
Against that backdrop, analysis of the fifteen
evaluation reports on the priority area 'water'
only allows limited statements to be made
regarding the GTZ‟s overall portfolio for that
sector. The selected development measures
do not constitute a representative selection of
projects and programmes within the sector.
Nevertheless, recurrent core statements and
identified trends do make it possible to draw
conclusions about the need for further action
and/or about factors of success and failure. In
parallel, but only to a minor degree, good ap-
proaches and 'products' (best practices) in the
implementation of individual projects/ pro-
grammes can be identified and recommended.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 25
2. Objectives and activity areas
2.1 Brief description of the evalu-
ated projects and programmes
As mentioned in Section 1.2.1, the technical
orientation, setting of priorities and objectives
systems of the evaluated development meas-
ures differ in part substantially. Only a small
portion of the projects and programmes are
based on mutually comparable concepts (e.g.
Yemen, Kenya and Zambia). However, in view
of the quite complex objectives systems in
some cases, it does not appear expedient to
differentiate the projects/ programmes accord-
ing to sectoral priorities, nor would that be of
relevance for this cross-section evaluation. In
the following section, the projects and pro-
grammes are therefore grouped according to
type of evaluation and presented in the form of
brief individual summaries.
2.1.1 Interim evaluations
The following eight projects/programmes were
the subject of interim evaluation:
Nile Water Initiative – NWI
Planning and Management of Water Re-
sources in the Nile Basin, Africa suprare-
gional
Overall objective: 'The Water Policies of the
Nile Basin countries have converged closer
towards joint cross-border management of the
Nile.'
The purpose of this project is to integrate the
challenge of integrated, transboundary man-
agement of the Nile into the national water
policies by means of reform-policy advisory
services, hence achieving convergence of the
riparian policies. The IWRM approach provides
the foundation. The lead executing agency is
the Nile Basin Initiative (NBI), based in En-
tebbe, Uganda. NBI was established in 1998 to
encourage regional dialogue between the Nile
riparian states and, in doing so, reduce con-
flicts over the allocation of limited water re-
sources. The project's implementing organisa-
tions are the ministries of water affairs in the
NBI member countries (Egypt, Ethiopia, Bu-
rundi, Democratic Republic of the Congo,
Kenya, Rwanda, Sudan, Tanzania and
Uganda). The target groups are the water us-
ers of the Nile watershed. Being oriented to-
wards transboundary water resources man-
agement, the project is in conformance with
the BMZ strategies for regional development
cooperation in Africa. Since work is being done
at the regional (transnational), the national
political level as well as the institutional level,
both the macro level and the meso level are
involved. Basically, three different areas are
combined: confidence building, capacity devel-
opment among personnel at the water minis-
tries, and revision of water policies and legal
frameworks in the NBI member countries. Indi-
rectly, collaborative planning of investment
projects is to be simplified, hence promoting
economic development throughout the region.
Integrated Water Management Programme,
Algeria – PDEA
Overall objective: 'The capacities of the water-
sector institutions (Ministry of Water, regional
authorities) for integrated water management
are improved.'
This programme's priority area is to provide
water-policy advice. By incorporating the sus-
tainable use of resources into the country's
water policy, the programme aims to promote
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 26
the efficient use of water and curtail the other-
wise increasing use of untreated water. Capac-
ity development in the sector institutions is
being promoted by advisory and training
measures at the national, regional and local
levels (multi-level approach). The main themes
of action are integrated water resources man-
agement, sector planning, regulation of water
supply and sanitation (WS&S), organisational
development and knowledge management.
The lead executing agency is the Algerian
Ministry of Water, which is responsible for
managing the entire country's water resources.
The ministry is also the implementing organisa-
tion for WS&S planning and regulation. Other
implementing organisations include the na-
tional water utility, regional river basin agen-
cies, the water management agency in Béchar
and the community of Béni Abbès. The target
group is the entire population of Algeria. Im-
proved water resource development and man-
agement should lead indirectly to improve-
ments in the people's hygiene and health situa-
tion.
Water Sector Programme, Yemen - WSP
Institutional Development of the Water Sec-
tor
Overall objective: 'Municipal water supply,
wastewater management and water resources
management in the programme region is im-
proved.’
This cooperative programme aims to improve
urban WS&S and WRM in Yemen by providing
technical, process and organisational advisory
services. The five components of the coopera-
tive programme include a reform of the institu-
tional framework conditions, human resource
development, the development of independent,
commercialised WUs, IWRM and strengthen-
ing of local actors. The programme pursues a
multi-level approach. Its thematic priority areas
at the national level are decentralisation and
commercialisation, both of which are to be
implemented and institutionalised at the local-
level water utilities (WUs). The regional-level
priority area is the generation of WRM plans,
which are to be mainstreamed at both the na-
tional and the local level. At the regional level,
16 WUs receive advisory services geared to
enhancing their operational efficiency in terms
of organisational structure, technical processes
and procedures, and financial/commercial
matters, plus the introduction of a management
information system (MIS). At the local level,
governorate and district administrative authori-
ties receive technical and methodological sup-
port to create water catchment committees and
water user groups and conduct public aware-
ness campaigns and planning water resource
development measures. The lead executing
agency and implementing organisation is the
Ministry of Water and Environment. Additional
implementing organisations include the Na-
tional Water Resources Authority and the Min-
istry of Local Administration. Together with the
WUs and representatives of the local admini-
strations, technical and managerial personnel
from those institutions function as intermediar-
ies. The target group are the people of Yemen.
Efficient Water Resources Management in
Irrigated Agriculture in the Jordan Valley
and Highland Areas, Jordan – WMIA
Overall objective: 'Governmental water institu-
tions, operators and farmers manage the water
resources in an efficient and sustainable man-
ner.'
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 27
Component objective: 'The efficiency of water
resources management in irrigated agriculture
in the Jordan Valley and the Highlands is in-
creased.' 'Governmental water institutions,
operators and farmers manage the water re-
sources in an efficient and sustainable man-
ner.'
The evaluation was based solely on compo-
nent 2 of the 'Management of Water Re-
sources Programme, Jordan', which evolved
from four previous projects in 2006. For that
reason, the component objective is of rele-
vance. The thematic priority areas of the pro-
gramme are WS&S, irrigated agriculture and
WRM planning. The programme itself is de-
voted to the sustainable use of existing water
resources and to a fair reconciliation of conflict-
ing interests between households, the industry,
tourism and agriculture. It takes a multi-level
approach. At the national level, it contributes to
decision-making through capacity development
within the ministry and a critical examination of
the institutional framework conditions. Regional
WUs receive advisory support in technical and
operational matters. Their technical and man-
agement personnel are kept abreast of cutting-
edge technology and management practice by
appropriate training. At the local level, the pro-
gramme cooperates closely with irrigation
farmers. The programme component com-
prises three subcomponents: participatory
irrigation management with a focus on institu-
tion building and capacity development among
water-user associations, the use of treated
wastewater for irrigation with a focus on farmer
training, and groundwater management with a
focus on reduction of extraction rates. The use
of treated wastewater for irrigation purposes
makes the allocation of scarce freshwater re-
sources more flexible. A system for monitoring
the quality of soil, water and field crops was
introduced as part of the programme. The lead
executing agency is the Ministry of Water and
Irrigation. The implementing organisations are
the Jordanian Water Authority and the Ministry
of Agriculture. The programme's target groups
are the entire population of the country and, in
particular, irrigation farmers in the Jordan Val-
ley and surrounding highland areas.
Water Sector Reform Programme, Kenya –
WSRP-KEN
Overall objective: 'Sustainable access of the
urban poor to safe drinking water and basic
sanitation is increased and the water resources
management improved.'
The aim of this programme is to improve water
resources management (WRM), access to safe
drinking water and basic sanitary services for
the urban poor. Poverty relevance is a central
advisory element. The five components of this
programme are: support for the Water Ministry
in sector reform, regulation of the water sector
and pro-poor financing, commercialisation of
water utilities (WUs), capacity building for the
Water Resources Management Authority, and
introduction of recycling-oriented sanitation
services (EcoSan) - all of which reflect the
programme's multi-level approach. The lead
executing agency and implementing organisa-
tion is the Ministry of Water and Irrigation.
Other implementing organisations include the
regulatory committee, the trust fund, the Water
Resources Management Authority (WRMA)
and the commercialised water utilities (WUs) -
plus, at the local level, the water user associa-
tions and catchment area advisory committees.
The target group encompasses the water us-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 28
ers and the consumers of WS&S services in
poor urban areas. Access to WS&S services is
improved by the construction of water kiosks
and public sanitation centres. Indirectly, the
programme helps to improve the hygiene and
health situation and, hence, the living condi-
tions of the country's population.
Drinking Water and Sanitary Programme,
Peru – PROAGUA
Overall objective: 'The management of drinking
water supply and wastewater disposal in se-
lected cities of Peru is sustainably improved.'
This programme is geared to improving access
to drinking water while lowering the cost of
supplying water and improving its quality. Ad-
visory services provided to sector institutions
and water utilities are generally geared to im-
proving the operative and commercial man-
agement of eight WUs, implementing inte-
grated WS&S investments at two WUs to
strengthen the 'management of the social envi-
ronment', expanding the extent of community
participation, and generally improving the train-
ing structures for WUs. The programme works
with WUs at the meso level and with sector
institutions at the macro level. It aims to im-
prove the efficiency of water utility operations
and services in cities. At the sector level, a
national system of capacity development struc-
tures for advisory support and training, i.e. for
further development within the sector, is to be
created. The programme promotes sector re-
form. The lead executing agency is the Peru-
vian Vice Ministry of Construction and Sanita-
tion within the Ministry of Housing, Construc-
tion and Sanitation. The implementing partners
are the National Superintendence of Sanitation
Services, the WUs in selected small and me-
dium-size towns and cities and the National
Association of Water Utilities. The target group
is the population of the various WU service
areas, especially poor people living in urban
peripheries. The programme is intended to
improve living conditions and to reduce the
frequency of illnesses caused by contaminated
water.
Water Sector Reform Programme, Zambia –
WSRP-ZMB
Overall objective: 'The poor population's ac-
cess to safe drinking water and sanitary facili-
ties is improved and integrated water re-
sources management is introduced.'
The programme aims to improve the supply of
drinking water and sanitation services to poor
people and to introduce the IWRM approach. It
pursues a multi-level approach, as reflected in
the programme components. Those compo-
nents include policy advisory services and
regulation, advisory services for implementing
poverty-reduction approaches in the trust fund,
advisory services for the WUs for improving
their operative management, support for the
ministries with regard to sector coordination
and planning, and policy advisory services in
connection with IWRM. At the macro level, the
programme provides advisory services for
sector policy. At the meso level, it supports the
development of institutions, financing instru-
ments and national information systems. At the
micro level, it advises water supply and sanita-
tion companies and supports the development
of water-user groups. The lead executing
agency is the Ministry of Energy and Water
Development. The implementing organisations
are the Zambian National Water Supply and
Sanitation Council with its Water Watch
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 29
Groups, three commercial WUs, the Depart-
ment of Water Affairs and Water Board within
the ministry, the trust fund and the Ministry of
Local Government and Housing. The target
group is the poor peri-urban and rural popula-
tion. The programme is geared to improving
sanitation and hygiene conditions. The use of
unsafe water sources is to be further curbed by
the installation of water kiosks.
Wastewater and Solid Waste Management
in Provincial Centres, Viet Nam – WDPC
Overall objective: Municipal service providers
in six provincial centres are operating the im-
proved rainwater and sewage systems cost
efficiently and according to plan.
The project aims to improve wastewater and
solid waste management according to a con-
cept based on the multi-level approach. At the
Ministry of Construction, capacities for action in
the wastewater and solid waste sector are
being strengthened, and human resource train-
ing is to be supported by providing advisory
services to the technical and management
personnel. Provision of institutional advice to
the municipal enterprises is intended to ensure
proper operation of the surface-runoff and
sewage systems. Targeted training aims to
qualify the staff to operate and maintain the
systems. In addition, the legal basis for regulat-
ing the management of wastewater and solid
waste is being established at the macro level.
The Ministry of Construction is the lead execut-
ing agency, and the municipal wastewater
management companies (utilities) in the six
provincial capitals are the implementing or-
ganisations. The target group is the population
of the six partner communities. The project
contributes indirectly to health promotion by
improving settlement hygiene.
2.1.2 Final evaluations
Three of the fifteen projects/programmes were
concluded only recently but already subjected
to final evaluation:
Improvement of Water Supply - Eastern and
Volta Region Assistance Programme,
Ghana – EVORAP
Overall objective: The continual, needs-driven
supply of hygienically sound drinking water to
the populations of selected small towns in the
Eastern and Volta Regions is ensured.
The intention of this programme was to con-
tribute to a qualitatively and quantitatively im-
proved drinking water supply in the Volta and
Eastern region. Its multi-level approach en-
compassed provision of technical, organisa-
tional and policy advisory services. The the-
matic priority areas of the advisory services
and training at the national, regional and local
levels were the operation and management of
provincial water supply systems, organisational
development and hygiene education. At the
macro level, the advisory services were geared
to reforming the sector policy and legal frame-
work. At the micro level, the programme was
oriented towards rehabilitating the drinking
water supply networks in the selected small
towns with the active participation of the local
population. The lead executing agency was the
national Community Water and Sanitation
Agency (CWSA - a regulatory and controlling
body responsible for water supply in the rural
areas and small towns). The implementing
organisations were the District Water and Sani-
tation Teams (DWST) and the Water and Sani-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 30
tation Development Boards (WSDB). The tar-
get group of the programme comprised the
populations of 29 small towns in the Eastern
and Volta Regions, including the staff of the
sector institutions. As an indirect impact, the
project was intended to contribute to an im-
provement in the hygiene and health situation
of the population and, hence, their quality of
life.
Infrastructure Programme - Water Supply
and Wastewater Disposal, Kosovo -
WSWDP
Overall objective: Operational efficiency in the
regional water companies Peja and Prizren is
enhanced.
This programme was geared to enhancing the
operational efficiency of the regional water
companies in Peja and Prizren. It was strictly
confined to the WU level. The companies in
question received support for their organisa-
tional development, financial and customer
management, mapping of their WS&S net-
works and drinking water connections, plant
evaluation, water loss management, and the
introduction of a geographical information sys-
tem (GIS). A management information system
(MIS) was also introduced in order to improve
the companies' financial decision-making basis
by means of an auditable bookkeeping system.
A maintenance database was constructed and
maintenance plans prepared in order to ensure
trouble-free operation over the long term, as
well as the preservation of the plants them-
selves and the investments they represent.
WS&S is intended to operate with commercial
efficiency based on the operating-cost-
recovery principle and, later on, with full cost
coverage. The lead executing agency was the
Government of Kosovo, and the implementing
organisations were the regional water utility
companies in Peja and Prizren. The target
group was the population of both regions. Indi-
rectly, the programme aimed to prepare the
companies to become independent legal enti-
ties and be responsible for WS&S manage-
ment. The people were supposed to gain en-
sured access to WS&S.
Management of Water Catchment Areas for
the Río Lerma in the Toluca Valley and for
the Río Balsas, Mexico – GICA
Overall objective: Management of the aquifer
catchment in Toluca Valley by water sector
actors is improved.
This project aimed to improve water resources
management in the state of Mexico by creating
an institutional structure for a water resources
management plan and by improving the quali-
fications of actors in the water sector. This
encompassed process advisory services, train-
ing and follow-up assistance of pilot measures.
The project concept was based on a multi-level
approach, meaning that the work was per-
formed at state, regional and local levels. Wa-
ter resources management activities at state
institutions were systemised, advisory services
and training offered to partner organisations
and principal actors, and water users mobilised
and educated about the conscious use of wa-
ter. The lead executing agency was the na-
tional water commission (CONAGUA), and the
regional CONAGUA office in the state of Mex-
ico was the implementing partner. The target
group was the population of the project region.
Indirectly, the project intended to promote sta-
bilisation of decentralised water resources
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 31
management while improving the region's
drinking water supply situation.
With regard to the comparability of these
evaluation reports, it should be noted in con-
nection with the 'Infrastructure Programme
Water Supply and Wastewater Disposal, Kos-
ovo' that the final evaluation was performed
eight months before the end of the project term
(June 2009), i.e. in October and November of
2008. Hence, in the opinion of the evaluators, it
cannot, in the literal sense, be regarded as a
final evaluation.
2.1.3 Ex-post evaluations
Four of the evaluated projects/programmes
were concluded some time ago and therefore
assessed within the framework of an ex-post
evaluation:
Improvement of Water Supply in the Tigray
Region, Ethiopia – IWSTR
Overall objective: A reliable and sustainable
water supply management system for provin-
cial and rural areas is established, is opera-
tional at selected locations and can be repli-
cated by the Ethiopian agencies concerned.
This project aimed to introduce a water supply
management system in the provincial and rural
region. The project focused mainly on meas-
ures at the regional administrative level and
consisted largely of training and the introduc-
tion of software for spare-parts management at
repair shops. Other contributions included the
drilling of wells and the supply of pumps and
large-scale water meters. The lead executing
agency was the Ministry of Water Resources.
The implementing organisation was the Tigray
Water Resources Development, Mines and
Energy Bureau. The target group was the
population of 8 pilot and 19 replication towns.
An indirect aim of the project was to improve
the supply of water throughout the Tigray Re-
gion.
Indo-German Watershed Development Pro-
gramme, India – IGWDP
Overall objective: Governmental and nongov-
ernmental implementing organisations are
applying methods for improving the sustainabil-
ity and breadth of impact in integrated, gov-
ernment-sponsored watershed management
programmes.
The project was geared to improving public
programmes for management of watershed
areas. It provided advisory services to gov-
ernmental and non-governmental institutions
and self-help groups and implemented capac-
ity development measures within the partner
organisations. The project pursued a multi-
level approach and operated at national, re-
gional and local levels. The lead executing
agency and implementing organisation was the
Ministry of Agriculture. Ministerial staff and
evaluators from other governmental and non-
governmental organisations served as media-
tors. The target group was made up of land
users in selected, representative watersheds.
The indirect aim of the project was to improve
the supply of food and household income.
Rural Water Supply in Maradi, Niger –
PHV/MI
Overall objective: Continuous, year-round sup-
ply of safe drinking water to the rural popula-
tion of the programme region is assured.
This project aimed to improve the supply of
drinking water in rural areas of the Maradi re-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 32
gion by installing/rehabilitating dug wells. The
measures comprised the application of new
and appropriately modified dug-well construc-
tion techniques and the creation of a basis for
controlled, self-responsible, resource-
conserving operation of the wells by the village
communities. The project therefore intervened
primarily at the local level. The lead executing
agency was the Ministry of Water. The regional
water directorate served as the implementing
partner, and the village communities were the
target group. By increasing the supply of water
and reducing the time required for fetching
water, the project was to have the indirect ef-
fect of expanding/securing the economic activi-
ties and increasing household income accord-
ingly.
Upgrading of Municipal Services, Turkey
Overall objective: The inter-institutional, pro-
fessional network and the relevant training
measures in the water supply, wastewater
disposal and waste management sector are
improving the political, institutional and techni-
cal framework conditions for the provision of
municipal services in selected cities.
The project aimed to improve the framework
conditions for WS&S. By means of a thorough
administrative reform and attendant decentrali-
sation, the municipalities and/or the concerned
WUs were to be given additional opportunities
for engaging in self-responsible management.
The efforts focused on establishing an inter-
institutional, professional network. The project
operated mainly at the municipal administrative
level. The lead executing agency was the Min-
istry of the Interior, and the implementing or-
ganisations were the Institute of Public Admini-
stration for Turkey and the Middle East and the
Turkish Association of Towns and Municipali-
ties. The target group consisted of the popula-
tions of the towns concerned. The project
measures addressed policy-makers as well as
technical and administrative personnel of city
and municipal facilities. Indirectly, the project
was intended to improve the quality and sus-
tainability of municipal services and as well as
public awareness of the problem through in-
tensified public relations activities.
2.2 Analysis of objectives systems
A major share of the objectives definitions
underlying the evaluated projects and pro-
grammes contain at least two partial objectives
and/or sub-sectors of the water sector (see
also Section 1.2.1). Eight of the evaluated
projects/programmes have a very complex
objectives system made up of phases, compo-
nent objectives and corresponding indicators.
The programme 'Institutional Development of
the Water Sector, Yemen', for example, has a
total of 22 overall-objective and component-
objective indicators, and the 'Water Sector
Reform Programme, Kenya' has a total of 24
overall-objective and component-objective
indicators.
In the evaluation reports, the objectives sys-
tems formulated for the projects and pro-
grammes were given different ratings, but most
of them were classified as suitable. It was ob-
served in most of the evaluations that, in the
course of implementation, the indicator sys-
tems in particular required adjustment with
respect to the formulations in the GTZ offers,
because the levels of intervention and, often,
the results constellations tend to vary. For
three of the projects/programmes (Kosovo,
Niger, Turkey), the respective evaluation mis-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 33
sion reformulated the objectives systems for
evaluation purposes, because the stated ob-
jectives, indicators and results chains were
considered imprecise with regard to results
constellations and causalities and did not allow
comprehensive documentation of results di-
mensions. It must be noted in that connection
that some of the projects/programmes (in par-
ticular with regard to the final and ex-post
evaluations) were conceived and designed
prior to introduction of the Development-Policy
Framework for Contracts and Cooperation
(AURA) and were therefore still subject to the
old objectives-oriented project planning
(ZOPP) system. No results models from the
implementation phase are available for these
projects/programmes, Instead, the correspond-
ing models were subsequently formulated
within the scope of evaluation.
The present objectives systems are discussed
below with regard to the setting of priorities in
their activity areas and results levels. It should
be pointed out that in-depth reflections on the
respective definitions of objectives are only to
be found in some of the evaluation reports (8
of 15). None of the evaluation reports include a
pertinent analysis of how well the poverty,
gender and environmental dimensions are
included in the formulation of objectives.
2.2.1 Activity areas, priority areas
and indicators
Figure 1 below illustrates the distribution of
priority areas and activity areas based on the
ormulated objectives systems. There is a con-
spicuous focus on two areas: 'water supply'
and 'water resources management', while only
a very small portion of the pro-
jects/programmes have the activity area
'wastewater management/ sanitation services'
as part of their objectives system.
Urban areas account for approx. 60 % of the
set priorities in the projects and programmes,
and it should be noted that four of the pro-
jects/programmes are or were active in both
urban and rural areas (Figure 2).
The heterogeneity of the projects/ programmes
makes it hard to identify clear-cut trends in the
development of objectives systems. It is ap-
parent, however, that the themes '(drinking)
water supply', 'water resources management'
and 'water management' have been continu-
ously included in most objectives systems
since 1995/1996. The themes 'wastewater
management' and 'sanitation services', by con-
trast, were only found in the objectives defined
for two projects/ programmes, beginning in
2003.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 34
Figure 2: Proportions of urban and rural priorities in the evaluated pro-jects/programmes
Since 2001, the urban water sector has been
given increasing attention in the objectives
systems of projects and programmes, while
the rural water sector has found very little con-
sideration at the objectives level. This corre-
sponds to the trends established in the pro-
gramming of new projects and programmes
according to BMZ directives.
In the opinion of the evaluators, only little at-
tention was paid to the dimensions 'poverty',
'gender' and 'sustainability' at the formulation-
of-objectives level. Poverty relevance was only
included at the objectives level of two evalu-
ated projects/programmes (Kenya and Zam-
bia). The concept of sustainability is mentioned
in four of the reviewed objectives systems
(Ethiopia, India, Kenya, Peru), but since 2004,
that dimension has been scarcely included at
all in the formulation of objectives (namely in
only one of nine objectives systems formulated
since then).
9%
43%17%
26%4%
Sector Policy / Regulation
Water Supply
58%
42%
urban
rural
Figure 1: Distribution of activity areas in the evaluated
projects/programmes
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 35
Figure 3: Trends in the thematic orientation of objectives systems
At the level of indicators for achievement of
objectives (decisive for the determination of
effectiveness), the reviewed objectives sys-
tems clearly emphasise the results constella-
tion 'drinking water supply - standards of sup-
ply - operational efficiency'. Corresponding
indicators were formulated for ten of the fifteen
development measures, albeit to varying de-
grees. Altogether, some 48% of the indicator
formulations can be allotted to that results
constellation. This is somewhat surprising,
since only one programme (Kosovo) displayed
a direct connection between the actual formu-
lation of objectives and improvements in the
operation of water utilities. In addition, only six
of the projects/programmes intervene at the
WU level in the sense of improving the supply
of drinking water. (Despite the formulation of
corresponding indicators, no direct correlation
was established for the projects/programmes
in Algeria, Ethiopia, Niger and Mexico.) Con-
spicuously, too, indicators concerning the
'wastewater management/sanitation services'
sector were formulated for only three of the
projects/programmes (Algeria, Yemen and Viet
Nam), whereas the project in Jordan had one
indicator relating to the area 'reuse of re-
claimed water'.
The results constellation 'sector policy - regula-
tion - governance' is found in the indicator sys-
tems of two programmes (Africa supraregional
and Algeria).
IWRM principles were taken into account in the
indicator systems of only five of the evaluated
projects/programmes (Algeria, Yemen, Kenya,
Mexico, Zambia), despite the fact that the con-
cepts 'water resources management' and/or
'water management' were included in the for-
Water Supply
Management of Water
Resources /
Water Management
Wastewater
Management /
Sanitation
urban
20081995 2000
rural
constant since 1995
constant since 1996
increasing since 2003
decreasing strongly since 2001
increasing strongly since 2001
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 36
mulation of objectives for nine of the evaluated
projects.
In the objectives systems of the evaluated
projects/programmes, indicators designed to
register poverty, gender and environmental
dimensions are included on a very small scale
that is not considered appropriate in principle
to the objectives of German DC. Only four
projects/ programmes have an indicator for
determining the poverty dimension (Yemen,
Kenya, Mexico and Zambia), and only three
(Yemen, Kenya and Zambia) register the gen-
der dimension via appropriate indicators. Like-
wise four projects/programmes (India, Yemen,
Jordan and Kenya) have objectives systems
that include indicators for the environmental
dimension. This stands in stark contradiction to
the fact that, as dealt with in Sections 4.1 and
4.2 below, most of the projects/programmes
had DAC markers identifying the poverty, gen-
der and environmental dimensions as secon-
dary objectives.
All told, it can be said of the project/ pro-
gramme indicator systems that only a very
small share of the formulated appraisal factors
relate to concrete, results-based 'outcomes'
(e.g. supplies, poverty reduction, health). At
the same time, measurable factors for estab-
lishing indirect results are also missing.
Good examples of (objectives-level) indicators for documenting the poverty dimension:
'Institutional Development of the Water Sector, Yemen': 'A survey conducted among the popula-
tion living in the assisted municipalities - with special focus on representatives of local women's
groups - confirms that measures in water supply and sewerage have contributed to an im-
provement in their overall living conditions’.
'Water Sector Reform Programme, Kenya': 'At least 1.6 million poor people in the assisted
towns have more access to safe, affordable drinking water’ (report by water sector trust fund).
'Water Sector Reform Programme, Zambia': 'The share of people in poor urban settlements with
access to hygienically safe drinking water at socially acceptable prices has risen from 54%
(2005) to 80% (reports by the sector information system)’.
All projects and programmes have very long
results chains, and the actual results constella-
tion of the respective project or programme is
therefore projected behind the 'attribution gap'
(range of indirect and highly aggregated re-
sults) in the results model. Results counting as
'indirect' can therefore only be 'sensed', not
tangibly registered. The question arises as to
whether the cautious formulation of indicators
could be attributable to a certain desire to 'stay
on the safe side'. A little more courage to 'face
the consequences' of choosing and formulating
indicators would be appropriate, since many
evaluation reports refer to 'palpable' results in
the relevant project context, whereas the cor-
responding databases are missing due to im-
precise formulation of indicators or a lack of set
priorities. In the opinion of the evaluators, the
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 37
evaluated projects and programmes can be
presumed to have achieved far more results
than were actually recorded.
2.2.2 Intervention and results
levels
German TC projects and programmes are
designed according to a results model in which
relationships are established between the im-
plemented activities and the ensuing direct and
indirect results. The GTZ results model de-
scribes the connection between a project or
programme and development-policy changes
by way of the project's or programme's active
inputs, activities, outputs, the use of those
outputs by other actors, the direct benefits
deriving from such use (outcomes or direct
results) and other forms of developmental pro-
gress that can be described on the basis of
plausible hypotheses as indirect benefits of the
respective project or programme.
By way of analogy to the major differences in
thematic and conceptual orientation, the over-
view of the evaluated projects and pro-
grammes in the water portfolio shows a very
heterogeneous pattern with respect to the di-
rect results level of the respective projects and
programmes. Of decisive importance here too
is the structuring of a 'multi-level approach' in
the design of the projects/programmes. The
following illustration (Figure 4) attempts to
show how the anticipated direct results of the
selected projects and programmes concentrate
on the meso level with respect to the develop-
ment and strengthening of sector institutions
and their intermediaries, or to 'networking'
within the water sector. Some 75% of the pro-
jects and programmes achieve results in that
area. Roughly two thirds of the projects and
programmes are geared to improving sector
management and regulation. However, only
about one third of the projects and pro-
grammes aim to have a direct effect on
changes in the framework conditions and re-
requisites for development at the macro level.
The proportion of projects having concrete
micro-level results with regard to quality-of-life
improvements for the population is situated at
around 40%. Another two thirds of the projects
and programmes achieve results in the devel-
opment of capacities and competencies among
the intermediaries and target groups, hence
establishing an essential basis for improving
living conditions by use of investments and
sector-specific know-how.
The diagram also visualises the fact that only
three of the programmes (Yemen, Kenya and
Zambia) have a complete, conceptually main-
streamed multi-level approach spanning all
results levels. It is also apparent that, with the
exception of the three projects/programmes
'Nile Water Initiative, Africa supraregional',
'Infrastructure Programme, Kosovo' and 'Rural
Water Supply in Maradi, Niger', all projects and
programmes have substantial results constella-
tions that span all levels. On the basis of the
selected projects and programmes, the results
orientation appears not to be subject to any
time-dependent trends.
As far as the programme „Nile Water Initiative,
Africa supraregional' is concerned, it should be
explained that the multi-level approach is dif-
ferentiated within the regional context. Within
the context of a supraregional programme, the
macro level is the level occupied by intergov-
ernmental bodies, while the meso level is con-
cerned with the respectively highest national
institutions.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 38
The 'Rural Water Supply in Maradi, Niger' pro-
ject purposely dispensed with a multi-level
approach in the project design. This was at-
tributed to excessive weakness on the part of
the lead executing agency and a great need for
direct intervention at the target-group level
(construction of dug wells for the supply of
water). Planned prior to 1995, this develop-
ment measure constitutes a type of project for
which a rationale can hardly be found any-
more, as it predated both the introduction of
the Development Policy Framework for Con-
tracts and Cooperation - AURA (2002) and the
orientation of TC towards results and sustain-
able development.
Figure 4: Direct results levels of the evaluated projects/programmes
In-depth information about the objectives
and results of the individual projects and
programmes can be found in the respec-
tive brief descriptions contained in Section
2.1. Further analyses and observations re-
garding the overarching development-
policy objectives of the projects and pro-
grammes are contained in Section 3.1
(relevance rating), 3.3 (impact rating) and
4.1 (poverty reduction and MDGs).
Macro Level
Meso Level
Micro Level
Improvement of Living
Conditions
Capacity Development
of Intermediate
Organisations
Capacity Development
of Sector Institutions
Improvement of
Framework Conditions
Af
ri
ca
N
.A
.
Al
ge
ri
a
Ye
me
n
Jo
rd
an
Ke
ny
a
Pe
ru
Za
mb
ia
Vi
et
na
m
Gh
an
a
Ko
so
voM
ex
ic
o
Et
hi
op
ia
In
di
a
Ni
ge
r
Tu
rk
ey
Capacity Development
of Target Groups
Sector Coordination
and Regulation
"Sector Networking"
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 39
3. Rating according to DAC Criteria
In the following sections, the evaluated pro-
jects and programmes are rated according to
the DAC criteria, and the results are corre-
spondingly summarised. To improve the trans-
parency and traceability of the respective rat-
ings, the factors determining success and fail-
ure for rating the individual DAC criteria are
explained by means of several examples from
the evaluated projects and programmes.
Within the scope of the independent evalua-
tions, the ratings for the DAC criteria 'rele-
vance', 'effectiveness', 'impact' and 'efficiency‟,
as well as the overall rating, are based on a
six-point scale (see also Annex 3: GTZ Guide-
lines on Evaluating the Success of Projects
and Programmes):
Table 2: Assessment scale for the DAC criteria relevance, effectiveness, impact and efficiency,
and for the overall rating
Success level Rating Description
'Successful'
'very good'
(1) Very good results, significantly better than expected.
'good'
(2)
Good results, fully in line with expectations, with no sig-
nificant defects.
'satisfactory'
(3)
Satisfactory results; falling short of expectations but with
positive results dominant.
'Unsuccessful'
'unsatisfactory'
(level 4)
Unsatisfactory results; significantly below expectations,
and negative results dominate despite identifiable positive
results.
'inadequate'
(5)
Clearly inadequate results: despite several positive partial
results, the negative results clearly dominate.
'useless'
(6)
The project/programme is useless, or the situation has
deteriorated on balance.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 40
The criterion 'sustainability' is rated according to the following four-point scale:
Table 3: Assessment scale for the DAC criterion sustainability
Sustainability level Description
'very good'
(level 1)
With a high degree of probability, the overall success of the project/programme
(positive to date) will continue unchanged or even increase.
'good'
(level 2)
With a high degree of probability, the overall success of the project/programme
(positive to date) will decrease only minimally and remain significantly positive
overall (normal situation – to be expected)
'satisfactory'
(level 3)
With a high degree of probability, the overall success of the project/programme
(positive to date) will decrease significantly but remain positive.
'inadequate'
(level 4)
The overall success of the project/programme is inadequate at the time of
evaluation, and there is a high degree of probability that it will not improve.
According to the applicable rating principles, a
project or programme can only be given a
'successful‟ overall rating in terms of develop-
ment policy if its direct results (effectiveness),
indirect results (impact or „overarching devel-
opment results‟) and sustainability all achieve a
rating of at least „satisfactory‟ (level 3).
In the present evaluation sequence for the
priority area 'water', a decision was taken and
justified in each evaluation and for each of the
five DAC assessment criteria as to whether it
was „very important‟ (weighting 3), „important‟
(weighting 2) or „less important‟ (weighting 1)
within the specific context of the project or
programme. In the absence of a special reason
making it more or less important, a given crite-
rion was held to be „important‟ (standard
weighting 2). In the evaluation of 'Rural Water
Supply in Maradi, Niger' (joint GTZ/KfW ex-
post evaluation), the performance assessment
for all DAC criteria was given a standard
weighting of 2.
3.1 Relevance
3.1.1 Relevance rating
The term relevance defines the extent to which
the objectives of a development measure
match the needs of the target groups, the poli-
cies of the partner countries and partner insti-
tutions, global development goals and the ba-
sic development-policy orientation of the Ger-
man Federal Government.
'Are we doing the right thing?'
With 2.0 ('good') as its average rating, rele-
vance emerged as the most successful evalua-
tion criterion. In fourteen of fifteen pro-
jects/programmes, the development measure
was successfully aligned with the policies of
the partner country, the needs of the respec-
tive target groups, the global development
objectives, and the basic development-policy
orientation of the German Federal Govern-
ment. Five of the projects/programmes (Africa
supraregional, Jordan, Peru, Viet Nam and
India) received a relevance rating of 'very
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 41
good'. Four of the projects/programmes are
still ongoing and were rated within the scope of
an interim evaluation. Only one programme
(Algeria) was rated 'unsatisfactory' according
to an interim evaluation (see also Table 4 and
Figure 5).
The average relevance rating varies according
to when the evaluation took place: While the
average rating in the interim evaluations, at
1.8, was still well within the good range, it
dropped to 2.3 or 2.5 in the final and ex-post
evaluations. That trend is attributable to the
fact that ratings given in connection with in-
terim evaluations are still strongly geared to-
wards the intended, conceptually targeted
orientation of the respective project or pro-
gramme, as measured against the develop-
ment-policy directives. To the extent that de-
viations or limitations are recognised at that
early stage, a need for realignment is usually
formulated with reliance on the respective per-
formance capability, without downgrading the
rating. This way, interim evaluation is able to
inject fundamentally positive impulses into the
projects and programmes, and realignment is
still possible thanks to the remaining length of
term. For a final evaluation, however, the situa-
tion differs in that corrections are no longer
possible, and any given limitations must, ulti-
mately, be rated with respect to the actually
achieved development-policy orientation. The
respective rating is then, apparently, more
critical in connection with an ex-post evaluation
(as compared to final evaluation), particularly
with regard to the orientation of interventions
towards the actual requirements of the target
groups.
33%
40%
20%7%
very good
good
satisfactory
not satisfactory
Figure 5: Distribution of relevance ratings
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 42
Table 4: Relevance ratings of evaluated projects/programmes
Country Project/Programme Type of
evaluation Rating Weighting
Africa su-
praregional NWI
Interim
evaluations
1 2
Algeria PDEA 4 3
Yemen WSP 2 3
Jordan WMIA 1 2
Kenya WSRP-KEN 2 3
Peru PROAGUA 1 3
Zambia WSRP-ZMB 2 3
Viet Nam WDPC 1 2
Ghana EVORAP Final
evaluations
3 2
Kosovo WSWDP 2 2
Mexico GICA 2 2
Ethiopia IWSTR
Ex-post
evaluations
2 2
India IGWDP 2 2
Niger PHV/MI 3 2
Turkey QADI 3 2
Overall (average) 2.0 2.3
Orientation towards key questions for as-
sessing the 'relevance' of a development
measure leads to the success factors. The
evaluated development measures were
rated „positive‟ in the individual evalua-
tions, if
they are in line with the policies and
strategies of the partner country and part-
ner institutions;
poverty analyses are conducted, and
poverty reduction was specified as a cen-
tral objective in accordance with poverty
reduction strategies;
thematic priority areas of BMZ for the wa-
ter sector are included in the design of the
measure and are in line with the sector-
specific policies, country strategies and
position papers (basic development-policy
orientation of the German Federal Gov-
ernment);
conformance with international standards
and conventions is given;
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 43
both the partners and the target group
attach high priority to the measures;
integrative and cooperative elements are
included, and the region in question is re-
garded as an integral whole in the case of
transboundary initiatives;
capacity development measures are in-
cluded;
a multi-level approach is accounted for;
gender analyses were conducted, and a
gender-responsive concept is in place.
Within the scope of the evaluations, the
following factors determining failure led to
downgrading of the relevance rating. De-
velopment measures were rated „nega-
tive‟ if
the core problem is not recognised, hence
resulting in erroneous setting of priorities;
the national strategies stand in conflict
with the basic development-policy orienta-
tion;
there is a conflict of objectives between
the short-term and long-term orientation of
the project or programme;
no relevance to poverty reduction is es-
tablished, and no poverty analysis was
conducted;
a gender-responsive conceptual design
and a gender analysis are lacking;
a multi-level approach is inadequately
implemented in that too few interventions
are provided at the target-group level (mi-
cro level).
Regarding the conduct of a poverty analysis, it
must be conceded that it is hardly possible to
perform the relevant investigations for regional
programmes as complex as the „Nile Water
Initiative, Africa supraregional'. In principle, the
requisite time and effort would be prohibitive –
a fact that should be considered for the rele-
vant rating.
The 'Indo-German Watershed Development
Programme, India' received a very good rele-
vance rating. The objective of the programme,
namely to improve the living conditions of peo-
ple living in the water catchment areas, was in
accordance with the essential national and
international issues and priorities of the Gov-
ernment of India at all levels. The programme
was tailored to the particular requirements of
the target groups, who were involved in its
planning and implementation from the very
beginning. All partner institutions and involved
intermediaries attached high priority to the
programme. With its priority set on the devel-
opment and piloting of innovative methods and
techniques, the programme was in line with the
sector-specific policies of the partner countries,
with the policy of the Government of India and
with that of the non-governmental organisa-
tions (NGOs) participating as intermediaries.
This includes both the BMZ country strategy
for India, in which 'watershed management' is
stated as a priority, and the GTZ sectoral ap-
proach for rural development. Moreover, it
addressed central concerns of the UN conven-
tion to combat desertification and strategies for
adapting to climate change. The programme
complied with generally established technolo-
gies and encompassed income-generating
measures on private and public property, so
that both property owners and the landless
population benefited. The multi-level approach
was rendered evident by interventions at the
national, regional and local levels.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 44
The 'Nile Water Initiative' (Africa suprare-
gional) also received a very good relevance
rating. As part of the Shared Vision Pro-
gramme of the Nile Basin Initiative on the inte-
grated development and administration of
shared water resources, the project helps ri-
parian states reform their national water poli-
cies and adjust them to the shared vision. The
project is in line with the MDGs and the BMZ
water sector strategy with its prioritisation of
IWRM, river basin management and trans-
boundary cooperation in Africa. The relevant
evaluation also comes to the positive conclu-
sion that the project pursues the GTZ strategy
for capacity development and the multi-level
approach in the multinational context. The
project concept is such that the effects on wa-
ter users in the Nile Basin (micro level) are
achieved via long results chains. The project's
strong presence at the macro and meso levels
is of decisive importance for its relevance.
Actually, the intended impacts on the target
group should have been described more
closely in the evaluation in order to better sub-
stantiate the rating of relevance as 'very good'.
The 'relevance' of the development meas-
ures in Yemen, Kenya and Zambia was rated
'good' in the interim evaluations. In the opinion
of the evaluators, however, a rating of 'very
good' would have been appropriate, since
those measures represent exemplary models
in terms of relevance:
The slightly downgraded rating for
'Institutional Development of the Water
Sector, Yemen' in the relevant evaluation
was founded on the premise that irrigated
agriculture, as a sub-sector of relevance
to water-resource problems, remained
unaccounted for in the priority area strat-
egy paper on the grounds of develop-
ment-policy directives. However, German
DC activities in this sub-sector would not
necessarily be expedient, because other
donors already cover this theme. Conse-
quently, that aspect should not have a
negative impact on the relevance rating.
With regard to the priorities of the lead
executing agency and implementing part-
ner and to conceptual embedment in the
country, as well as the sectoral and multi-
sectoral strategies of German DC, the
programme is highly relevant. It pursues a
successful approach to capacity devel-
opment and draws support from the suc-
cessful endeavours of women's groups. A
poverty study was conducted in advance
of programme building.
The 'Water Sector Reform Programme,
Kenya' is not only in accordance with Ke-
nyan water-sector policies and strategies,
but also with international themes and
standards. The programme enjoys high
priority at the lead executing agency, the
implementing organisations and the target
groups. It is suitable for solving central
development issues, contributes directly
toward improving the living conditions of
the poor population and is therefore in line
with the basic development-policy orienta-
tion of the GTZ. Its special focus on
gender equality is reflected by the water
kiosk approach. The entire reform pro-
gramme, with its water kiosk concept,
EcoSan concept and envisaged pro-poor
water rates structure, is geared to improv-
ing water services for the urban poor and
therefore has a strong relevance to pover-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 45
ty, even though no separate poverty anal-
ysis was conducted.
The 'Water Sector Reform Programme,
Zambia' is suitable for solving the coun-
try‟s central development issues because
it is aimed at improving the supply of wa-
ter and sanitation services, IWRM and
poverty reduction. It is in line not only with
national sector policies and strategies, but
also with international themes and stan-
dards as well as with the basic develop-
ment-policy orientation of the GTZ. The
programme's poverty relevance is illumi-
nated by its pro-poor orientation in peri-
urban rural areas. This includes the de-
velopment of pro-poor tariffs. The concept
is gender-responsive, and the responsible
institutions and target groups attach high
priority to the programme, as reflected by
the increased budget from the govern-
ment and by an increasing willingness to
pay for water and sanitation.
The 'Integrated Water Management Pro-
gramme, Algeria' stands as an example of a
programme with a poor relevance rating. As
analysed by the evaluation team, the measure
is in line with the sector policy and strategies of
the partner country, and it supports the impor-
tant process of reform in the Algerian water
sector, but the short-term supply of water – as
opposed to the sustainable management of
water resources – is seen as its most important
function. Accordingly, there is a conflict of ob-
jectives between the rather short-term orienta-
tion of the Algerian policy and the requirements
of sustainability pursuant to the IWRM ap-
proach in accordance with the BMZ sector
strategy. Consequently, the programme is not
in line with the basic development-policy orien-
tation of the client. Capacity development is
emphasised as a solution to the competency
deficits, although the core problem is a lack of
institutional mainstreaming with regard to sus-
tainability. Though they constitute part of the
target group, future generations in Algeria will
not benefit from the programme. Neither a
poverty analysis nor a gender analysis was
performed. While the partner does have a posi-
tive opinion of the programme's concept, the
majority of the key questions for assessing the
programme's 'relevance' could not be an-
swered positively.
In many evaluation reports, a general weak
point with regard to the relevance of projects
and programmes is stated as the fact that only
a small share of the interventions are situated
at the target group level and geared to directly
improving the living conditions of the poor
population (see also Section 2.2.2). In fact,
most projects lack the appropriate planning-
phase analyses of target groups, poverty and
the gender situation.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 46
3.1.2 Weighting of relevance
No clear-cut patterns emerge with regard to
the weighting of relevance in the individual
evaluation reports. The relevance criterion had
an average weighting factor of 2.3 (interim
evaluations 2.6, final evaluations 2.0, and ex-
post evaluations 2.0). In five of the evaluations,
the factor was upgraded (weighting factor 3;
see also Table 4) always in connection with
interim evaluations. This leads to the conclu-
sion that, either at or after the end of the
measures, more importance was attached to
criteria other than relevance, whereas the stra-
tegic orientation of the projects and pro-
grammes played a decisive role in the course
of implementation.
Main reasons for enhanced weighting of rele-
vance were, for example:
the fact that it is one of GTZ's largest wa-
ter programmes, therefore enjoys high
priority, and is characterised by good con-
ceptual embedment in the entire sector
(Yemen);
the objective of improving the supply of
drinking water to the urban population
while protecting the water resources
(Kenya);
the fact that the development measure
does not focus on solving problems of a
local nature, with limited content, instead
addressing all levels and pursuing a sys-
tem-changing capacity development ob-
jective (Peru);
the objective of improving the supply of
drinking water to the urban population, in
combination with the fact that the water
sector enjoys priority in the Zambian na-
tional development plan, and that Ger-
many is the lead donor in the water sector
(Zambia).
In the case of the 'Integrated Water Man-
agement Programme, Algeria', the higher
weighting of relevance was not entirely under-
standable, and no corresponding substantia-
tion was evident.
In the above context, three of the pro-
jects/programmes were conspicuous in that
their relevance was given higher weightings,
but their relevance rating was actually too low.
Apparently, the evaluators' analysis is marred
by major uncertainty in the use of the available
tools.
3.2 Effectiveness
3.2.1 Effectiveness rating
Effectiveness is seen as the extent to which
the direct results (objectives) of the develop-
ment measure are being achieved (comparison
of actual situation with targets).
'Are we achieving the objectives of the devel-
opment measure?'
On the whole, the evaluated projects and pro-
grammes achieved an average effectiveness
rating of 2.5 ('good'). In more than half of the
projects and programmes good achievement of
objectives was established (9). Five others
performed satisfactorily, and only one project
(Niger) was rated 'unsatisfactory' in terms of
achieving objectives, based on comparison of
the actual situation with targets (see also Table
5 and Figure 6).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 47
The average effectiveness rating varies ac-
cording to the timing of the evaluation: In the
interim and final evaluations, the average rat-
ing was 2.3, dropping to 3.0 in the ex-post
evaluations. Apparently, the degree to which
the objectives are achieved is interpreted
somewhat more critically with increasing 'dis-
tance' to the actual implementation.
Table 5: Effectiveness ratings of evaluated projects/programmes
Country Project/Programme Type of
Evaluation Rating Weighting
Africa su-
praregional NWI
Interim
evaluations
2 2
Algeria PDEA 3 3
Yemen WSP 2 2
Jordan WMIA 2 2
Kenya WSRP-KEN 2 2
Peru PROAGUA 2 3
Zambia WSRP-ZMB 2 2
Viet Nam WDPC 3 2
Ghana EVORAP Final
evaluations
2 2
Kosovo WSWDP 2 2
Mexico GICA 3 3
Ethiopia IWSTR
Ex-post
evaluations
3 2
India IGWDP 2 2
Niger PHV/MI 4 2
Turkey QADI 3 2
Overall (average) 2.5 2.2
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 48
Figure 6: Distribution of effectiveness ratings
Success factors for improving effectiveness,
i.e. achievement of objectives:
Development measures were given a 'positive'
rating if
the project/programme partners and GTZ
programme management all share a
common understanding of the objectives
and cooperation;
the target group understands and appre-
ciates the benefits of the measure and
exercises its rights and obligations;
the national authorities accept revised
policies and regulations;
guidelines are generated and best prac-
tices disseminated and recognised by the
partner structure;
'meso-level environment management' is
formally mainstreamed in the organisa-
tional structure and implemented at the
micro level in the form of such concrete
measures as hygiene education, informa-
tion campaigns, customer surveys and/or
focus-group discussions;
capacity development reflects the quality
and coherence of the training and capac-
ity-building package , and systematic as-
certainment of needs enables better coor-
dination of supply and demand;
TC and FC are optimally intermeshed in
cooperative projects and programmes, i.e.
synergy effects are generated and the re-
sults of TC measures can be multiplied by
FC investments, and vice-versa.
The effectiveness of the 'Water Sector Re-
form Programme, Kenya' was rated 'good'. A
comparison of the actually achieved indicators
with the targets shows that all four overall-
objective indicators were partially achieved and
that full achievement within the programme
term (2013) is probable, since the overall de-
velopment is demonstrably positive. At the time
of evaluation, 6 of 20 component indicators
had already been fulfilled. Since the pro-
gramme began, sector coordination by the
ministry (component 1), regulation by the regu-
latory authority and pro-poor financing by the
trust fund (component 2) improved. Commer-
cialisation of WUs is being broadly imple-
mented (component 3). Strengthening of per-
formance capabilities at the WRM authority is
reflected in an improved contribution to sector
regulation and management (component 4).
0%
60%
33%
7%
very good
good
satisfactory
not satisfactory
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 49
Initial EcoSan pilot measures were introduced
(component 5).
As explained in the preceding Section 2.2.1,
the reviewed objectives systems clearly at-
tached priority to the results constellation
'drinking water supply – standards of supply –
operational efficiency'. All in all, approximately
one half of all indicator formulations relate to
that results constellation. In 'Rural Water
Supply in Maradi, Niger', for example, 6 of
the 7 formulated objectives-level indicators
relate to supply standards (supply coverage
rate, minimum supply volume, water quality)
and operation (maintenance/repair, water-well
hygiene, prevention of overuse).
The results constellation 'sector policy - regula-
tion - governance' is found in the indicator sys-
tems of only two projects and programmes
(Africa supraregional and Algeria). Conse-
quently, the 'recorded' direct results of the
evaluated projects and programmes were pri-
marily achieved as technical improvements
and improvements in the standards of supply,
and less so in terms of institutional and legal
change processes and the creation of sustain-
able institutional capacities. In addition, it
proves enormously difficult to document indica-
tors which were formulated first and foremost
for the policy-making area, if these are impre-
cisely worded and excessively ambitious, and
have no follow-up documentation available. In
the interim evaluation report on the 'Integrated
Water Management Programme, Algeria',
for example, no plausible contributions regard-
ing the indicators for the areas of sector devel-
opment planning, investment planning, regula-
tion and water resources policy planning were
documented.
The recording of qualitative aspects (changes
in behaviour, quality, satisfaction, etc.) in the
achievement of objectives is still a weak point.
Relevant results are usually 'assumed' or 'an-
ticipated' instead of substantiated. This point
has been criticised by numerous evaluators
conducting individual evaluations, especially
because the projects and programmes are
widely assumed to have more results than are
actually being recorded. Such assumptions
derive mainly from random interviews with
target-group representatives, which are held in
connection with all such evaluations and are
accordingly interpreted by the respective
evaluators.
Since only a very small fraction of the indica-
tors formulated for the evaluated projects and
programmes relate to the recording of con-
crete, results-based outcomes (e.g. supply
situation, poverty reduction, health), it is rec-
ommended that, in future, more indicators be
formulated and more appropriately designed
studies and documentations be prescribed,
e.g. in reporting and in advance of scheduled
evaluations. It is also important in this connec-
tion that the appropriate indicators be docu-
mented with concrete facts, figures and data. A
number of best practices cited in the current
set of evaluated projects and programmes are
available to that end.
Factors contributing to failure reduce the effec-
tiveness of development measures: In the
evaluations under review, the achievement of
objectives received a 'negative' rating if, for
example:
the structural context of the sector im-
pedes sustainable capacity development;
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 50
a number of different but overlapping as-
sistance programmes undermine the
stipulations (conditionalities) of the donor
countries;
delayed outflow of FC funds in connection
with cooperative projects and pro-
grammes makes the shared TC/FC indi-
cators unachievable;
adjustment and further development of
the technical conceptual design are ne-
glected during the implementation phase;
further education/training and subsequent
follow-up assistance for water user asso-
ciations by the responsible authorities is
neglected;
rapid population growth is not accounted
for in the design of the measure or in its
formulated objectives;
conflicts within the GTZ team cause de-
lays in implementation and substantially
disrupt the working relationships;
internal conflicts have a negative influ-
ence on the partner, and partner(s) have
a negative influence on the measure.
Some good examples of documented objectives achievement and results:
In the 'Eastern and Volta Region Assistance Programme, Ghana', the establishment and intro-
duction of an integrated database (InfoSys) for continuous data acquisition and monitoring in
the water sector was facilitated and a uniform monitoring system established (MOM – 'Mainte-
nance – Operation - Monitoring'). These instruments have since proven comparatively effective
for use in monitoring the sustainable operation of water supply systems and have been nomi-
nated as best practice for countrywide introduction. Additionally, the programme regularly re-
ceived statistics on the occurrence of water-induced diseases such as typhus, bilharziasis,
dracunculosis/Guinea worm, malaria and diarrhoea from hospitals in the project region. Those
data were then analysed to detect effects on the health & hygiene situation.
In the 'Drinking Water and Sanitary Programme, Peru', a strategic planning and management
procedure (MPPI) was instituted at the participating WUs. A complementary monitoring system
(SIME) was developed and a knowledge-management procedure established within the com-
plex institutional structure. The corresponding instruments have enabled routine data availability
and dissemination. They have since been implemented as best practices in neighbouring coun-
tries, too. Through the GTZ knowledge-management structures, the instruments have been
made available to a broad audience.
The 'WMIA Jordan' project, for example, was
geared to reducing groundwater extraction
(indicator: component objective). However, no
fresh data of relevance are available. While
some information is available concerning re-
duced abstraction rates from the aquifer in
question, the extent to which this is attributable
to the project remains unclear (measurability
and attribution problem).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 51
The 'Rural Water Supply in Maradi, Niger'
project was an example of failure. The effec-
tiveness of this development measure was
rated as 'unsatisfactory'. Only four of its eight
indicators were fully or partially met. The main
reason for the negative rating was the lack of
durable self-administration structures and the
resultant poor operating standards. At the end
of the project, water committees were manag-
ing the operation of about 85% of all the con-
structed wells (indicator: objectives level). At
the time of ex-post evaluation in 2008, 90% of
the water user committees had ceased their
activities. In 2003, women accounted for 50%
of all water-well committee members (indicator:
objectives level). By 2008, that was true of only
10% of the still-active committees. Since most
(with a few exceptions) of the water user com-
mittees who had been taught how to keep the
well environs clean and the extracted water
hygienic, have discontinued their activities, the
well systems are no longer being kept clean or
free of cattle (indicator: objectives level). In this
project, important coordination processes be-
tween TC and FC have failed to materialise, as
has conceptual adaptation regarding a stricter
focus on capacity development among the
target group and intermediary institutions.
For the 'Integrated Water Management Pro-
gramme, Algeria', the effectiveness of the
development measure was given a 'satisfac-
tory' rating based on comparison of the actual
situation with targets. However, on the as-
sumption that existing cooperation problems
will be eliminated, only 3 of 12 indicators were
identified in the evaluation report as being
'potentially achievable'. At the time of the
evaluation, i.e. six years into the measure, not
a single indicator could be classified as
'achieved'. Indeed, the achievability of 8 of the
12 indicators was assessed as 'questionable'
or 'hardly probable'. Consequently, the corre-
sponding rating is basically inconsistent, and
the programme should be regarded as an ex-
ample of failure.
The level of funds provided by German TC
appears to have very little influence on the
effectiveness of the projects and programmes.
No corresponding statistical connection can be
ascertained on a rank correlation basis. There
does, however, appear to be a closer link be-
tween the duration of implementation and the
achievement of objectives. Simultaneously, the
effectiveness of projects and programmes
appears to exert substantial influence on both
the impact and the sustainability of the meas-
ures (see also Table 6).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 52
Table 6: Effectiveness correlations
Characteristic
Correlation
value/
Rank correlation
Funding level (overall) – effectiveness - 0.03
Funding level (per month of implementation) –
effectiveness + 0.06
Duration of implementation – effectiveness - 0.38
Effectiveness – impact + 0.53
Effectiveness - sustainability + 0.67
3.2.2 Weighting of effectiveness
Similarly to the weighting of relevance, there
are no clear patterns to be seen with regard to
the weighting of effectiveness in the individual
evaluation reports. This criterion was weighted
with an average factor of 2.2 (interim evalua-
tions 2.3, final evaluations 2.3, ex-post evalua-
tions 2.0), which was upgraded in three
evaluations (weighting factor 3; see also Table
5). This was the case in two interim evalua-
tions and one final evaluation.
In the opinion of the evaluators, it is surprising
that the actual achievement of objectives on
the basis of the formulated objectives systems
did not enjoy greater importance in the as-
sessment of projects and programmes. Basi-
cally, one would expect final and ex-post
evaluations to rate the achievement of objec-
tives, along with sustainability, as the most
crucial characteristic. At the same time, closer
analysis of the observed objectives achieve-
ment trends enables quicker identification of
potentially required conceptual adaptations,
especially in interim evaluations key question:
'Why will the objectives probably not be
achieved?').
Principal reasons for stronger weighting of
effectiveness in the present evaluations in-
cluded, for example:
implementation of the measures is objec-
tives-oriented and in shared understand-
ing with the partner (Algeria);
effectiveness is a central criterion for final
evaluation (Mexico);
it is of decisive importance that the essen-
tial results of the measure be evident in
the interim evaluation after completion of
two phases, i.e. after nearly six years, so
that they can be consolidated during the
last phase of the programme (Mexico).
3.3 Overarching development re-
sults (impact)
3.3.1 Assessment of overarching
development results (impact)
Overarching development results (impact)
relate to the 'extent to which the project or
programme helps to achieve the planned over-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 53
arching objectives, and other indirect results
occur‟.
Are we contributing to the achievement of
overarching development objectives and re-
sults?
The evaluated projects and programmes
achieve an average impact rating of 2.7 ('satis-
factory'). While two thirds of the projects and
programmes (10) were found to have made
satisfactory contributions to the achievement of
overarching development results, one pro-
gramme (Yemen) received a 'very good' rating,
and two projects/programmes were rated
'good'. Only a single project (Turkey) received
an 'unsatisfactory' rating for its contribution to
the achievement of indirect results (see also
Table 7 and Figure 7).
The average impact rating again varies accord-
ing to the point in time of the evaluations: The
average ratings in the interim and final evalua-
tions were situated at 2.5 and 2.7, respectively,
falling to a poorer 3.3 on average in the ex-
post evaluations. As with the effectiveness
rating, it would appear that here, too, the
achievement of indirect results is interpreted all
the more critically with the passing of time
since implementation. Particularly with regard
to ex-post evaluations, this is understandable,
because once several years have passed
since a measure's completion, an appropriate
review can and should be performed.
It becomes apparent in the analysis of evalua-
tion reports, especially of interim and final
evaluations, that it is hardly possible to record
the results of the projects/programmes down-
stream of their 'attribution gap' and that little
attention is devoted to that area. With regard to
impact, the evaluation reports only offer either
conjecture on results constellations or as-
sessments of conditions that have not yet oc-
curred. Often, it is argued that the impact will
only emerge after a considerable length of
time, so no further investigations are possible.
In the opinion of the evaluators, however, the
projects/programmes could very well attempt
to substantiate indirect results when, for exam-
ple, concrete measures are implemented at the
intermediary and target-group levels. A typical
example of such a case would be the system-
atic collection and documentation of data on
water quality and the occurrence of water-
induced diseases in the course of implementa-
tion (Ghana). This could yield information on
results trends. Also, random surveys can be
conducted among the target groups at the
beginning and during implementation to shed
light on qualitative improvements in vital natu-
7%20%
67%
7%
very good
good
satisfactory
not satisfactory
Figure 7: Distribution of overarching development results rating
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 54
ral resources and altered behaviour (Yemen).
Such surveys can likewise provide important
insights into such matters as how changes in
the range of services offered by utility compa-
nies are perceived (Peru). In an ideal case, this
could be founded on the establishment of an
appropriate baseline.
Table 7: Rating of overarching development results of the evaluated projects/programmes
Country Project/Programme Type of
evaluation Rating Weighting
Africa su-
praregional NWI
Interim
evaluations
2 2
Algeria PDEA 3 2
Yemen WSP 1 2
Jordan WMIA 3 2
Kenya WSRP-KEN 2 2
Peru PROAGUA 3 2
Zambia WSRP-ZMB 3 2
Viet Nam WDPC 3 1
Ghana EVORAP Final
evaluations
2 3
Kosovo WSWDP 3 2
Mexico GICA 3 2
Ethiopia IWSTR
Ex-post
Evaluations
3 3
India IGWDP 3 2
Niger PHV/MI 3 2
Turkey QADI 4 2
Overall (average) 2.7 2.1
Success factors contributing to enhancement
of the overarching development results can be
summarised as follows on the basis of the
results of the reviewed evaluations: Develop-
ment measures are given a 'positive' rating if
the measure's contribution to the over-
arching result can be plausibly substanti-
ated, and the formulation of the overarch-
ing result leaves no room for interpreta-
tion;
micro-level results extending beyond the
target group occur, e.g. the founding of a
successful NGO;
structural impacts, e.g. the decentralisa-
tion of WS&S and the advancement of
sector reform beyond the level of objec-
tives achievement, are supported;
the overarching results are documented
not only by qualitative statements, but
also by quantitative statements founded
on a baseline study;
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 55
a contribution toward achievement of the
MDGs is made, and target group surveys
confirm a qualitative improvement of the
situation;
a role-model function emerges, and the
imparted approaches and concepts be-
come common property;
the negotiating positions in political dia-
logue and the ability to solve problems on
the part of both the target group and the
stakeholders are strengthened.
The impact of 'Institutional Development of
the Water Sector, Yemen' was rated 'very
good'. The objective of this programme is to
improve the health situation of the poor popula-
tion while securing an important part of the
target group's natural resource base. No quan-
titative statements can be made in that re-
spect, because no corresponding baseline
study is available. However, real qualitative
improvements are identified by way of informa-
tive surveys. For example, the fact that female
village multipliers have joined together to es-
tablish an NGO was regarded as positive. The
programme's structural impact above and be-
yond the objectives level, though, was decisive
for the 'very good' rating. Throughout the coun-
try, there has been a paradigm shift in the wa-
ter sector. The approaches and concepts
communicated by the reform policy have be-
come common property, with consequential
positive effects in other areas, too (e.g. the
decentralisation of administrative structures).
In that respect, the contribution made by the
development measure was definitely demon-
strable.
Factors determining failure with potential influ-
ence on impact can be summarised as follows:
Development measures are rated 'negative' if
the aggregated results are rendered un-
realistic by the programme (excessively
high expectations, very long results
chains) and can therefore only be as-
sumed ('perceived');
no verifiable contribution toward poverty
reduction is made;
there are no positive effects on other parts
of the broader sectoral and regional con-
text;
there are no broad-based effects from, for
example, model character, and the gen-
erated work results are only utilised by
people who were involved in their crea-
tion.
The impact of the 'Upgrading of Municipal
Services, Turkey' project, for example, was
rated as 'unsatisfactory'. Overarching results
had been expected with regard to better con-
servation of natural resources, safe access to
clean drinking water and greater participation.
The project failed to develop broad-based ef-
fectiveness, because the generated working
aids were only utilised by a few individuals
(presumably those who had been involved in
their creation). The low number of training
measures implemented in the municipalities
was cited as another reason for the poor rat-
ing. In addition to the existing financing con-
straints, the development of municipal infra-
structure is also perceived as constrained (an
argument that would also have been of rele-
vance for the effectiveness rating). The project
is only known to a small minority of municipali-
ties, and even in the pilot communities it has
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 56
not been possible to influence the quality of
municipal services. Note: This project is the
one with the shortest term. The term was
shortened after the project began, and the
project came to an abrupt end.
3.3.2 Weighting of overarching de-
velopment results (impact)
The impact of the projects and programmes
was weighted with an average factor of 2.1.
Again, there are minor differences between the
weighting assigned in the performed interim
evaluations (1.9) and that found in the final and
ex-post evaluations (2.3 in each case). In one
final and ex-post evaluation each, the impact
was weighted with a higher factor of 3, and in
one interim evaluation it was downgraded to
the factor 1 (see also Table 7).
Overall, then, major importance was ascribed
to the indirect results of projects and pro-
grammes. While these results constellations
frequently had not yet become identifiable at
the time of the interim and ex-post evaluations
(especially in the case of long results chains),
the evaluators believe that more importance
should have been attached to this criterion in
the ex-post evaluations, because indirect re-
sults do make real contributions to qualitative
performance assessment.
One explorable hypothesis in favour of not
attaching more importance to impact in evalua-
tions would be that, with regard to indirect re-
sults, the results chains of the TC projects and
programmes often have very long, abstract
formulations, and that many results are pro-
jected into a very highly aggregated area. Such
links are difficult to record, and robust data are
rarely available. Moreover, independent inves-
tigations of these are very complicated.
Principal reasons for altering the weighting of
impact in the evaluations:
upgrading: This criterion is of decisive
importance for the project or programme,
particularly with regard to institution de-
velopment and to projects involving ca-
pacity development (Ethiopia);
upgrading: Since the indirect results have
to do with improving the people's health
and hygiene situation and with generally
improving their quality of life while con-
tributing to the provision of basic services,
there is a chance that an especially large
number of people can benefit from the
positive results. An improved quality of life
for target groups should find expression in
this category (Ghana);
downgrading: Due to delays in the imple-
mentation of the first phase, most of which
could not be influenced, the TC contribu-
tion had not yet had a chance to provide
the full scope of planned outputs or,
hence, to achieve the anticipated results
(Viet Nam).
3.4 Efficiency
3.4.1 Efficiency rating
Efficiency is a measure of the degree to which
the resources invested in a development
measure are appropriate compared to the out-
puts and results achieved.
'Are the objectives being achieved cost-
effectively?'
The evaluated projects and programmes
achieved an average efficiency rating of 2.8
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 57
('satisfactory'). Six of the projects/programmes
were rated 'good' or 'satisfactory', while three
others (Ethiopia, Algeria and Niger) received
an efficiency rating of 'unsatisfactory' (see also
Table 8 and Figure 8). The projects and pro-
grammes examined in ex-post evaluation tend
to have been rated somewhat lower with re-
gard to the cost efficiency of their outputs.
Here, the average rating was 3.5, that is, an
entire point lower than the average rating (2.5)
of the projects and programmes in the interim
evaluations. The average efficiency of the pro-
jects and programmes in the final evaluations
was 2.7.
The relatively large discrepancy between the
efficiency ratings found in the ex-post evalua-
tions and those cited in the interim and final
evaluations is certainly also due to the fact
that, when evaluation is being performed dur-
ing or even at the end of an implementation
period, relatively little information is available
with regard to the specific level of funding (es-
pecially for investments: funds employed per
user) and about the actual achievement of
objectives. At the same time, for the purposes
of interim evaluation, less thought tends to be
given to alternative forms of service delivery
than to optimising the existing conceptual de-
sign.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 58
Table 8: Efficiency ratings of the evaluated projects/programmes
Country Project/Programme Type of
Evaluation Rating Weighting
Africa su-
praregional NWI
Interim
Evaluations
3 2
Algeria PDEA 4 3
Yemen WSP 2 2
Jordan WMIA 2 2
Kenya WSRP-KEN 2 2
Peru PROAGUA 2 2
Zambia WSRP-ZMB 2 2
Viet Nam WDPC 3 2
Ghana EVORAP Final
Evaluations
3 3
Kosovo WSWDP 2 2
Mexico GICA 3 2
Ethiopia IWSTR
Ex-post
Evaluations
4 1
India IGWDP 3 1
Niger PHV/MI 4 2
Turkey QADI 3 2
Overall (average) 2,8 2,0
Figure 8: Distribution of efficiency ratings
There appears to be no link between the level
of funding (total funds employed per month of
implementation) and the respective efficiency
rating. In the IWSTR Ethiopia project, for ex-
ample, the average monthly level of funding
was EUR 30,526, while EUR 97,980 was spent
0%40%
40%
20%
very good
good
satisfactory
not satisfactory
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 59
per month in PDEA Algeria and EUR 187,850
in PHV/MI Niger. The efficiency of those pro-
jects/programmes, however, was rated 'unsat-
isfactory'. On the other hand, both the WSP
Yemen programme and the WSRP Kenya
programme received 'good' efficiency ratings
despite their high levels of funding per month
of implementation (EUR 118,421 and EUR
113,008, respectively; see also Table 9).
Table 9: Comparison of term, level of funding and efficiency ratings
Country Project/Programme
Term
[months]
Level of funding
Rating Total
per
month
Yemen WSP 114 €13.5 M €118,421
'good'
Jordan WMIA 111 €2.5 M €22,523
Kenya WSRP-KEN 123 €13.9 M €113,008
Kosovo WSWDP 108 €7.7 M €71,296
Peru PROAGUA 156 €8.2 M €52,564
Zambia WSRP-ZMB 108 €8.1 M €75,000
Africa su-
praregional NWI
120 €10.0 M €83,333
'satisfactory'
Ghana EVORAP 126 €6.2 M €49,206
India IGWDP 183 €12.8 M €69,945
Mexico GICA 70 €2.4 M €34,286
Turkey QADI 50 €1.7 M €34,000
Viet Nam WDPC 107 €8.8 M €82,243
Ethiopia IWSTR 95 €2.9 M €30,626
'unsatisfactory' Algeria PDEA 99 €9.7 M €97,980
Niger PHV/MI 107 €20.1 M €187,850
The link between project term and efficiency
does not appear to be very distinct. Statisti-
cally, relevance also appears to have no plau-
sibly verifiable effect on efficiency. Viewed
against the backdrop of the factors of success
and failure discussed below as applied to effi-
ciency, however, some degree of correlation
should be presumed. Thus, not only the devel-
opment-policy alignment with sector and coun-
try strategies is decisive, but also the coordina-
tion actually performed with other do-
nors/actors (e.g. on the basis of a cooperation
agreement) and adjustment of the envisaged
interventions (e.g. on the basis of shared plan-
ning).
In connection with the reviewed evaluations, a
number of success factors were identified with
the capacity to enhance efficiency. Accord-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 60
ingly, a development measure was given a
'positive' rating if
coordination takes place between donors,
and the fields of endeavour are mutually
complementary;
coordination takes place with other Ger-
man DC measures, and potentials for fur-
ther cooperation are utilised;
synergies between TC and FC are sys-
tematically exploited in a cooperative pro-
ject/programme;
the personnel concept is convincing, i.e.
internal and external, short-term and long-
term advisory services (including those
rendered by national evaluators) are har-
monised, and synergies are fostered by
well-developed knowledge management;
necessary adjustments in the timing of
phases are coordinated with BMZ in due
time;
synergies result from interventions at dif-
ferent levels.
The efficiency of the 'Water Sector Reform
Programme, Zambia' was rated 'good'. The
programme is characterised by a high level of
donor coordination, which involves not only the
German donors but also the international do-
nor community and NGOs in the sector reform
programme. Promotion of sector development
was pursued by splitting the responsibilities
between the donors and identifying special
fields. German and Danish DC, for example,
assumed leadership in the field of IWRM. The
programme supports such programme-based
approaches as SWAp and joint financing
measures. Its high level of cost efficiency is
particularly apparent in the financing of water
kiosks, which make a major contribution to-
ward the achievement of objectives (invest-
ments amounting to EUR 8 per person). Only
the fact that the outputs geared to drafting the
relevant legislation for the water sector had not
brought the desired success gives rise to a
minor downward adjustment.
Nevertheless, some of the evaluations recog-
nised a few factors determining failure that
reduced efficiency. Accordingly, a development
measure was given a 'negative' rating if
its objectives, outputs and activities are
neither coordinated with those of other
projects and programmes nor designed
for task-sharing (joined-up development
cooperation, SWAp and basket funding);
consensus-building and coordination of
resource inputs between the various
German DC organisations is either inade-
quate or unmanaged;
the partner shows too little willingness to
implement;
there is too little exchange between the
programme management and the partner;
individual/personal interests impede co-
operation between donors;
the advisory approach is faulty and not
results-based;
the project‟s/programme‟s own resources
(personnel, material and equipment, etc.)
are used inefficiently, and antici-
pated/unforeseen additional costs accrue;
the complexity of the personnel concept
has a negative influence on the cost/effect
ratio, e.g. too many experts not only cost
more, but even have counterproductive
effects;
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 61
the continuity of the measure is inter-
rupted;
investments are lacking, e.g. when the
allocation of FC funds is delayed in a co-
operative project or programme.
The efficiency of the 'Rural Water Supply in
Maradi, Niger' project was rated 'unsatisfac-
tory'. The average cost of investment per
newly constructed or rehabilitated water well
(approx. EUR 16,000) increased two-fold to
roughly EUR 33,000 per well, if the consulting
and advisory services are also counted. Con-
sequently, the outlays for advisory services are
exaggerated in comparison with the achieved
results. With regard to the requisite advisory
and awareness-raising measures for the civil-
ian population, it would have been expedient to
cooperate with NGOs, but that was not the
case. Differences of opinion between the FC
and TC experts gave rise to problems affecting
the synergetic interaction and coordination of
the project structure. Experience drawn from
the implementation phase was not utilised for
refining the approaches and technical models.
In addition, the operation of parallel project
infrastructures for TC and FC caused addi-
tional costs.
The 'Improvement of Water Supply in the
Tigray Region, Ethiopia' project was also
given an 'unsatisfactory' efficiency rating. The
project began with a study of the water sector
in Tigray. That study was too general in nature
and of average quality. Manuals developed on
the basis of that study proved to be of limited
use and rather impractical. Overall, the first
implementation phase was deemed inefficient,
partially because the implementation partner
had difficulty fulfilling its obligations. At the
same time, the commissioned consulting firm
had problems delivering its services. Due to
domestic-policy difficulties, implementation
was interrupted for approximately three years
between the two phases. Much data and in-
formation was lost during that period. A requi-
site tariff study was implemented much too
late, i.e. in April 2004, and neither cooperation
with similar projects in the sector nor an ex-
change of knowledge took place.
Nearly all evaluation reports speak of a definite
and general need for improvement when it
comes to the actual mainstreaming of meas-
ures in the respective sector interventions. In
many places, it was noted that the DC inter-
ventions of the various actors were not opti-
mally intermeshed despite established coordi-
nation mechanisms. This was attributable, on
the one hand, to delimitation problems be-
tween the DC actors' respective interventions
and on the other, to the fact that the coordina-
tion mechanisms are often still of an essentially
informal nature, i.e. not anchored in concrete
agreements. Hence, the coordination process
frequently never proceeds beyond an ex-
change of information.
Regarding modes of delivery, it must be noted
that very few of the evaluation reports look into
the concept of service delivery. This is surpris-
ing in that, in 13 of 15 projects and pro-
grammes, not only GTZ personnel but also
consulting firms and NGOs were commis-
sioned to provide services. Consequently, it
would hardly be possible to perform a consis-
tent analysis in that respect. The following two
aspects regarding efficiency were mentioned in
a small number of evaluations:
the importance of qualified national
evaluators and/or consulting firms and
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 62
NGOs for capacity building and sector-
specific advisory services and training as
a potentially more efficient alternative to
the deployment of seconded experts and
national staff (Niger, Peru);
the design of external expert assignments
which was at times still suboptimal due to
substantial commitment of funds, prob-
lems with integration into TC structures,
management and flow of information, in
addition to qualifications sometimes re-
garded as inadequate, primarily with re-
gard to language proficiency (Algeria,
Yemen).
3.4.2 Weighting of efficiency
On the whole, the efficiency of the evaluated
projects and programmes was the criterion
given the lowest weighting, i.e. 2.0 on average.
There were conspicuous differences between
interim evaluations (average weighting 2.1),
final evaluations (2.3) and ex-post evaluations
(1.5). In two of the four ex-post evaluations, the
cost efficiency of objectives achievement was
classed as less important (weighting factor 1),
while that criterion was weighted more strongly
in one interim evaluation and one final evalua-
tion (see also Table 8).
Main reasons for changes effected in the
weighting of efficiency:
upgrading: The criterion is amenable to
direct influence by GTZ and the pro-
gramme's management (Algeria);
upgrading: The establishment of basic
principles for effective monitoring sys-
tems, the organisation and training of self-
help organisations and the significance of
institutional interaction can be highlighted
with the help of this criterion. In the con-
ceptual sense, these are crucial elements
of the measure (Ghana);
downgrading: If impact and sustainability
are both good, it is of lesser importance
whether or not the activities were per-
formed inefficiently. Even when activities
are performed with extreme efficiency, the
result cannot be satisfactory in combina-
tion with poor sustainability (Ethiopia);
downgrading: There is little room for ex-
erting influence on the input-output ratios,
total expenditures and the nature of out-
puts from intermediary organisations (In-
dia).
In the opinion of the evaluators, the arguments
cited in favour of giving more weight to effi-
ciency as a criterion are inconsistent. It is deci-
sive that the employed resources, the modes
of delivery and the corresponding outputs and
results be documented. Wherever relatively
high effort is required for establishing capaci-
ties or for catalysing processes of change and
bringing about improvements in quality (e.g.
water supply), the question arises as to the
cost efficiency of the employed instruments
and of potentially available, more efficient al-
ternatives – not, however, as to their strategic
importance or the extent to which they can be
influenced.
In the view of the evaluators, the reasoning
behind the effected downgrading of efficiency
must be seen in connection with unclear or
ineffective control mechanisms.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 63
3.5 Sustainability
3.5.1 Sustainability rating
Sustainability is a measure of the probability
that the positive results of the development
measure will continue beyond the end of assis-
tance.
'Are the positive results durable?'
Most of the evaluated development measures
were given positive ratings, i.e. deemed suc-
cessful, in terms of sustainability. The average
rating here was 2.9 and sustainability therefore
the DAC criterion with the poorest average
rating. Nine of the projects and programmes
were classed as 'satisfactory' and four as
'good'. Two of the projects/programmes
(Ethiopia and Niger) received an unsatisfactory
sustainability rating (see also Table 10 and
Figure 9).
Experience shows that the assessment of sus-
tainability tends to fall in the course of a project
cycle. The reviewed evaluation sequence was
also marked by a gradual deterioration of this
criterion's ratings: from 2.7 in the interim
evaluations to 3.0 in the final evaluations and
3.3 in the ex-post evaluations. Particularly at
the 'capacity development' results level, it often
takes until the end of implementation or even
longer for the structures to show whether they
will keep on developing and utilising the TC
outputs in the intended interest of achieving
objectives following completion of the project
or programme. At that point, sustainability as-
pects are no longer 'anticipated' but actually
recorded.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 64
Table 10: Sustainability ratings of the evaluated projects/programmes
Country Project/Programme Type of
evaluation Rating Weighting
Africa su-
praregional NWI
Interim
evaluations
2 2
Algeria PDEA 3 3
Yemen WSP 3 3
Jordan WMIA 2 2
Kenya WSRP-KEN 3 2
Peru PROAGUA 2 2
Zambia WSRP-ZMB 3 2
Viet Nam WDPC 3 1
Ghana EVORAP Final
evaluations
3 2
Kosovo WSWDP 3 3
Mexico GICA 3 3
Ethiopia IWSTR
Ex-post
evaluations
4 3
India IGWDP 2 3
Niger PHV/MI 4 2
Turkey QADI 3 2
Total (average) 2.9 2.3
Figure 9: Distribution of sustainability ratings
0%27%
60%
13%
very good
good
satisfactory
not sufficient
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 65
For the sustainability criterion, too, the per-
formed evaluations identified a number of suc-
cess factors with the capacity to enhance sus-
tainability. A development measure is consid-
ered sustainable if
sustainability is deemed important in the
course of implementation, and all (eco-
nomic, social, political and ecological) ob-
jectives dimensions are given timely, well-
balanced consideration at the design
stage;
participatory structures are established at
the target-group level (micro level) to en-
able participation in dialogue on the use of
water as a resource;
structural risks in further interventions are
identified;
the lead executing agency shows accep-
tance, and the basic instruments and
methods for the further development of
partner structures are created and inter-
nalised within those structures;
the aims and content of the existing plan-
ning remain intact despite possible
changes in responsibilities, objectives and
content;
sufficient personnel is trained in and for
the sector;
follow-up assistance is provided (espe-
cially for FC participation);
WS&S is recognised as a service and in-
tegrated into the day-to-day IWRM ap-
proach;
examples not only of 'best practice' but
also of 'lessons learned' are compiled and
disseminated.
The sustainability of the 'Nile Water Initiative,
Africa supraregional' was rated 'good'.
Through extensive training, the staff of the
water ministries in the member countries were
taught how to revise and adapt national water
policies as necessary. Water directives were
drawn up and 'best practice' examples com-
piled. The ecological sustainability of the
measure was ensured by incorporating the
recommendations on regulation into the na-
tional legislation.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 66
A good example of taking into account the objectives dimensions of sustainability:
In the 'Institutional Development of the Water Sector, Yemen' programme, the four objectives
dimensions (political, economic, social and ecological) are equitably represented. The economic
dimension is reflected in the commercialisation of water both as a prerequisite for economic de-
velopment and as a commodity. Through such aspects as pro-poor water policy and tariffs, con-
servation of resources for subsequent generations, and access to clean drinking water, the so-
cial dimension plays an important role for all concerned. The programme accounts for the eco-
logical target dimension in the areas of resource conservation, efficient use of water and curbing
the overuse of water resources. All these objectives dimensions are interlinked in the pro-
gramme's five components. The programme links together the various sub-sectors of the water
sector, promotes political dialogue and is therefore able to address and moderate rival objec-
tives dimensions. The programme's broad partner spectrum and its cooperation and alliances
with other donors and sectors promote support for sustainable development in the water sector
in all four objectives dimensions.
Factors determining failure with negative
effects on sustainability can also be formu-
lated for that criterion on the basis of indi-
vidual evaluations. Accordingly, a project
or programme is considered less sustain-
able if
the lead executing agency displays lit-
tle willingness to accept advice in con-
nection with higher-level political ob-
jectives;
implementation partners hesitate to
implement new laws and regulations,
e.g. by introducing break-even tariffs
and providing funds for the operation
and maintenance of WS&S;
inefficient overlapping of institutional
tasks is not accounted for;
the target group attaches no priority to
the measures, e.g. to the introduction
of a management system (software) as
opposed to the construction of infra-
structure (hardware) as a visible sign
of progress;
the conceptual approach lacks sys-
tematic capacity development compo-
nents, and too little attention is paid to
operating and self-administration struc-
tures;
retention of the value of investments is
systematically neglected;
heavy subsidisation of the water sector
has distorting effects;
ecological effects are not taken into
account.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 67
Statistical analysis of the relationship between
the duration of implementation and the sus-
tainability rating reveals strong interdepend-
ence – even more so than the dependence of
sustainability on the volume of employed funds
(see also Table 11). This result shows how
important the duration of advisory services
actually is, especially against the backdrop of
specific results levels in German TC (priority
area: capacity development, see also Section
2.2.2).
Table 11: Sustainability correlations
Characteristic
Correlation
value /
Rank correlation
Funding level (overall) – sustainability - 0.23
Funding level (per month of implementation) - sustainability + 0.04
Duration of implementation – sustainability - 0.74
3.5.2 Weighting of sustainability
Apart from relevance (see also Section 3.1.2),
the DAC criterion sustainability was the most
important rating element in the reviewed
evaluations. The average sustainability weight-
ing was 2.3, with the interim evaluations at-
taching less importance to the criterion (aver-
age weighting 2.1) and the final and ex-post
evaluations attaching more (2.7 and 2.5, re-
spectively). All in all, the weighting of sustain-
ability was increased in six evaluations and
decreased in one. In particular, two out of three
final evaluations and two out of four ex-post
evaluations upgraded the criterion (see also
Table 10.)
Main reasons for changes made in the weight-
ing of sustainability:
upgrading: Improvements achieved
through the work of the programme must
endure beyond the end of the programme
(Algeria);
upgrading: Decisive criterion with regard
to institutional strengthening and capacity
development projects (Ethiopia);
upgrading: Sustainability is the most im-
portant criterion in an ex-post evaluation
(India);
upgrading: The programme is able to
build on a number of ongoing previous
projects or programmes, and special im-
portance is attached to ecological sus-
tainability, particularly in water-deficient
countries (Yemen);
upgrading: Due to premature termination
of the project, special importance is at-
tached to the partner's problem-solving
capacities and to independent continua-
tion of the measure (Mexico).
The evaluators performing the cross-section
evaluation were unable to follow the reasons
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 68
for upgrading this criterion as put forward in the
final evaluation of the „Infrastructure Pro-
gramme Water Supply and Wastewater Dis-
posal, Kosovo'. It was argued there that some
actors, judging by the e-VAL results, harboured
doubts about the programme's sustainability.
This, then, appears to be an assessment – not
an argument for weighting a criterion.
The low weighting of sustainability in the
'Wastewater Disposal in Provincial Centres,
Viet Nam' programme was founded on the fact
that, due to delayed implementation during the
initial phase, the TC contribution had been
unable to render the planned outputs and,
hence, achieve the anticipated results. That
line of reasoning too is more of a justification
for inadequate achievement of objectives (ef-
fectiveness).
In line with the GTZ concept of sustainable
development, there can be scant justification
for downgrading the criterion sustainability,
particularly since the evaluated projects and
programmes are very complex, and their prior-
ity areas contain results constellations that
focus on establishing durable capacities (see
also Section 2.2.2). Consequently, the sustain-
ability criterion should, in the opinion of the
evaluators, be perceived as particularly impor-
tant in the course of implementation and drawn
upon as a crucial factor of performance as-
sessment when all interventions have ceased.
3.6 Synopsis
3.6.1 Weighting of DAC criteria
As explained above, different degrees of im-
portance are attached to the various DAC crite-
ria in the evaluations (in terms of upgrading or
downgrading from a standard weighting of 2.0).
The criteria 'relevance' and 'sustainability' were
considered to be especially important (average
weighting 2.3), as was 'effectiveness' (2.2)
(see Table 12).
In addition to absolute averages, the frequency
of upgrading and downgrading also illuminates
the importance of a given criterion. In this
sense, too, 'relevance' and 'sustainability' are
the most strongly emphasised criteria. The
actual achievement of objectives ('effective-
ness') is next in importance, while the contribu-
tion to overarching development results (i.e.
the results beyond the 'attribution gap') and the
cost efficiency of service delivery are viewed
as less important factors for the performance
rating. The rationale for each of the ratings is
explained in the preceding sections.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 69
Table 12: Average weighting of DAC criteria
DAC criterion Average weight-
ing
Number of up-
grades (weighting
factor 3)
Number of
downgrades
(weighting factor 1)
Relevance 2.3 5 0
Effectiveness 2.2 3 0
Impact 2.1 2 1
Efficiency 2.0 2 1
Sustainability 2.3 6 1
Differences in the weighting of individual DAC
criteria are also apparent between the various
types of evaluation, i.e. depending on the time
of evaluation, as illustrated below in Table 13).
Finally, in comparison with interim evaluations,
the focus in ex-post evaluations is seen to lean
more toward the registration of indirect results
and sustainability. That, however, is fully in line
with the corresponding terms of reference.
Table 13: Weighting of DAC criteria according to time (= type) of evaluation
DAC crite-
rion / Type
of
evaluation
Relevance Effectiveness Impact Efficiency Sustainability
Interim
evaluations 2.6 2.3 1.9 2.1 2.1
Final
evaluations 2.0 2.3 2.3 2.3 2.7
Ex-post
evaluations 2.0 2.0 2.3 1.5 2.5
The overview of all fifteen evaluations shows
how vital it is, with regard to the success of the
measures, that the various projects and pro-
grammes are in conformance both with the
sector strategies of BMZ and with the respec-
tive national policies and poverty reduction
strategies, and that they take into account key
guidelines on regional cooperation and integra-
tion, capacity development and IWRM. At the
same time, it is also important to systematically
pursue the concept of sustainable develop-
ment in all planning and to reconcile the objec-
tives dimensions of the project by way of a
multi-level approach. This requires not only a
systematic approach to capacity development,
but also closer attention to the duration of ad-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 70
visory services in the sense of efficient 'follow-
up assistance'.
3.6.2 Overall rating of evaluated projects
and programmes
With an average rating of 2.7, the evaluation
portfolio is situated in the 'satisfactory' (= 'suc-
cessful') range of assessment. Altogether,
there is a preponderance of positive results in
the evaluated projects/programmes. However,
the collectively satisfactory result was slightly
below expectations. Most of the individual pro-
jects/programmes (80%) were classified as
'successful', and only three pro-
jects/programmes (20%) were deemed 'unsuc-
cessful' (see also Table 14 and Table and Ta-
ble 15).
With regard to the projects and programmes
with an 'unsuccessful' overall rating of 4 ('un-
satisfactory'), the principle held true in each
case that a project or programme can only be
classified as 'successful' in terms of develop-
ment policy if its direct results ('effectiveness'),
indirect results ('impact') and sustainability are
all rated at least 'satisfactory'(3):
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 71
'Improvement of Water Supply in the Tigray
Region, Ethiopia' – The weighted overall rat-
ing of this project stands at 3.2 and is therefore
still within the 'satisfactory', 'successful' range.
Its sustainability, however, was rated 'unsatis-
factory' (level 4). The overall rating was there-
fore downgraded to within the 'unsuccessful'
range.
'Rural Water Supply in Maradi, Niger' – For
this project, the calculatory weighted rating
would be 3.6, or somewhere between 'suc-
cessful' and 'unsuccessful'. Since, however,
both the achievement of the objectives 'effec-
tiveness' and 'sustainability' were respectively
rated as 'unsatisfactory' and 'inadequate', the
overall rating was lowered to 'unsatisfactory'
('unsuccessful') in this case also;
'Upgrading of Municipal Services, Turkey' –
Despite a weighted overall rating of 3.2, this
project was classified as 'unsuccessful', be-
cause its impact was rated 'unsatisfactory'.
Table 14: Overall ratings of evaluated projects/programmes
Country Project/Programme Type of
evaluation
Rating Success
level Weight. Unweight. Total
Africa su-
praregional NWI
Interim
evaluations
2.0 2.0 2 Successful
Algeria PDEA 3.4 3.4 3 Successful
Yemen WSP 2.1 2.0 2 Successful
Jordan WMIA 2.0 2.0 2 Successful
Kenya WSRP-KEN 2.2 2.2 2 Successful
Peru PROAGUA 1.9 2.0 2 Successful
Zambia WSRP-ZMB 2.4 2.4 2 Successful
Viet Nam WDPC 2.5 2.6 3 Successful
Ghana EVORAP Final
evaluations
2.6 2.6 3 Successful
Kosovo WSWDP 2.5 2.4 2 Successful
Mexico GICA 2.8 2.8 3 Successful
Ethiopia IWSTR
Ex-post
evaluations
3.2 3.2 4 unsuccessful
India IGWDP 2.1 2.2 2 Successful
Niger PHV/MI 3.6 3.6 4 unsuccessful
Turkey QADI 3.2 3.2 4 unsuccessful
Total (average) 2.6 2.6 2.7 successful
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 72
The table also shows that the weightings of
individual DAC criteria do not have very much
influence on the final overall rating. The differ-
ence between the calculatory weighted and
unweighted overall ratings stays within the
decimal range and is, in principle, negligible.
Table 15: Distribution of overall ratings according to success level
Success level Rating Quantity Percentage
'Successful'
'very good' (1) 0 0%
'good' (2) 8 53%
'satisfactory' (3) 4 27%
'Unsuccessful'
'unsatisfactory' (4) 3 20%
'inadequate' (5) 0 0%
'useless' (6) 0 0%
Figure 10: Distribution of overall ratings
Statistical analysis on the basis of rank correla-
tions (see also Section 1.2.2) reveals a some-
what closer connection between the term of
the project/ programme and its overall rating
(Rho = - 0.32) than between the funding level
and the overall rating (Rho - 0.10). While the
level of significance of such correlations is by
no means statistically secure, it can neverthe-
less serve as a general indication of the impor-
tance of the duration of implementa-
tion/advisory services.
The following Table 16 contrasts the average
weightings of the DAC criteria with their aver-
age ratings. Clearly, the criteria 'relevance' and
'effectiveness' tend to be given 'good' ratings,
while 'sustainability' is much more likely to be
given a rating in the 'satisfactory' range. Within
the context of GTZ contract and cooperation
management according to the logic behind
AURA, 'effectiveness', i.e. the measures'
achievement of objectives, is considered par-
ticularly important (justification for the success
of the project/programme vis-à-vis the client
BMZ). In principle, then, this criterion should be
expected to get ratings well into the 'good'
range. This area, though, was afflicted by
some problems affecting the recording of re-
sults due to imprecise (and at times unrealistic)
indicators and, above all, a lack of fundamental
data for dealing with qualitative aspects. In the
opinion of the evaluators, the hypothesis that
more (direct and indirect) results were
0%
53%27%
20%
very good
good
satisfactory
not satisfactory
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 73
achieved by a given project or programme than
were ultimately recorded should be taken as a
basis for improved monitoring. Ultimately, the
significance of objectives achievement in terms
of its contribution to the indirect results ('im-
pact') and 'sustainability' of the measures must
also be taken into account (see also Table 6).
Good effectiveness does not necessarily lead
to good impact and good sustainability, but the
correlations are definitely relevant. 'Effective-
ness' and 'impact' were rated equal in some
40% of the evaluations. No project or pro-
gramme with only a 'satisfactory' rating for the
achievement of objectives was able to secure
a 'good' rating for 'impact' and 'sustainability'.
Only in two (2) projects/programmes (Yemen
and Niger) was 'impact' rated higher than 'ef-
fectiveness'.
Table 16: Comparison of average weightings and ratings of DAC criteria
DAC criterion Average weight-
ing
Average rating
Relevance 2.3 2.0
Effectiveness 2.2 2.5
Impact 2.1 2.7
Efficiency 2.0 2.8
Sustainability 2.3 2.9
As illustrated below in Figure 11, the average
overall rating varies significantly as a function
of the time and/or type of evaluation. Six of the
eight projects/programmes for which interim
evaluations were performed received an over-
all rating of 2 ('good'). Conversely, three of the
four projects/programmes assessed in ex-post
evaluations were rated 'unsuccessful', i.e. 4.
As already mentioned, a poor rating for a key
criterion led to a downgraded overall rating.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 74
Figure 11: Average rating according to type of evaluation
A differentiated view of the average overall
rating and of the average ratings of DAC crite-
ria according to region (see also Table 17)
yields no substantive results, because the
scope of random sampling differs widely from
region to region (Africa 6, Maghreb/Middle
East 3, Asia 2, Southeast Europe 2 and Latin
America 2 projects/ programmes each). In that
connection, the four evaluated pro-
jects/programmes in Asia and Latin America
show conspicuously good 'relevance'; this be-
comes apparent in their relatively good 'sus-
tainability' ratings. The 'sustainability' ratings of
the African and Southeast European pro-
jects/programmes are poorer by comparison.
Table 17: Average overall ratings and DAC criterion ratings according to region
DAC criterion / region Overall
rating Relevance
Effective-
ness Impact Efficiency
Sustain-
ability
Sub-Saharan Africa 2.8 2.2 2.5 2.5 3.0 3.2
Maghreb / Middle East 2.3 2.3 2.3 2.3 2.7 2.7
Southeast Europe 3.0 2.5 2.5 3.5 2.5 3.0
Asia 2.5 1.0 2.5 3.0 3.0 2.5
Latin America 2.5 1.5 3.0 3.0 2.5 2.5
2,22,6
3,5
1
2
3
4
Mid-term Evaluations Final Evaluations Ex-Post Evaluations
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 75
4. Rating of cross-cutting devel-
opment themes
4.1 Poverty reduction and MDGs
For more than 30 years now, the water supply
and sanitation sector has been counted among
the most important areas of German develop-
ment cooperation. In line with the BMZ 'Water
Sector Strategy' adopted in 2006 and the BMZ
'Waste Management Sector Strategy' adopted
in 2008, the objectives here coincide with
those stated in the MDGs and the Millennium
Declaration. German development cooperation
in the water supply and sanitation sector pres-
ently comprises projects and programmes with
a total volume of some EUR 4 billion and is
geared to improving the living conditions of
more than 80 million people worldwide. The
terms of reference are formed by the national
development strategies devised by the partner
countries on their own, including sector strate-
gies and poverty-reduction strategies. In ac-
cordance with the sector strategies, the Ger-
man contribution stands for sustainable, re-
source-conserving, target group-oriented ap-
proaches. The most important target group in
the water supply and sanitation sector is the
poor population9.
Target group: 'The key target group for German development cooperation in the water sector is the poor and extremely poor population, which currently has little or no access to safe and/or adequate drinking water, sanitation and/or water for farming. The urban slums and rural regions are especially important in this
context.'10
9 Waste Management Sector Strategy, BMZ, 2008.
10 Water Sector Strategy, BMZ, 2006.
It must be noted at this point that the majority
of the projects and programmes evaluated in
2008 were planned and commenced prior to
the conclusion of the Millennium Declaration
and the adoption of the cited sector strategies.
They are therefore based on prior approaches
to German development cooperation. Never-
theless, most of the projects and programmes
do relate to poverty, the sector strategies and
the MDGs, though perhaps to different extents
and at times abstractly. All evaluation reports
contain more or less detailed descriptions of
the extent to which the evaluated measures
help(ed) achieve the MDGs. In most cases,
that impact is situated at the overarching level
at the end of a long results chain. Only in a
very few cases is an 'anticipated' contribution
documented.
All evaluated projects and programmes are
positioned in the water sector and designed to
potentially achieve MDG 7 ('Ensure environ-
mental sustainability'):
improved management of water resources
constitutes a direct contribution towards
achieving MDG 7A ('Integrate the princi-
ples of sustainable development into
country policies and programmes and re-
verse the loss of environmental re-
sources');
improved access to WS&S constitutes a
direct contribution towards achieving
MDG 7C ('Halve, by 2015, the proportion
of the population without sustainable ac-
cess to safe drinking water and basic
sanitation'). The relevant indicators are (i)
the proportion of the population with ac-
cess to improved drinking water supply
and (ii) the proportion of the population
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 76
with access to improved sanitation facili-
ties.
It should be noted, however, that this global
assessment is of only very limited validity with
regard to the 'Upgrading of Municipal Services,
Turkey' project, because its impact was rated
'unsatisfactory' (level 4).
An analysis of the objectives systems of the
evaluated projects and programmes with re-
gard to MDGs shows that the improvement of
water supply (WS) is mainstreamed in seven of
the projects/ programmes and that wastewater
management/sanitation services (WM/S) are
anchored in the objectives systems of four of
those seven. The improvement of WS and
WM/S is mainstreamed both in the formulation
of objectives and in one indicator of three (3)
projects/ programmes. Only a fraction of the
projects/programmes have an objectives sys-
tem that takes into account the achievement of
MDG 7C, in particular the improvement of
WM/S. Hence, in relation to the entirety of the
evaluated development measures, the con-
crete contribution toward increasing the pro-
portion of the population with access to im-
proved sanitary facilities by 2015 is compara-
tively small.
The evaluated projects/programmes also con-
tribute to the achievement of additional MDGs
(see also Table 18):
10 projects/programmes (66%) contribute
to the achievement of MDG 1 ('Eradicate
extreme poverty & hunger'), for example
by creating jobs and potentials for eco-
nomic activities and promoting irrigated
agriculture;
11 projects/programmes (73%) contribute
to the achievement of MDG 4 ('Reduce
child mortality'), for example by reducing
water-induced diseases, in particular diar-
rhoea, and by improving hygiene in baby
care;
8 projects/programmes (53%) contribute
to the achievement of MDG 5 ('Improve
maternal health'), for example by reducing
water-induced diseases, improving hy-
giene at child birth, increasing the avail-
ability of hygienically safe drinking water,
and creating adequate sanitation services;
5 projects/programmes also contribute to
MDG 2 ('Achieve universal primary educa-
tion'), 5 projects/programmes to MDG 3
('Promote gender equality and empower
women'),
1 project/programme to MDG 6 ('Combat
HIV/AIDS, malaria and other diseases')
and 1 project/ programme to MDG 8 ('De-
velop a global partnership for develop-
ment').
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 77
Table 18: Overview of contributions by evaluated projects/ programmes to the achievement of MDGs
Country Project/
Programme
Type of
evaluation
W
S
W
M/
S
Contribution toward achievement of Mil-
lennium Development Goal (MDG) num-
ber:
7 1 2 3 4 5 6 8
Africa
supra-
regional
NIW
Interim
evaluations
X X X
Algeria PDEA X X X
Yemen WSP Z I Z I X X X X
Jordan WMIA X X
Kenya WSRP-KEN Z I Z I X X X X X X
Peru PROAGUA Z I Z X X X X
Zambia WSRP-ZMB Z I Z I X X X X X X
Viet
Nam WDPC X X X X
Ghana EVORAP Final
evaluations
Z I X X X X X
Kosovo WSWDP X X X X X
Mexico GICA X X X
Ethiopia IWSTR
Ex-post
evaluations
Z X X X
India IGWDP X X
Niger PHV/MI Z I X X X X X
Turkey QADI X
Total (average) 15 10 5 5 11 8 1 1
X=contribution to MDG, Z=covered by an objective in the objectives system, I=covered by an
indicator in the objectives system
4.1.1 Conceptual problems
Frequently, poor settlements have only low-
priority access to the services of utility compa-
nies. A corresponding context was established
in approximately 50% (7) of the evaluated pro-
jects/programmes. Ten of the evaluated pro-
jects/programmes therefore aim to improve the
operational efficiency of water utilities (WUs)
while creating the requisite institutional frame-
work conditions for privatisation and/or com-
mercialisation. Drinking water supply is the key
area. In these projects, poverty reduction can
only be achieved indirectly by way of long re-
sults chains. Only 4 of the 15 pro-
jects/programmes take into account a 'waste-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 78
water component' like that called for by the
BMZ sector strategy and which is predomi-
nantly geared to improving the living conditions
of the poor population.
Consideration of poor people's needs is impor-
tant in all phases of development measures,
especially in connection with:
the conceptual design of the measure;
the selection of water supply systems and
types of equipment, e.g. water kiosks;
the selection of sanitation systems and
technologies, e.g. EcoSan;
the design of tariff systems;
health and hygiene education;
public awareness-raising work at the sec-
tor institutions;
the structuring of capacity development.
In 12 of the 15 evaluated projects/ pro-
grammes, the poverty relevance of projects
and programmes is not evident in the objec-
tives system, i.e. in the objectives and indica-
tors, but is nevertheless given a positive rating
with regard to the assessed relevance of the
project or programme. In many of the pro-
jects/programmes, a result is projected on the
assumption that the measure contributes to
poverty reduction. One example of this is the
argument that improved WS&S always helps
reduce poverty. Likewise, the argument that
measures taken in rural settings automatically
help reduce poverty is inconsistent.
Demonstrating the plausibility of indirect results
of projects and programmes is a major chal-
lenge with regard to poverty reduction. Poverty
indicators derived from a baseline study, and
their systematic monitoring during both the
implementation phase and the follow-up assis-
tance phase of the project/programme, as well
as the formation of 'control groups' before im-
plementation commences, provide information
on results actually achieved.
In fact, little effort has been invested at the
project/programme level in this area until now.
In many places, people are afraid of the 'ex-
penditures' that such investigations can in-
volve. In this respect, training measures and
the dissemination of good approaches from
comparable projects and programmes should
enhance the determination of results, particu-
larly with regard to poverty reduction. Again,
the formulation of indicators is the point of
departure.
'Best Practice'
The programmes 'Water Sector Reform Programme, Kenya', 'Drinking Water and Sanitary Pro-
gramme, Peru' and 'Water Sector Reform Programme, Zambia' are characterised by direct pov-
erty relevance, not only in their objectives systems but also in their consideration of the poor
population for the design of tariff systems. Other projects and programmes (e.g. 'Improvement
of Water Supply in the Eastern and Volta, Ghana') are characterised by good poverty-reducing
effects despite their not being embedded in the objectives systems.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 79
4.1.2 Mainstreaming in development strategies
The poverty reduction strategies provide the frame of reference for improving the living conditions of
the poor population. Analysis of the reviewed evaluation reports shows that most of the projects and
programmes are successfully embedded within the development strategies of the partner countries, as
evidenced by their 'good' relevance ratings. This mainstreaming is particularly emphasised (see also
Section 3.1.1) in 5 projects/programmes (Yemen, Ghana, Kenya, Peru and Zambia). Direct integration
into national poverty reduction strategies receives a particularly positive rating. The practical imple-
mentation of corresponding policies and, hence, the complementary orientation of short-term poverty
reduction measures with long-term sustainability, however, has proven difficult.
4.1.3 Poverty marker in the offers and its distribution
To enable the evaluation of contributions to poverty reduction, each and every project and programme
of German technical cooperation must be categorised according to different poverty markers. The first
question is whether the project/programme in question is one with a direct results chain or one with an
overarching approach at the macro and sector levels. Projects and programmes with direct results,
that is, ones with a definable, directly accessible (via a short results chain) circle of beneficiaries, are
for example regional measures devoted to basic health, water supply, basic education and advice to
organisations providing direct services for a target group. Typical examples of projects and pro-
grammes with overarching results include government advisory service projects, promotion of nation-
wide systems (e.g. a sector information system), and advice on organisation and management at the
macro/sector level.
The following 'poverty markers' are used for classifying the projects and programmes in GTZ offers
submitted to the BMZ11
:
direct poverty reduction (target group-oriented)
o SHA (Self-help oriented poverty reduction)
SUA (Other direct poverty reduction esp. basic social services)
comprehensive poverty reduction (at macro/sectoral level, long results chains)
o MSA (projects situated at macro/sectoral level)
general development approach
o EPA (no direct or comprehensive poverty orientation)
Five of the evaluated projects/programmes (33%) contribute directly to poverty reduction (i.e. in a tar-
get group-oriented manner; marker SHA or SUA). Seven of the evaluated projects/programmes (47%)
make comprehensive contributions to poverty reduction via long results chains (marker MSA). Two
11 Erfolgreiche Armutsbekämpfung in der Arbeit der GTZ – Eine Handreichung, GTZ, 2008.
( Successful poverty reduction in the work of GTZ – A guiding framework, GTZ, 2008)
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 80
projects/programmes (13%) have, at most, indirect poverty relevance (EPA). For the 'Infrastructure
Programme - Water Supply and Wastewater Disposal, Kosovo', neither the GTZ offer nor the cor-
responding evaluation report contained or recommended any poverty marker.
The following Table 19 surveys the markers attached to each project/programme. It should be noted in
this connection that the analysis of DAC markers in the reviewed evaluation reports is inhomogeneous
and very fragmentary. Only three of the reports (Niger, Turkey and Viet Nam) include any reference to
the poverty markers contained in the GTZ offers. The relevant category assignments were confirmed.
Consequently, the poverty markers shown in the Table correspond to those in the GTZ offers. It was
not recommended that they be adjusted in connection with the evaluation.
Table 19: Poverty markers of the evaluated projects/programmes
As the overview shows, only one of the ongo-
ing programmes (Peru) that were assessed
within the scope of an interim evaluation has a
direct target group-oriented approach to pov-
Country Project/
Programme
Type of
evaluation
Poverty markers
SHA SUA MSA EPA
un-
speci-
fied
Africa
supra-
regional
NWI
Interim
evaluations
X
Algeria PDEA X
Yemen WSP X
Jordan WMIA X
Kenya WSRP-KEN X
Peru PROAGUA X
Zambia WSRP-ZMB X
Viet Nam WDPC X
Ghana EVORAP Final
evaluations
X
Kosovo WSWDP
Mexico GICA X X
Ethiopia IWSTR
Ex-post
evaluations
X
India IGWDP X
Niger PHV/MI X
Turkey QADI X
Total (average) 3 2 7 2 1
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 81
erty reduction (marker SHA). Six of the eight
ongoing projects/programmes are situated at
the macro/sector level and only contribute to
poverty reduction via long results chains
(marker MSA). By contrast, three of the four
completed projects/programmes that were
assessed within the scope of ex-post evalua-
tion (Ethiopia, India and Niger) were geared to
immediate, target group-oriented poverty re-
duction. All three began far ahead of the Mil-
lennium Declaration and the current sector
strategies: Ethiopia in 1996, India in 1989 and
Niger in 1995.
A prime example in this context is the 'Im-
provement of Water Supply - Eastern and
Volta Regions Assistance Programme,
Ghana'. This programme has the poverty
marker SUA. By strengthening the local gov-
ernment structures, it has direct poverty rele-
vance while indirectly contributing to the solu-
tion of structural problems. This cooperative
programme is part of Ghana's national poverty
reduction strategy.
Projects and programmes with a strong orien-
tation towards sector reform (macro level)
through their focus on sector-policy advisory
services, regulation and financing mechanisms
carry the marker MSA, because they only con-
tribute indirectly to poverty reduction via meso-
level adjustments at the intermediary organisa-
tions, e.g. utility companies. Especially in the
case of projects and programmes in which
poverty relevance is clearly embedded within
the objectives system (Kenya and Zambia), the
positive effect at the micro level is consistent.
In the opinion of the evaluators, the assign-
ment of markers is not transparent in all of the
projects/programmes. MSA markers in various
other projects/programmes (Algeria, Mexico)
are less plausible, because the results from the
intermediary organisations down to the target
group are less distinct in those projects. Such
projects/programmes actually should have
been assigned the marker EPA, since they
have a more general development-policy ap-
proach. In the case of the 'Integrated Water
Management Programme, Algeria', for ex-
ample, no reference at all is made to the solu-
tion of poverty issues specific to the partner
country and to the target groups. That, of
course, led to a poor relevance rating (see also
Section 3.1.1).
There appears to be a certain degree of fun-
damental uncertainty with regard to the as-
signment of poverty markers. The argument
that all projects and programmes in the water
sector contribute either directly or indirectly to
poverty reduction is inconsistent and would
imply that the poverty marker EPA is superflu-
ous. Hence, importance should be attached to
poverty analysis at the planning stage, and the
actual relevance to poverty should be given a
close, critical examination.
4.1.4 Target group differentiation and
poverty
analysis
Roughly half (7 out of 15) of the evaluated
projects and programmes exhibit objectives-
level target group differentiation in their objec-
tives systems. Six of the projects/programmes
also have such differentiation integrated into
their indicators. A basic distinction is drawn
between four target groups: urban populations,
provincial populations, rural populations and
poor populations:
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 82
three programmes (Kenya, Peru, Zambia)
are geared towards the poor urban popu-
lation;
one programme (Ghana) is geared solely
towards the provincial population;
two projects/programmes (Ethiopia,
Yemen) are geared towards the urban
and rural sections of the population.
Based on offers submitted to the BMZ and on
the evaluation reports, approximately half of
the evaluated projects and programmes have
no recognisably clear differentiation of target
groups. These projects and programmes are
generally geared towards the improvement of
sector institutions, WS&S companies or certain
regions.
As mentioned above, 12 of the evaluated 15
projects/programmes have direct or indirect
poverty relevance by reason of their corre-
sponding poverty markers. However, only in
the objectives systems of three programmes
(Kenya, Peru and Zambia) is there any clear-
cut pro-poor orientation. Thus, as analysed by
the evaluators, most of the evaluated projects
and programmes lack a clear poverty orienta-
tion, which is characterised as a key task in
most applicable sector strategies and con-
cepts.
It is surprising in this context that only a single
poverty analysis was performed (Yemen) and
that the absence of a poverty analysis only led
to downgrading of relevance in five of the
evaluations. In two programmes (Ethiopia and
Ghana), the analyses of the evaluation mis-
sions even perceive a risk that poor sections of
the population could be excluded the results of
the development measure (lack of acceptance
at target-group level for charge-carrying WS
facilities in place of traditional supply options).
In the 'Improvement of Water Supply in the
Tigray Region, Ethiopia' project, the target
group (provincial and rural population) is
clearly identified in the objectives system, but
only technical and institutional studies were
performed in preparation for the measure, i.e.
no socio-economic investigations for determin-
ing a willingness to pay. The requisite tariff
study was not performed until the end of the
measure.
Most particularly for projects and programmes
in least developed countries (LDCs), poverty
relevance should be integrated into the respec-
tive measures' concept and substantiated by
socio-economic studies, because it is there
that especially strong poverty reduction results
can be achieved.
4.1.5 Conclusions
As analysed by the evaluators, the rare exis-
tence of a definite pro-poor orientation in most
of the evaluated projects and programmes is
not necessarily attributable to a false concept.
It must be assumed that relevant impacts on
poverty can be demonstrated for nearly all
development measures. This orientation could,
however, be more plausibly represented if the
target groups of the projects and programmes
were more closely analysed and defined in the
objectives system. For the purposes of the
projects and programmes under review, a pov-
erty analysis would make a crucial contribution
to presenting the corresponding results poten-
tial and to establishing corresponding monitor-
ing activities. Especially in LDCs, poverty ori-
entation presupposes a strong results constel-
lation. Unfortunately, none of the evaluation
reports explicitly deal with the contribution of
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 83
development measures to broad-based eco-
nomic growth (participation of poor segments
of society in economic growth – pro-poor
Growth) in the respective partner countries,
because that area lies outside the terms of
reference prescribed for the evaluations.
Nevertheless, at least some of the projects and
programmes display either direct or 'semi-
direct' relevance to the target group. This con-
stitutes an essential prerequisite for a partici-
patory approach to the involvement of poorer
sections of the population in economic and
political processes. Some examples of this can
be found in the 'Drinking Water and Sanitary
Programme, Peru', in which the existing bond
between WUs and target groups/customers is
being strengthened by the institutionalisation of
'socio-economic environment management'. In
the rural, traditionally religious and conserva-
tive parts of Yemen, the 'Institutional Devel-
opment of the Water Sector, Yemen' pro-
gramme relies on the work of women's groups
within the village communities to establish local
water user committees and involve women in
downstream investment planning (construction
of cisterns).
The evaluation reports under review also pro-
vide no answer to the question of how well the
implemented projects and programmes were
actually able to sustainably contribute to a
more equitable distribution of water resources.
That area, too, was not included in the terms of
reference for the conducted evaluations. Until
now, the evaluators' analysis has done just as
little to document the relevant results at the
project/programme level as it has to provide
concrete data on the number of people who,
ultimately, benefitted from the measures in the
sense of poverty impacts.
4.2 Gender equality
4.2.1 Gender markers of the evalu-
ated projects and programmes
German development policy sees gender
equality as part of the policy dimension of de-
velopment and therefore attaches just as much
importance to it as to respecting human rights
and to promoting good governance and de-
mocracy. The mainstreaming of gender as-
pects is a key characteristic of quality at the
GTZ and derives from the concept of sustain-
able development. In addition to this, the GTZ
acts in pursuance of the gender mainstreaming
strategy of the German Government. The
overall goal is for women and men to draw
equal benefit from the development-policy
contribution of TC and be equally able to ac-
tively participate in shaping this contribution.
The relevant indicator is that, from 2006 on,
GTZ's final and ex-post evaluations increas-
ingly confirm that women and men are making
equal contributions to, and deriving equal
benefit from, TC projects and programmes.12
The gender (G) markers of DC projects and
programmes are intended to facilitate the as-
sessment of measures according to how much
they contribute to gender equality and to
strengthening the rights of women. They pro-
vide important information on anticipated re-
sults and help align development processes
with internationally agreed development objec-
tives of gender equality. The corresponding
12 GTZ Corporate Strategy on Gender Mainstreaming,
2006.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 84
markers employed within the OECD-DAC sys-
tem are13
:
marker G0: The development measure
does not have the potential to contribute
to gender equality and/or no gender-
specific results can be inferred from it.
The mere fact that men and women make
equal use of a development measure
does not justify allocation to the category
G-0, which may only be assigned in ex-
ceptional, well-founded cases;
marker G1: The development measure
has inferable positive effects on gender
equality, although gender equality is not a
main objective of the development meas-
ure. It is a significant secondary objective
of the project/programme (at the results
level) but is not solely decisive for its im-
plementation. The approach and proce-
dure for fostering gender equality should
be clearly mainstreamed in the project
design. The mere avoidance of negative
effects on the gender-equality situation
does not justify allocation of the marker
G1;
marker G2: Gender equality is a principal
objective of the project/programme and is
clearly expressed both in the project ob-
jective and in the entire hierarchy of ob-
jectives. Gender equality is at the core of
the project design, and direct results are
intended in this area.
13 BMZ/GTZ/KfW: Impacts on Gender Equality in Devel-
opment Cooperation Interventions: Gender Markers in
Technical and Financial Cooperation, 2006.
Positive gender-related results rarely occur all
by themselves, but must be actively integrated
into the project approaches. The available
ways and means of contributing toward greater
gender equality in connection with water pro-
jects and programmes differ according to pro-
ject approach and region. The evaluated pro-
jects and programmes devoted to the thematic
priority area 'water' are analysed below with
respect to their gender equality orientation.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 85
Table 20: Gender markers of the evaluated projects/programmes
Country Project/
Programme
Type of evalua-
tion
Gender markers
G0 G1 G2 unspecified
Africa supra-
regional NWI
Interim
evaluations
X
Algeria PDEA X
Yemen WSP X
Jordan WMIA X
Kenya WSRP-KEN X
Peru PROAGUA X
Zambia WSRP-ZMB X
Viet Nam WDPC X (X)
Ghana EVORAP Final
evaluations
X
Kosovo WSWDP X
Mexico GICA X
Ethiopia IWSTR
Ex-post evalua-
tions
X (FP)
India IGWDP X
Niger PHV/MI X (FP)
Turkey QADI X (FU)
Total (average) 4 11 0 0
Three projects/programmes (Ethiopia, Niger
and Turkey) were assigned the erstwhile 'F
markers' FP and FU, with FP corresponding to
G-1 and FU corresponding to G-0.14
All three
G-0 markers are consistent, because the pro-
jects/programmes (Africa supraregional, Kos-
ovo and Turkey) are, by definition, not oriented
towards gender equality. The 'Infrastructure
Programme Water Supply and Wastewater
14 In December 2000, the previous 'F categories' were
replaced with 'G markers' according to the DAC sys-
tem.
Disposal, Kosovo' is the only one of the fif-
teen evaluated projects/ programmes in which
a gender analysis was performed at the begin-
ning of the measure. In connection with the
'Nile Water Initiative - Planning and Man-
agement of Water Resources in the Nile
Basin', gender equality was written into the
national water policies.
The gender aspect was anchored as an indica-
tor in the objectives systems of two pro-
grammes (Yemen and Zambia). Accordingly,
both programmes are classified as G-1. This
marker is consistent, because gender equality
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 86
is, by definition, a secondary objective of both
programmes. In five projects/programmes
(Ghana, India, Kenya, Mexico and Peru) gen-
der equality is integrated into the design of the
respective measure but is neither substanti-
ated by a corresponding objective nor by an
indicator. However, according to statements in
the evaluation reports, these pro-
jects/programmes did include gender-specific
activities with discernible results at the target-
group level.
In the 'Improvement of Water Supply in the
Eastern and Volta Regions, Ghana' pro-
gramme (marker G-1), however, the concep-
tual design was not gender-responsive, nor
was a gender analysis performed. Gender
mainstreaming, though, is known to be in-
cluded in the Ghanaian sector policy. The re-
spective evaluation mission assumes that this
specification is being put into practice by ac-
tively involving women in the implementation
process.
In the 'Indo-German Watershed Develop-
ment Programme, India' (marker G-1), a
gender analysis was performed not at the be-
ginning of the measure, but in the course of its
implementation (1995). The measure has ef-
fected visible, albeit not quantifiable, gender-
specific changes, because the needs of
women were taken into account. Women also
benefit equitably from the results of the pro-
gramme. Behaviour toward women in the sec-
tor environment has changed recognisably,
and women's participation in public decision-
making processes is increasing. They have
gained better access to resources and are now
playing a more active role in the community.
Analysis of the evaluation reports also showed
that it may not have been consistent to assign
the marker G-1 to four of the pro-
jects/programmes (Algeria, Ethiopia, Jordan
and Viet Nam). Gender equality is neither a
secondary objective nor the principle objective
in those projects/programmes:
'Integrated Water Management Programme,
Algeria': In the original version of the offer,
gender equality was accounted for with an
indicator. The corresponding component, how-
ever, was discontinued, and no gender analy-
sis was performed. The underlying IWRM ap-
proach, which provides for the promotion of
women, was weakened between phase 1 and
phase 2. The mere promotion of women's
groups, coupled with the argument that women
play an important role with regard to the sensi-
tisation and participation of female villagers,
does not justify allocation to the category G-1.
'Improvement of Water Supply in the Tigray
Region, Ethiopia': No gender analysis was
performed, and no gender-responsive themes
were reflected in the reporting. The explanation
for assigning marker G-1 was that all WS&S
projects generally support women, because
they are the ones responsible for matters of
water supply and health. Hence, improving
WS&S, it is said, reduces their time expendi-
ture, whereas the contention that is also gen-
erates additional leisure time was already fre-
quently disproven. The assumption that all
measures taken to improve WS&S contribute
to gender equality does not justify assigning
marker G-1.
'Efficient Water Resources Management in
Irrigated Agriculture in the Jordan Valley
and Highland Areas, Jordan': Basically, the
fact that gender aspects are an important part
of Jordanian policy and that the decentralisa-
tion process in the water sector opens up new
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 87
possibilities for women to take part in the deci-
sion-making process without restrictions based
on status does not in itself justify assignment of
marker G-1. Nor, in principle, does the project-
engendered, anticipated potential for women to
become more closely integrated in water user
groups and in public life justify the assignment
of marker G-1.
According to the submitted reports, five of the
evaluated projects/programmes (Jordan,
Kenya, Kosovo, Niger and Zambia) make
highly aggregated contributions to the
achievement of MDG 3 ('gender equality').
Particularly in the 'Water Sector Reform Pro-
gramme, Zambia', this result is understand-
able, because gender equality is main-
streamed at the objectives level. By contrast,
the 'Institutional Development of the Water
Sector, Yemen' programme, which also has
gender aspects integrated within its objectives
system, is not assumed to have an effect on
MDG 3. The contribution made by the 'Infra-
structure Programme Water Supply and
Wastewater Disposal, Kosovo', with the
marker G-0, to achieving MDG 3 is inconsis-
tent. Again, it was argued that all projects and
programmes contributing to improved WS&S
also contribute indirectly to gender equality,
because women normally take care of their
families' health and hygiene matters. The
'WMIA Jordan' project does contribute indi-
rectly to achieving MDG 3, because many
women are employed as seasonal workers.
There is, however, a conspicuous lack of con-
crete data for documenting any genuinely gen-
der-related results in any of the pro-
jects/programmes (for example: surveys con-
cerned with easier access to water, improved
household hygiene, etc.). Only in the evalua-
tion of the 'Water Sector Reform Pro-
gramme, Zambia' was a gender-responsive
survey made of some 850 households. That
survey identified and recorded such definitely
positive effects of improved water supply on
such aspects as workload, leisure time and
health.
4.2.2 Conclusions
Many projects and programmes, sometimes in
cooperation with parallel measures, have the
potential to promote gender equality and to
implement activities identified in gender analy-
ses. In projects and programmes leading to the
incorporation of gender equality as a cross-
cutting theme in national water policies and
development strategies, however, practical
implementation often proves difficult, and no
relevant approach is implemented at the tar-
get-group level. The line of reasoning that all
projects in the water sector contribute to gen-
der equality and, hence, to the achievement of
MDG 3 is inconsistent. Although gender analy-
ses should, in principle, form part of the project
or programme‟s preparation process, corre-
sponding investigations were only performed at
the beginning of one project (Kosovo) and
during one programme (India).
Unfortunately, reliable data to document gen-
der-specific aspects and results of the pro-
jects/programmes are not available. Positive
effects and results constellations have
been'anticipated' rather than recorded and
documented until now. Only one evaluation
report, i.e. the ex-post evaluation of 'Rural
Water Supply in Maradi, Niger' programme
shows that, by the end of the programme, the
share of women in the newly established and
operational water user committees for super-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 88
vising the operation of the newly built and re-
habilitated well systems had reached 50%. By
the time of the ex-post evaluation, however,
that same percentage could only be found in
the roughly 10% of committees that were still
active.
In addition to conceptual mainstreaming of the
cross-cutting theme 'gender equality' in a given
project or programme, crucial importance is
also attached to the incorporation of appropri-
ate objectives and indicators in the objectives
system of the respective project or programme.
Then, when formulating the indicators, it is
likewise crucial to precisely formulate/define
the measurable appraisal factors thereby ena-
bling the straightforward recording of direct
results with regard to gender equality. Gender
analysis, which can also be performed after
implementation has begun – that is, during an
ongoing project or programme – can provide
information on the results dimensions and
serve as a basis for subsequent monitoring.
The most important recommendations drawn
from the evaluation reports on the cross-cutting
theme gender are:
gender analyses should always be per-
formed (even after the measure has
commenced);
indicators enabling straightforward regis-
tration of direct and indirect gender equal-
ity results need to be formulated;
reform programmes should actively pro-
mote the integration of gender equality
into national water policies as a cross-
cutting theme;
targeted awareness-raising and training
measures should be offered to women in
order to strengthen their role and, hence,
to initiate changes in the traditional gen-
der role model. The targeted assignment
of women to management-level positions
should be promoted, and initiatives and
forms of organisation geared to women
should be created and strengthened.
4.3 Effects on the partners' capac-
ity for action
4.3.1 Capacity development within
the context of the evaluated
projects and programmes
Capacity development is a holistic process
through which people, organisations and socie-
ties mobilise, maintain, adjust and expand their
capacities for sustainable development. Sup-
port for capacity development by external part-
ners is a central instrument of development
cooperation with which people, organisations
and societies are empowered to develop and
expand their capacities for action and man-
agement. Basically, the capacity development
process requires the support of the involved
actors and presumes their assumption of
'ownership', i.e. high levels of identification with
and commitment to the targeted changes.
Ownership may, however, not develop until the
reform process has already set in; external
partners can foster the development of owner-
ship and assume temporary co-responsibility
during the process.
GTZ's specific approach to supporting capacity
development derives from its concept of sus-
tainable development. Development is under-
stood as a permanent process of seeking,
negotiating and learning on the part of all
stakeholders that defies detailed advance
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 89
planning. Its basic principles are a holistic ap-
proach and process/value orientation. This
approach cannot be implemented via financial
transfers, but depends on the joint responsibil-
ity of the partner(s) and GTZ in the sense of
mutual accountability15
.
According to its definition, capacity develop-
ment takes place at four levels: at the individ-
ual level, the institutional level and the net-
working and system development levels. This
breakdown displays certain parallels with the
intervention levels, but it also provides an addi-
tional analysis model for the strategic orienta-
tion of projects and programmes. The Paris
Declaration promotes the development of ca-
pacity development strategies at the national
and sector levels, and GTZ has participated in
the drafting of corresponding strategies in
many different countries. German TC attaches
special importance to verifying the results of
capacity development. Hence, monitoring and
evaluation of capacity development measures
enjoy priority at GTZ. The monitoring systems
of projects and programmes should do more to
record changes in the capacities (for action
and management) of individuals and organisa-
tions, and developments in social contexts, via
appropriate indicators. All this must be ac-
counted for both in offers for projects and pro-
grammes and in operational planning.
The evaluation reports neither contain a pre-
cise definition of the capacity development
interventions that were conceived and imple-
mented in the respective projects and pro-
15 Das Verständnis der GTZ von Capacity Development -
Ein Orientierungs- und Handlungsrahmen für die GTZ,
2006 ('GTZ‟s Understanding of Capacity Development:
A Guiding Framework for Corporate Action
grammes, nor do they describe the corre-
sponding results. If at all, the respective texts
merely contain fragmentary assessments and
reflections. It must also be noted in this con-
nection that the use of the term 'capacity de-
velopment' in the evaluation reports (as well as
in the GTZ offers) does not follow any gener-
ally applicable definition. In most cases, 'ca-
pacity building' is equated with capacity devel-
opment, but such terms as 'training', 'institu-
tional capacity building' and 'institutional
strengthening' are also used synonymously. All
in all, in the simultaneous absence of an ap-
propriate concept or strategy, it is very difficult
to identify and categorise the respective inter-
ventions at the level of the individual projects
and programmes. At the same time, it is not
practicable within the scope of cross-section
evaluation to identify factors of success and
failure that are relevant to capacity develop-
ment, because they were not described.
The following Table 21 summarises the dimen-
sions and results levels of all fifteen evaluated
projects/programmes. It must be noted in this
connection that the information provided was
'filtered out' of the evaluation reports by the
authors of the cross-section evaluation. The
interventions and levels indicated are therefore
those that were actually identified, not those
that were 'targeted' or expected. The docu-
mentation includes no consistent depiction of
the respective interventions and results con-
stellations.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 90
Table 21: Levels of capacity development in the evaluated projects/programmes
Country Project/
Programme Training OE
Network-
ing
System
devel.
Results
level
CD
concept
Africa
supra-
regional
NWI X X macro
Algeria PDEA X X micro-meso
Yemen WSP X X X X multi-level X
Jordan WMIA X X micro
Kenya WSRP-
KEN X X X X multi-level
Peru PROAGUA X X X X meso-
macro
X
Zambia WSRP-
ZMB X X X X multi-level
Viet Nam WDPC X meso
Ghana EVORAP X X meso
Kosovo WSWDP X X meso
Mexico GICA X X meso
Ethiopia IWSTR X X meso
India IGWDP X X X micro X
Niger PHV/MI X X micro
Turkey QADI X meso
The overview shows that the explicit priority
area of actually implemented capacity devel-
opment interventions is situated within the
individual dimension in terms of personnel
training. Such measures are often combined
with the provision of advisory services for insti-
tutional or organisational development of
meso-level executing agencies and intermedi-
ary organisations. Often, concrete assistance
is provided for process optimisation at utility
companies. Only a minor share of the pro-
jects/programmes include measures geared to
system development and network formation. It
is also apparent that, in analogy to the focusing
of results levels (see also Section 2.2.2), there
is a heavy concentration of capacity develop-
ment measures at the meso level. Only about
one third of the projects/programmes intervene
in the sense of capacity development at the
macro level. What is more surprising is that
only three programmes (India, Yemen and
Peru) have a documented, monitored capacity
development strategy or concept.
Comparison of the conceptual results levels of
the evaluated projects and programmes shown
in Figure 2 with the 'actual' intervention levels
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 91
summarised in Table 21 reveals discrepancies
in some of the projects/programmes in the
sense that the realities of implementation devi-
ate from the concept or planning. To the extent
that such discrepancies are noticed, the plan-
ning is rarely modified. For example, PDEA
Algeria intervenes only at the micro and meso
levels, despite the fact that a significant macro-
level contribution was anchored in the pro-
gramme concept. By way of contrast,
EVORAP Ghana unsuccessfully attempted to
implement a multi-level approach as designed
and with due attention to the micro and macro
levels. According to the evaluators' analysis,
most of the programme's work is done at the
meso level. The WMIA Jordan project, on the
other hand, provides most of its outputs with a
micro-level focus, even though the original
planning called for decidedly more attention to
the meso and macro levels.
The most important capacity development
interventions of selected projects and pro-
grammes are compiled below to convey a
general idea of their spectrum. (The PDEA
Algeria and IWSTR Ethiopia programmes were
omitted, because the respective evaluation
reports shed no light on interventions under-
taken for capacity development purposes):
The project entitled 'Nile Water Initiative -
Planning and Management of Water Re-
sources in the Nile Basin' (NWI) focuses
macro-level capacity development interven-
tions in the areas 'improving individual capaci-
ties' and 'system building' by providing task-
specific and strategic training measures for the
staff of national sector ministries (political
analysis and dialogue, communication, strate-
gic planning) and by exerting positive influence
on framework conditions through support for
national water policies (bearing in mind trans-
boundary cooperation and IWRM principles);
EVORAP Ghana: Training measures for WU
personnel (financial, commercial and technical
operations) plus organisational advice for the
development of utility companies;
IGWDP India: Training measures, information-
gathering tours, networking of intermediary
organisations (NGOs);
WSP Yemen: Human resources development
and training (HR component in the pro-
gramme), organisational advice for the estab-
lishment of municipal WUs and water authori-
ties, networking of sector institutions, and advi-
sory services to generate a water policy;
WMIA Jordan: Training measures for mem-
bers of water-user groups and advisory staff at
the lead executing agency, and advisory ser-
vices for establishing water user groups and
committees;
WSRP Kenya: Training measures for staff of
regional water authorities, WUs and the higher-
level sector authority, organisational advice for
water authorities and the trust fund for the
establishment of financing mechanisms and for
the introduction of appropriate technologies
(water kiosks, sanitation systems), regional
networking and system development (regula-
tion, pro-poor policy development and invest-
ment planning);
WSWDP Kosovo: Training measures for WU
staff (financial, commercial and technical op-
erations) and assistance to WUs for introduc-
ing technical improvements in technical opera-
tions;
GICA Mexico: Training measures for staff of
water and regulatory authorities and for per-
sonnel of non-governmental intermediaries,
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 92
and organisational advice for staff of water and
regulation authorities;
PHV/MI Niger: Training measures (and
awareness-raising measures) for village water
user committees, building contractors and staff
of regional water authorities, advisory services
for the establishment of village water user
structures, and task-specific assistance to
regional water authorities;
PROAGUA Peru: Training measures for per-
sonnel of WUs, advisory services for the estab-
lishment and development of a countrywide
human resource development system for the
water sector, organisational advice for utility
companies, establishment of association struc-
tures for WUs and sanitary utility companies,
and establishment of an exchange system;
WSRP Zambia: Training measures for per-
sonnel of regional water authorities and WUs,
organisational advice for water authorities and
the trust fund for establishing and developing
financing mechanisms and introducing appro-
priate technologies (water kiosks, sanitation
systems), regional networking and system
development (regulation, pro-poor policy de-
velopment and investment planning);
QADI Turkey: Project and investment planning
training measures for staff at local government
units;
WDPC Viet Nam: Training measures on tech-
nical operations for staff of national sanitation
utilities.
The evaluation reports contain little information
about the actual results of the implemented
capacity development interventions. Measures
are referred to as positive, but without citing or
explaining why they were regarded as suc-
cessful, one such case being Peru, where a
national programme for human resource de-
velopment based on the capacity development
concept was successfully established. The
respective progress reports would have been
useful here as a source of information. The
evaluation report did show, however, that the
extensive training measures at the utility com-
pany level were particularly successful in pro-
fessionalising the operations. In addition, train-
ing measures for decision-makers in national
policy dialogue are of decisive importance.
Especially the larger sector programmes with a
systematic, well-balanced multi-level approach
(Yemen, Kenya, Zambia) stand as good ex-
amples of the very complex results constella-
tion of capacity development interventions.
At the same time, it is evident from the reports
of the evaluators who performed the individual
evaluations that the time allowed for establish-
ing or decentralising capacities and capabilities
for action, or for processes of change, was
usually too short. Time and again, it is recom-
mended in this connection that advisory ser-
vice cycles be lengthened in the sense of
process support. The recommendations made
by the authors of the individual evaluations
concentrate on the need for systematic expan-
sion and focusing of capacity development
interventions in order to fill the existing 'capac-
ity gaps' in the partner countries.
4.3.2 Conclusions
In principle, capacity development was found
to be taking place at all levels in the evaluated
projects and programmes, and there was a
very definite setting of priorities in the area of
training measures and organisational consul-
tancy. Adequate consideration of the areas
'networking' and 'system development' – the
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 93
areas with the greatest results options – within
the scope of a 'multi-level approach' is, how-
ever, only evident in the evaluation reports of a
minor share of the projects/programmes. In the
more complex projects and programmes, the
evaluators also noticed relevant difficulties with
the interlinkage of technical, organisational and
policy advisory services (main dimensions:
steering of inputs and professional training).
It was surprising to note the apparent lack of a
systematic approach and the basically inade-
quate conceptual base in the greater part of
the evaluated projects and programmes. With-
out a capacity development strategy that also
the implementing partner can consider trans-
parent, a given project or programme can
hardly produce system-oriented results. In this
respect and in many cases, the national water
policies and sector strategies prescribe general
guidelines. In the view of the evaluators, each
and every development measure should for-
mulate an appropriately oriented development
strategy and routinely examine it with regard to
relevance and implementation. In this respect,
the introduction of 'Capacity WORKS' as a
toolbox for corresponding analytical and plan-
ning processes should yield an effective im-
provement.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 94
5. GTZ's concept of sustainable
development
5.1 GTZ's concept of sustainable
development within the context
of the evaluated projects
and programmes
The three principal characteristics of the work
performed by GTZ in accordance with its con-
cept of sustainable development are:
a holistic approach: combining the eco-
nomic, social and ecological aspects (ob-
jectives dimensions) with sectoral, organ-
isational and policy advice at the micro,
meso and macro levels;
a process-oriented approach: help to-
wards self-help, creating transparency
with regard to the interests of different
stakeholders and promoting interaction
between the state, civil society and the
private sector;
a value-oriented approach: promoting
democracy, the rule of law, human rights,
gender equality, good governance and a
social and ecological market economy.
Fourteen of the evaluation reports adopted a
largely stereotypical, positive view of these
principles, substantiating the context of the
respective project or programme in a very ab-
stract manner with regard to the objectives
dimension and all three approaches (see also
the examples cited below: Zambia and Tur-
key). Only in the rarest of cases is there any
recognisably concrete relevance to the actual
interventions, and then only by way of very
long results chains. In the opinion of the au-
thors of this cross-section evaluation, the rele-
vant parts of the reports are, in their present
form, not very informative. No in-depth analysis
is possible on that basis.
Only in the case of the 'Improvement of Wa-
ter Supply in the Tigray Region, Ethiopia'
project did the evaluators deny its claim of
having a sustainable approach to development
because it dealt almost exclusively with eco-
nomic aspects of WU operations and was
practically non-existent at the target-group
level. It gave no consideration to the interests
and needs of the target groups and did not
monitor development in the supply situation. At
the same time, important factors like ground-
water availability and the economic viability of
poor households went unaccounted for in the
plans for expanding the water supply.
Table 22 below represents an attempt to realis-
tically record how well the characteristics of
sustainable development were actually ac-
counted for in the corresponding parts of the
individual evaluations. Twelve projects and
programmes are seen to be pursuing a holistic
approach, but only eight of the measures have
a process-oriented concept. Simultaneously,
most of the projects and programmes (11) are
acknowledged to have an explicit orientation
towards values, and the same number of pro-
jects and programmes do a well-balanced job
of combining the economic, social and ecologi-
cal objectives dimensions. A failure to give
balanced consideration to the objectives di-
mensions was found in four of the pro-
jects/programmes (Africa supraregional, Ethio-
pia, Niger and Turkey).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 95
Table 22: Consideration of the characteristics of sustainable development in the evaluated pro-
jects/programmes
Country Project/
Programme
Holistic
approach
Process
orientation
Value
Orienta-
tion
Objectives dimensions
Africa
supra-
regional
NWI X X economic, ecological
Algeria PDEA X X economic, social, ecological
Yemen WSP X X X economic, social, ecological
Jordan WMIA X X X economic, social, ecological
Kenya WSRP-KEN X X X economic, social, ecological
Peru PROAGUA X X X economic, social, ecological
Zambia WSRP-ZMB X X X economic, social, ecological
Viet
Nam WDPC X economic, social, ecological
Ghana EVORAP X X economic, social, ecological
Kosovo WSWDP X economic, social, ecological
Mexico GICA X X X economic, social, ecological
Ethiopia IWSTR economic
India IGWDP X X X economic, social, ecological
Niger PHV/MI X economic, social
Turkey QADI X economic, social
A holistic approach is established for the 'Wa-
ter Sector Reform Programme, Zambia', for
example, where the social, economic and eco-
logical objectives dimensions are combined by
the programme's focus on the supply of drink-
ing water and sanitation at socially acceptable
cost with incorporation of the IWRM approach.
At the same time, all the advisory services
provided at the sectoral, institutional and politi-
cal levels are based on a systematic multi-level
approach. Key areas of macro-level advice are
water policy and donor harmonisation. At the
meso level, the programme intervenes via
organisational advice (priority area WUs) and
the development of financing and sector infor-
mation systems. This is supplemented by sup-
port for the establishment and strengthening of
water user groups. With regard to networking,
the programme fosters exchange between the
numerous sector institutions and civil society
organisations and the private sector. In the
sense of good governance, the establishment
of a regulatory authority led to more transpar-
ency and quality improvements within the sec-
tor (monitoring of standards, tariffs, etc.). The
programme's value orientation is expressed in
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 96
terms of regard for human rights in the sense
of creating access to safe drinking water, fos-
tering democratisation through a participatory
approach and intervening at micro-level to
identify locations for new water kiosks, and
emphasising a social market economy by in-
troducing a socially equitable tariff structure.
A holistic approach aligned with the IWRM
concept was ascertained for the 'Drinking
Water and Sanitary Programme, Peru'. The
ecological (wastewater management and
treatment), economic (better economic per-
formance capabilities on the part of WUs) and
sociopolitical dimensions (sociopolitical envi-
ronment management of WUs) all stand in a
well-balanced relationship to one another,
even though the ecological dimension is lag-
ging behind because of minor impacts on sani-
tation so far. The programme pursued a multi-
level approach that is to contribute to sustain-
able development, especially through interac-
tion between the macro and meso levels.
Thus, participation in the design of the sector
policy and regulatory framework is geared both
to fostering the sustainability of progress made
by the WUs and to disseminating the learning
effects achieved at the WUs. The intended
coordination of training and advisory services
for the water sector also reaches beyond the
WUs participating in the programme to benefit
the entire sector. If the programme is brought
to a successful conclusion, the strengthened
performance capacities of the WUs and their
sustainably improved supply services can be
expected to contribute to sustainable develop-
ment. The development measure is based on a
process- and value-oriented approach. Particu-
larly at the local level, one of the programme's
key priorities is to improve dialogue between
the WUs and the social groups in the local
area. In addition, the interests of WU personnel
were appropriately considered, and importance
was attached to improving the internal com-
munication.
5.2 Conclusions
It is apparent that the objectives systems under
review, some of them very complex, with their
many indirect and highly aggregated results
constellations, basically cater to all objectives
dimensions and approaches of GTZ's sustain-
able development concept. Thanks to this 'an-
ticipated results constellation', the larger sector
programmes in particular, with their very ex-
plicit multi-level approach, are practically al-
ways in line with the concept. The extent to
which these relationships really can be attrib-
uted to actual project interventions should at
least be discussed within the scope of the
evaluations in order to promote learning effects
and further development.
Certainly, the good relevance ratings of the
evaluated projects and programmes indicate
that the approaches to sustainable develop-
ment were adequately considered in the con-
ceptual design. Conversely, the rather medio-
cre success of the projects and programmes
with regard to overarching results (impact) and
sustainability lead to the conclusion that the
actual implementation of these approaches still
leaves room for improvement. In the opinion of
the evaluators, the underlying contradictions
should be addressed somewhat more candidly.
Regarding the rather unbalanced consideration
given to the social dimension, greater attention
to poverty studies and/or target-group analyses
at the planning stage could yield improvement.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 97
6. Sectoral assessment
6.1 Cooperation and task-sharing
among German interface or-
ganisations
Table 23 below surveys the cooperation be-
tween and coordination of German DC inter-
face organisations in all fifteen evaluated pro-
jects and programmes. A basic distinction is
drawn between 'cooperation', which normally
takes place within the scope of a cooperation
agreement and involves common goals,
strategies and areas of responsibility (tasks),
and 'coordination', in which both sides keep
each other informed about interventions and
planning.
As the table shows, more than half of the pro-
jects and programmes involve cooperation with
KfW, including formalised cooperation in seven
'cooperative projects/ programmes'. The Ger-
man Development Service (DED) is involved in
four projects/programmes via cooperation
agreements, while CIM (Centre for Interna-
tional Migration and Development), the Federal
Institute for Geosciences and Natural Re-
sources (BGR) and Inwent - Capacity Building
International participate by way of cooperation
and coordination.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 98
Table 23: Cooperation between the evaluated projects/programmes
and German DC interface organisations
Country Project/
Programme CP KfW DED CIM BGR Inwent
Africa
supra-
regional
NIW
Algeria PDEA discon-
tinued16
phase I
only
Yemen WSP X X X X X X
Jordan WMIA (X) (X) (X) (X) (X)
Kenya WSRP-KEN X X X (X)
Peru PROAGUA X (X) (X)
Zambia WSRP-ZMB X X X X
Viet
Nam WDPC
X X X X
Ghana EVORAP X X
Kosovo WSWDP (X)
Mexico GICA (X)
Ethiopia IWSTR
India IGWDP X
Niger PHV/MI X X
Turkey QADI
X Cooperation (X) Coordination of activities
16 During phase I the programme was established as a cooperative programme with KfW. The results of components 2 and 3
were arranged directly with KfW, and objectives achievement was dependent on FC. The cooperation was discontinued
when German FC was withdrawn from Algeria.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 99
The 'Water Sector Reform Programme,
Zambia' stands as an example of complex
relationships between German DC organisa-
tions. Working on the basis of a formalised
cooperation agreement, GTZ, KfW, DED and
CIM try to establish synergies in their interven-
tions and, above all, to avoid the delivery of
overlapping services. DED and CIM focus
primarily on providing advisory services to
WUs and local water authorities (operation,
monitoring), while KfW promotes the develop-
ment of infrastructure. Simultaneously, utility
employees receive targeted training through
Inwent measures. In a different programme
component, BGR provides assistance for the
management of groundwater resources. With-
out such cooperation, the objectives could
never be achieved.
In projects and programmes involving coopera-
tive relationships between German interface
organisations, joint achievement of objectives
calls for a high level of coordination and joint
planning. In most of the projects/programmes,
the partners in cooperation meet those re-
quirements. It is important in this connection to
unequivocally clarify who is responsible for
lead management. Experience drawn from the
'Rural Water Supply in Maradi, Niger' project
shows that vague responsibilities and the lack
of a clearly assigned control mandate for the
GTZ or the KfW can be very disadvantageous
in terms of objectives achievement and, above
all, efficiency.
In view of the complex cooperative relationship
between the German interface organisations
involved in the 'Institutional Development of
the Water Sector, Yemen' programme, and
considering the fact that three German consult-
ing firms were also involved in the programme,
the evaluators had reservations about the effi-
ciency of such a structure. The question arises
as to whether services rendered by the BGR
could perhaps be provided at lower cost by
local or German consultancy firms.
Especially for non-formalised cooperation, the
evaluation reports identify problems concern-
ing the coordination of interface organisations.
Usually, 'coordination' hardly extends beyond
simple 'information', i.e. no real synergetic
interlinkage takes place between interventions
in these projects/ programmes. Hence, coop-
eration with the KfW, for example, is at the
same level as with other donor institutions,
namely that of non-committal exchanges of
information that do not enable concerted action
in the interest of German DC regulations and
directives.
Despite the heterogeneity of the assessments
contained in the evaluation reports, progres-
sively formalised cooperation has improved the
image of 'joined-up development cooperation'.
Nevertheless, the 'coexistent' activities of
German DC actors persist in many places.
There remains a lack of clear-cut steering
mechanisms with which the actors would be
able to coordinate their measures even in the
absence of formal contractual relations.
6.2 Embedment of development
measures in the activities of the
international donor community
The fact that the evaluated projects and pro-
grammes generally received good relevance
ratings indicates that their underlying concep-
tual designs are highly integrated into the na-
tional sector strategies and synergize with the
interventions of the international donor com-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 100
munity. Maximum embedment in concerted
assistance measures is a decisive success
factor with regard to relevance. At the same
time, this is an area that, basically, receives
very positive assessment for all evaluated
development measures.
As analysed by the authors of the cross-
section evaluation, however, the following dis-
tinction must be drawn: for a large share of the
projects and programmes, the causal relation-
ship between the strategic orientation of the
respective project or programme and the
common objectives and results dimensions of
the donor community interventions of a given
country is largely theoretical. That alone, how-
ever, does not constitute effective embedment.
It also becomes apparent that, basically, coor-
dination only takes place within projects and
programmes characterized by a strong macro-
level presence. In reality, projects and pro-
grammes that principally act at the meso or
micro level – including most of those dealt with
in the present evaluation sequence – fail to be
involved in donor coordination and harmonisa-
tion, even though this would be easily feasible.
It was also noticed that there is no ongoing
cooperation between the 'Management of
Water Catchment Areas for the Río Lerma
and Toluca Valley and Río Balsas, Mexico'
project and other activities of the international
donor community, since these have different
regional and/or thematic priority areas. Accord-
ingly, the project has no harmonised sector-
wide approach.
An indicator for genuine cooperation within the
international community should be the exis-
tence of joint planning and interventions. This
holds true for three (3) of the evaluated pro-
jects/programmes (Yemen, Kenya and Zam-
bia):
'Institutional Development of the Water Sec-
tor, Yemen': Both the presence and the profile
of the international donor community within the
sector are high. Within the scope of its own
advisory activities, the programme works to
update and implement the sector policy and
strategy and to cooperate closely with the KfW
in the development of programme-oriented
joint financing (SWAp). The 'Water Sector
Support Programme' (WSSP) is a preliminary
form of SWAp. The field testing of innovative
methods and approaches, as called for, inter
alia, by the sector strategy was incorporated
into the programme by way of the 'milestones
concept' introduced in cooperation with the
KfW. Its further development is being facilitated
by SWAp and other tools. The fact that the
programme is very much present at all relevant
formal and informal decision-making and im-
plementing levels – hence yielding tangible
synergies – was given a particularly positive
rating.
'Water Sector Reform Programme, Kenya':
The international donor community is focusing
its cooperation on the development of a SWAp
for the country. All donor activities in the Ken-
yan water sector are being coordinated by a
steering group headed by the KfW in collabora-
tion with Swedish and French DC. Within that
scope, the GTZ supported, among other
things, the operationalisation of a SWAp com-
mittee, which then set up the rules and princi-
ples of cooperation for the sector. All this is
now anchored within the framework of the
cooperation agreement entered into by the
donor organisations. Formalised cooperation
between the GTZ, Danish and Swedish DC
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 101
and UNICEF is being managed by a joint
committee that initiates joint budget planning
and operations planning on the basis of the
sector strategies, among other things.
6.3 Implementation of the multi-
level approach in advisory pro-
jects and programmes
GTZ pursues a holistic approach. To achieve
this, social, economic and ecological aspects
are combined in the system of objectives. Ad-
visory support of a sectoral, political and or-
ganisational nature is provided at the macro
level, the meso level and the target group-
oriented micro level. The GTZ‟s work is based
on a values concept comprising the promotion
of democracy, the rule of law, good govern-
ance, human rights and a social and ecological
market economy. The GTZ‟s process-oriented
approach is geared to providing help towards
self-help, the transparent representation of
stakeholder interests, and mediation between
the state, civil society and the private sector. In
so doing, GTZ's goal is to embed its develop-
ment measures in the activities of international
donors and to apportion the tasks among all
implementing organisations. This attaches
importance to cooperative projects and pro-
grammes marked by complementary technical
and financial cooperation. The IWRM approach
plays a special role in the water sector and
should be integrated into all development
measures.
The assessment of the underlying approaches
is handled differently from case to case in the
reviewed evaluation reports. Assessment of
the technical and methodological approach is
not uniformly systematic. While some evalua-
tors attach much importance to a detailed
analysis of the methodology and investigate all
the various approaches taken, other reports
make do with a more comprehensive, abstract
assessment. In most cases, the strengths and
weaknesses of a measure's conceptual design
are reflected in the effectiveness rating.
As already mentioned in Sections 2.2.2 and
4.3.1, only three programmes (Yemen, Kenya
and Zambia) were found to have a complete,
conceptually well-balanced, multi-level ap-
proach mainstreamed at all impact levels.
In the project entitled 'Nile Water Initiative -
Planning and Management of Water Re-
sources in the Nile Basin', the design of the
methodological approach was of limited flexibil-
ity, because the project had to be closely
geared to the existing organisational structures
and donor landscape. All in all, the project
aimed to harmonise the water policies of the
member states and, hence, to effect change at
the policy-making level. In this project, the
multi-level approach was implemented within
the regional context, that is, the macro level is
occupied mainly by international committees
and working groups.
In the 'Improvement of Water Supply in the
Tigray Region, Ethiopia' project, the concept
of working first in eight pilot towns and then in
19 replication towns was rated 'good', and the
combination of competence building with the
direct GTZ contribution was emphasised as
particularly positive thanks to the effective
combination of theory and practice. The envis-
aged multi-level approach was, in principle, not
implemented, because the project focused
more on advising the Water Resource Devel-
opment Bureau than on addressing the re-
gion's population. On this point, the evaluators
recommend a stronger micro-level orientation.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 102
At its core, the 'Integrated Water Manage-
ment Programme, Algeria' is a policy advi-
sory programme with an IWRM approach
geared to improving governmental perform-
ance at the national, regional and local level.
Its priority aim is to foster capacity develop-
ment among the staff of the sector institutions.
There are no plans to aim the programme
more toward a sector-wide approach (SWAp).
The programme is active at the national, re-
gional and local level (multi-level approach).
Close cooperation with the population facili-
tates exchanges between the administration
and the water users themselves. The working-
group approach was given an emphatically
positive rating as an important instrument of
institutional learning via cooperation with the
partner.
The 'special evaluation issues' chapter of the
final evaluation report on the 'Improvement of
Water Supply in the Eastern and Volta Re-
gions, Ghana' programme deals with issues
concerning the measure's basic concept. The
priority area of this development measure is
the provision of drinking water in small rural
towns by way of locally managed 'water and
sanitation development boards' established
within the scope of the programme. The pro-
gramme implemented a holistic approach to
the provision of technical, organisational and
policy advisory services. The multi-level ap-
proach includes advising and supporting the
sector institutions at the national and regional
level and assisting the district and municipal
levels. The interlinkage of the objectives di-
mensions was analysed in detail. The process-
and value-oriented approach also comes to
bear in this programme. Its local-level help
towards self-help and efforts to foster the de-
centralisation process were seen in a particu-
larly positive light, as were its close coopera-
tion and interfacing with the water programme
in Benin. The instruments and advisory ap-
proaches developed there for the areas of
decentralisation, operation and maintenance of
provincial water supply systems and knowl-
edge management were further developed and
modified for Ghana. The fact that this pro-
gramme invested little effort in local-level cam-
paigns and public awareness activities was
rated negatively, as was its lack of adequate
wastewater management aspects.
By contrast, the 'Rural Water Supply in Ma-
radi, Niger' project consciously dispensed with
the concept of a multi-level approach. Its prior-
ity was to work at the micro level and build a
strong relationship with the target group. The
holistic, participatory approach in the sense of
'integrated water resources management' in-
volving a systematic balancing of economic,
social and ecological objectives was not of
primary interest. Cooperation between the KfW
and the GTZ within the scope of this coopera-
tive project was given a negative rating. Nei-
ther was the conceptual design (technical
models, operation and maintenance concepts)
jointly adjusted, nor were the objectives and
indicators standardised, and no systematic,
collective planning of operations and work took
place. Finally, due not least to protracted per-
sonal conflicts between the participating ex-
perts, it was not possible to establish an effi-
cient, direct communication structure and cul-
ture between the FC and TC projects.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 103
6.4 Utilisation of GTZ's knowledge
management methods
The evaluation reports under review are not
very informative with regard to GTZ knowledge
management. The majority of the reports make
no mention of it at all. Most of the few evalua-
tions that do broach the subject (Africa su-
praregional, Algeria, Peru) confirm that the
staff of the respective project or programme
took part in the regional sector networks and
staff conferences, where they presented the
results of their projects/programmes to the
respective working groups. Participation in the
sector networks is generally described as a
worthwhile instrument of exchange and infor-
mation management. On the other hand, all
reports qualify that assessment to the extent
that an actual exchange of technical informa-
tion in the sense of best-practice dissemination
to the benefit of the projects and programmes
rarely takes place. The single positive example
of relevance is the dissemination of strategic
planning and monitoring instruments devel-
oped within the scope of the 'Drinking Water
and Sanitary Programme, Peru' by the sector
network and the staff conference.
Indeed, the subject of knowledge management
is given very scant treatment within the pro-
jects and programmes. On this point, the
evaluators see major potential for improve-
ment, but their reports offer no concrete ap-
proaches or proposals.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 104
7. Contract and cooperation man-
agement
7.1 Modes of delivery
Primarily in connection with efficiency ratings
for the reviewed projects and programmes, the
mode of delivery played an important role in
the relevant evaluation reports. As a specific
issue, the modes of delivery employed in all
fifteen evaluated projects and programmes are
significant in that twelve of the pro-
jects/programmes involved more or less exten-
sive inputs from consulting firms acting on
behalf of the GTZ. In one case (Niger), GTZ
International Services (GTZ-IS) delivered ser-
vices as a KfW contractor.
Nevertheless, the reports make little reference
to the respective mode of delivery. Often, the
service delivery structures and/or the involved
organisations/companies are not even named.
The following table provides an overview of
contractors and subcontractors involved in the
individual projects and programmes. It should
be noted in this connection that the information
provided was obtained by way of supplemen-
tary research, because the documentation in
the reports was incomplete.
Table 24: Contractors and subcontractors involved in the evaluated
projects/programmes
Country Project/
Programme Contractor / Subcontractor
Africa su-
praregional NWI GTZ
Algeria PDEA GTZ / AHT
Yemen WSP GTZ / GOPA, GFA, Dorsch Consult
Jordan WMIA GTZ
Kenya WSRP-KEN GTZ / GFA
Peru PROAGUA GTZ / RODECO, AKUT
Zambia WSRP-ZMB GTZ / RODECO
Viet Nam WDPC GTZ / GFA
Ghana EVORAP GTZ / RODECO
Kosovo WSWDP GTZ / RODECO
Mexico GICA GTZ / RODECO
Ethiopia IWSTR GTZ / GFA, RODECO
India IGWDP GTZ / RODECO
Niger PHV/MI GTZ (TZ) – GTZ-IS (FZ)
Turkey QADI GTZ
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 105
In their reports, the evaluators were very re-
served in their assessment of service delivery,
particularly in those pertaining to the interim
evaluations. For example, the interim evalua-
tion of the 'Integrated Water Management
Programme, Algeria' noted that, due to the
large share of external experts, the evaluator
assignments were not optimally adjusted to the
needs of the programme. Problems were per-
ceived primarily in terms of inadequate qualifi-
cations (language proficiency) and difficult
incorporation into the programme structures
(flow of information, steering, consultation). In
the ex-post evaluation report on the 'Im-
provement of Water Supply in the Tigray
Region, Ethiopia' project, service delivery by
the consulting firm commissioned for the first
phase of implementation was described as
inadequate, but no concrete reasons were
stated.
Apparently, the 'Drinking Water and Sanitary
Programme, Peru' yielded good experience
concerning the integration of qualified national
consultancy firms for the area of long-term
organisational consultancy. Particularly with
respect to process orientation and sustainabil-
ity in the support of institutional change proc-
esses, continuous national-evaluator assign-
ments are regarded as more advantageous
than the seconding of external evaluators.
In the interim evaluation report on the 'Institu-
tional Development of the Water Sector,
Yemen' programme, the award of a consider-
able contract volume to outside consultants
was not always regarded as appropriate and
efficient due to the high attendant fixed costs
and commitment of funds. While the pro-
gramme did attempt to be economically effi-
cient by awarding block contracts for self-
contained tasks and components, hence avoid-
ing fragmentary, cost-intensive micro-
management, this approach harbours the risk
that the requisite networking with German DC
institutions may not take place on a sufficient
scale and that the knowledge of procedures
could remain inadequate.
The network of consultancy services from the
TC and FC portfolios is regarded as being in
general need of improvement. For most parts,
the joint planning of interventions and the har-
monisation of procedures are still problematic.
Moreover, delayed outflow of funds due to the
formulated joint achievement of objectives has
a negative influence on both the effectiveness
and the efficiency of the TC measures.
The 'Rural Water Supply in Maradi, Niger'
project constitutes a special case in this con-
text. Here, the FC consultancy services were
contracted directly from GTZ IS. The TC advi-
sory services were rendered by the GTZ's own
personnel. Due to substantial 'static' between
the FC and TC personnel, this project never
reached the point of appropriate joint action.
This was primarily due to steering problems
that could not even be rectified within the GTZ.
As a result, major conceptual deficits rendered
the measure unsuccessful, obviously due to
the steering problem.
7.2 Steering and cooperation with
partner institutions
The greater share of the evaluated projects
and programmes intervene in a very complex
partner structure. In many places, institutional
responsibilities for sub-sectors and/or compo-
nents are spread over a number of different
sector authorities. The reviewed evaluations do
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 106
not explicitly deal with the resulting steering
challenges in individual cases. Apart from
three projects/programmes operating within the
SWAp context, clear-cut steering mechanisms
are the exception. The challenge for TC actors
is to achieve expedient, institutionalised net-
working between the participating partner insti-
tutions. Often, this requires 'negotiations' con-
cerning strategic orientation.
Relevant weaknesses result from the fact that
most of the projects and programmes have a
strongly established focus on the meso level.
Their influence on the evolution of steering
structures remains small, and numerous
evaluators perceive it as a constraint. Here, the
observed weaknesses in the implementation of
systematic capacity development interventions
impede progress precisely at the upper results
level (meso to macro level).
The basic question to be identified from the
evaluation reports is 'Who should actually be
doing the steering?'. Here, the huge demands
being placed on the partner-structure person-
nel cannot be met at present by the corre-
sponding counterparts. Hence, one of the main
future tasks should be to promote steering
competencies and capacities by means of an
explicit multi-level approach. As recommended
by many of the evaluators, some important
steps in that direction should be taken at the
planning stage. A more systematic and critical
analysis of the strengths and weaknesses of
the respective partner structures should be
taken as a point of departure for mainstream-
ing capacity development approaches into the
project concept from the very beginning. In
addition, the partner structure should only have
to contend with a scope of tasks that it can
actually 'absorb'.
The factor 'confidence building', which is a
major topic in the evaluation reports, entails
maximum possible closeness to the partner,
also in the physical sense. A number of pro-
jects and programmes still have some potential
for further optimisation in that respect.
What is referred to as the institutionalisation of
monitoring, including the documentation of
results beyond quantitative parameters, pro-
vides a further basis for effective steering of
complex projects and programmes. Some
good examples of useful approaches to moni-
toring were described in the preceding sec-
tions. Nevertheless, most of the reviewed pro-
jects and programmes remain open to criticism
with regard to inadequate or poorly sustained
monitoring systems. In that respect, the infor-
mational content of the reviewed evaluations is
limited. Monitoring serves not only purposes of
project steering and accounting, but also and
above all else for initiating learning processes
from which, ultimately, strategic and concep-
tual adjustments can emerge. Results, lessons
learned and best practices should be recorded
and evaluated together with the partners. Part-
ners who receive too little or no such informa-
tion cannot fully discharge their steering man-
dates.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 107
8. Summary of conclusions and
lessons learned
8.1 Summarising conclusions
As part of the 'independent evaluations' in-
strument, the performed cross-section analysis
of the evaluations in the thematic priority area
'water' should make a valuable contribution to
discharging GTZ's accountability obligations. It
does not amount to an overview of a selected
group of projects and programmes considered
representative of the GTZ water portfolio, but
comprises exemplary evaluations which in their
entirety cover a very broad sector-specific
spectrum. Nevertheless, the compiled results
can provide information of interest for steering
and learning.
The analysed evaluation reports are not al-
ways fully comparable with each other, and
they had different methodological and content-
related priority areas. Each evaluation report
constitutes a discrete, very subjective assess-
ment and analysis of the respective context.
Nevertheless, some general conclusions can
be drawn and taken as a basis for further dis-
cussion.
Apart from a summary of performance as-
sessments, this synthesis report offers some
important points of reference particularly with
regard to the methodological orientation of the
evaluated projects and programmes and their
attention to the principles and concepts of
German DC: poverty relevance, consideration
of the principles of sustainable development
and systematic furtherance of performance
capabilities within the partner structure. These
are major cross-cutting themes into which the
evaluation can yield new insight.
Still, the ability of a cross-section evaluation to
describe and evaluate the concrete scope and
content of service delivery in the evaluated
projects and programmes remains limited.
Very little pertinent information is contained in
the individual evaluation reports. The synthesis
report therefore confines itself to compiling
some typical, readily understandable examples
of successful and unsuccessful implementa-
tion.
Analysis of the fifteen evaluations yields a
'successful' broad-brush picture of the imple-
mented projects and programmes. The analy-
sis shows good relevance of the development
measures to be the main success factor. At the
same time, it becomes obvious that the docu-
mentation of results achieved in the selected
group of projects and programmes is still too
poorly developed. There is reason to suspect
that the TC projects and programmes actually
achieve many more results than can be identi-
fied and recorded with the existing range of
instruments. This could be one reason for the
merely 'satisfactory' assessment of objectives
achievement. Another problem is the still rather
weak sustainability of the achieved results. On
this point, the evaluators see a crucial connec-
tion between the measures' strong meso-level
focus and the existing weaknesses in the
structuring of systematic capacity develop-
ment. In the presented sub-portfolio, system-
building and networking interventions are still
too marginal.
Most of the evaluated projects and pro-
grammes lack the kind of clear orientation
towards poverty reduction that is called for by
the BMZ sector strategies and the Millennium
Declaration, among others. This is not neces-
sarily due to false conceptual design of the
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 108
projects and programmes, and it does not
mean that results with regard to poverty reduc-
tion are not achieved. The projects and pro-
grammes could, however, more plausibly por-
tray this orientation by more closely analysing
the target groups and profiling them in the
system of objectives. A measure's contribution
to poverty reduction and to the achievement of
MDGs can be more plausibly substantiated
through the formulation and systematic moni-
toring of target group-oriented, pro-poor indica-
tors. A poverty analysis would help illuminate
the existing opportunities for achieving direct
results – opportunities that should be made
use of by targeted activities.
Important lessons learned, as a synthesis of
conclusions drawn in the individual evalua-
tions, were compiled in the respective parts of
the DAC criteria analysis (see Sections 3.1 –
3.6) to form 'success factors' and 'factors of
failure'. The following section recapitulates the
most important conclusions and lessons
learned from the evaluation reports.
8.2 Lessons learned from the
individual evaluations
8.2.1 Relevance of the development
measures
With regard to the 'relevance' of development
measures, it proves to be of decisive impor-
tance that the respective conceptual design
and formulation of objectives achieve a maxi-
mum degree of concordance with the policies
and strategies of the respective partner coun-
tries and partner institutions. At the same time,
the thematic priority areas of the BMZ in the
water sector should also be covered by the
conceptual design, and systematic orientation
towards sector-specific policies, country
strategies and position papers (basic develop-
ment-policy orientation of the German Gov-
ernment) should be established. It is also very
important, especially for technical planning,
that regard be paid to international standards
and conventions.
The 'relevance' rating ultimately reflects the
amount of priority attached to the respective
measures both by the partners and by the
target group. In that connection, one very es-
sential criterion is whether the target groups
and partner institutions participated in the
planning of the relevant project or programme
and whether their needs were systematically
incorporated into the concept. Experience
drawn from the present sequence of evalua-
tions shows that very many development
measures are still being designed without a
proper target-group analysis or differentiation.
This does not only impair the relevant orienta-
tion of the projects and programmes, but also
fails to build a basis for identifying and re-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 109
cording the dimensions of the results at the
target-group level.
Concerning the requisite poverty relevance of
water sector projects and programmes, poverty
analyses need to be performed at the planning
stage and taken as a basis to establish poverty
reduction as part of the formulation of objec-
tives in accordance with the poverty reduction
strategies. Projects with no – or no verifiable –
poverty orientation and no underlying poverty
analysis rarely achieve good relevance. In
equal measure, consideration of the gender
dimension and environmental aspect in the
project design, on the basis of appropriate
investigations and/or analyses, is a crucial
success factor.
With regard to water resources management, a
recognisable attempt to embed that area in
existing or planned regional cooperation has
proven to be a critical success factor for the
relevance of the respective projects and pro-
grammes. Integrated and cooperative ele-
ments should be 'evident', and transboundary
initiatives should give holistic consideration to
the region in question.
The nature of a multi-level approach is also
considered to be very important. The extent to
which it can actually be implemented as
planned is decisive. Precisely the implementa-
tion of interventions at the macro and micro
levels is a critical success factor. The present
evaluation reports show that implementation at
just those levels is difficult and that too few
interventions are planned for the target-group
level. This can lead to conflicting objectives in
connection with impacts on poverty.
8.2.2 Effectiveness of the develop-
ment measures
The basic prerequisite for the successful
achievement of the objectives of development
measures is a shared understanding of objec-
tives and cooperation between the partners
and the experts entrusted with implementation.
It is important that both the partner experts
(counterparts) and the respective target group
understand and accept the measure's benefits
and that they exercise their rights and carry out
their duties.
It is also important in this connection for TC
and FC in cooperative projects to be optimally
interlinked, so that synergy effects can be gen-
erated and the results of TC measures can be
amplified by FC investments, and vice-versa.
Experience to date shows that this is best
achieved in projects and programmes with
formalised cooperative relationships, whereas
it is also important that the coordination of
interventions by DC actors be accompanied by
the harmonised use of resources. As a rule,
this requires the transparent (also for the part-
ner) and generally accepted awarding of a
steering mandate to one of the institutions
included in the cooperative structure.
In view of the dynamic political and institutional
settings within which the projects and pro-
grammes are implemented, it has proven es-
sential with regard to effectiveness that guide-
lines governing the cooperative implementation
and best-practice dissemination be formulated
together with the partner structure and formally
recognised. This includes adapting and further
developing the technical concept within the
course of the implementation phase, if this may
become necessary.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 110
With regard to the creation of durable capaci-
ties and structures at the implementing-partner
level, the formulation of transparent capacity
development strategies geared to system-
oriented results has emerged as an essential
success factor. In this respect, the national
water policies and sector strategies often de-
fine the general guidelines. The scope, quality
and coherence of the offered package of train-
ing and capacity building must be made trans-
parent to the partners. It is also important to
enable better coordination of supply and de-
mand by way of systematic needs assess-
ments.
Simultaneously, the duration of the advisory
services and, above all, due allowance for
follow-up assistance phases count as crucial
factors for successful achievement of objec-
tives in TC projects and programmes.
8.2.3 Impact of the development
measures
The present evaluations show that the rating of
overarching development results in the major-
ity of the projects and programmes only ap-
pears as an approximation based on assump-
tions and abstractions via lengthy results
chains. It was determined in that connection
that the aggregated results to be achieved by
the respective project or programme are often
unrealistic: excessive expectations (level to
which the objectives are to be achieved, very
long results chains). In formulating systems of
objectives, it is therefore important to plausibly
substantiate the measure's contribution to
overarching results and to formulate the over-
arching results with no room for interpretation.
Overarching results were successfully docu-
mented in projects and programmes in which
not only qualitative assumptions but also quan-
titative statements (perhaps based on a base-
line study) could be made. In that respect, it is
regarded as important that a contribution be
made to poverty reduction and/or to MDGs and
that the target groups confirm in surveys that
there has been a qualitative improvement in
their situation and/or their living conditions.
Of decisive importance in that same context
are such structural and systemic results as the
decentralisation of WS&S and/or the estab-
lishment of successful NGOs at the target-
group level and the further pursuance of sector
reform beyond the level of objectives achieve-
ment (positive effects in other areas of the
broader sectoral and regional context). It is
also important for role-model functions to
emerge and for the communicated approach
and concepts to become common property.
8.2.4 Efficiency of the development
measures
The crucial success factor for the efficiency of
the evaluated projects and programmes was to
embed DC interventions within the sector in-
terventions as a whole. Importance was at-
tached both to coordination between the do-
nors themselves and/or the German interface
organisations and to complementary appor-
tionment of working areas. Systematically util-
ising relevant potentials and synergies at the
various levels is most promising when the cor-
responding cooperation arrangements are
formalised. Most notably, the harmonisation
and coordination of resource utilisation be-
tween the German DC organisations has fre-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 111
quently remained inadequate or poorly man-
aged.
Beyond this, the personnel concept must be
convincing, i.e. the services of internal and
external short-term and long-term experts must
be coordinated with each other, and synergies
must be fostered through well-developed
knowledge management. The complexity of
the personnel concept must have no negative
effects on the cost-effect ratio. Where neces-
sary, preference should be given to alternate
modes of delivery (such as calling in qualified
local consultants or NGOs).
Another critical success factor that came to
light is the initiation, in the course of implemen-
tation and after having been approved by the
BMZ, of any necessary adjustments to the set
objectives or conceptual design, a typical case
being the non-provision of investment re-
sources in cooperative projects (outgoing
funds problems in FC).
8.2.5 Sustainability of the develop-
ment measures
Sustainability is seen as an important criterion
for performance assessment, also in the
course of implementation. It is decisive that all
objectives dimensions (economic, social, politi-
cal and ecological) be incorporated into the
conceptual design of the respective measures
in a timely and well-balanced manner, and that
structural risks be addressed in further-
reaching interventions.
The design of systematic capacity develop-
ment approaches of a structure- or system-
building nature lays the foundation for sustain-
able achievement of objectives. This includes
establishing participatory structures at the tar-
get-group level (micro level), hence enabling
the target groups to participate in dialogue
concerning the use of water resources. On this
point, acceptance on the part of the lead exe-
cuting agency must either already exist or be
brought about. Indeed, with regard to the train-
ing of partner personnel, there appears to be a
'critical mass', that is, training a sufficient num-
ber of personnel in the sector counts as a suc-
cess factor.
One other crucial success factor for develop-
ment measures is the lead executing agency's
willingness to accept advice in connection with
higher political objectives (macro level) and to
implement new laws and regulations governing
such things as break-even tariffs.
Durably effective operational structures for
major supply systems in poor settings can only
be built up slowly and with an accordingly large
volume of advisory inputs. The targeted supply
standards can be raised within a relatively
short time through investments and/or techni-
cal interventions. Institutional advances in de-
velopment alone rarely allow sustainable op-
eration as soon as the technical goals have
been achieved. In many of the evaluated pro-
jects and programmes, this problem poses
risks for the sustainability of achieved objec-
tives. In this same connection, time require-
ments and limited influence on handed-down
modes of sociocultural behaviour in the use of
water resources are of utmost importance.
Prevailing framework conditions make it virtu-
ally impossible to synchronise the achievement
of objectives in terms of technical supply ca-
pacities, establishment of operating structures
and changes in the behaviour of the target
group. Nevertheless, such synchronisation is
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 112
frequently adopted as a basis for the achieve-
ment of objectives.
The reviewed evaluation reports call attention
to the fact that too little time is allowed for es-
tablishing or decentralising capacities, per-
formance capabilities and/or change proc-
esses. It proves successful in this respect to
lengthen the advisory cycles in order to provide
process support and to include follow-up assis-
tance phases in the planning (particularly for
FC involvement).
Sustainable achievement of objectives also
relies on institutionalised knowledge and
methods at the implementation structure and
target group levels. It is therefore important to
promptly document lessons learned from the
projects and programmes and to integrate
them into the partner organisation's knowledge
management. In this connection, examples of
best practice can be compiled and dissemi-
nated.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 113
9. Recommendations
9.1 General recommendations
The results of this cross-section evaluation of
independent evaluations in the thematic area
'water' show that evaluation as an instrument
can make a relevant contribution to the con-
ceptual design, steering and quality assurance
of TC projects and programmes. Of decisive
importance in this respect is that evaluations
and their results be taken into account together
with project progress reviews for the steering
of ongoing projects and programmes within the
GTZ and for planning new ones. It is recom-
mended that more space be devoted to the
recording and analysis of relevant cross-cutting
development themes in the evaluations. Cru-
cial lessons learned, particularly with regard to
concepts and methods, could be drawn from
this 'qualitative' part of the evaluation reports, if
it were more strongly emphasised and system-
atically processed than was the case in the
current evaluation sequence.
Since independent evaluations are only per-
formed in two thematic areas each year, there
will be no comparable evaluation sequence in
the water portfolio within the near future. It is
therefore recommended that key issues from
the evaluation be integrated into routine project
progress reviews. In the opinion of the evalua-
tors, closer attention to reflections on the DAC
criteria impact, efficiency and sustainability, in
particular, together with a critical examination
of the cross-cutting development themes cov-
ered by the PPRs, would greatly improve the
steering and planning bases. It is also recom-
mended that additional ex-post evaluations be
performed for projects and programmes of
strategic importance in order to intensify the
lessons learned. Especially the more recently
designed, complex sector projects and pro-
grammes should be able to yield key insights
into the achievement of objectives and results
and about successful methods and ap-
proaches.
The results of the evaluations also reveal that
the understanding of cross-cutting develop-
ment themes, their importance within the Ger-
man DC context and their underlying concepts
and guiding principles is still very heterogene-
ous. These cross-cutting themes should be
given more attention in 'day-to-day implemen-
tation' and, to some extent, also in the planning
of measures. The GTZ‟s knowledge manage-
ment tools can make a valuable contribution,
e.g. by addressing cross-cutting themes more
pointedly in sector networks and at staff con-
ferences.
The key performance monitoring instrument for
all projects and programmes is substantiation
of their results orientation in line with the re-
spective conceptual design. On the basis of
the reviewed evaluations, it became evident
that determination of results in the 'day-to-day
implementation' of projects and programmes
has been something of an exception to the
rule. Moreover, the results of projects and pro-
grammes are also still seen to be playing a
very subordinate role in the established form of
reporting. Against that backdrop, it would be
advisable to bring the instrument of project
progress reporting more closely in line with the
requirements of results-based performance
monitoring. Simultaneously, project progress
reporting should include a portrayal and dis-
cussion of methods, approaches and models
and perhaps disseminate them as best prac-
tices. In the view of the evaluators, the TC
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 114
profile could be further sharpened by incorpo-
rating successful approaches (e.g. the capacity
development strategy, best advisory practices,
identification of lessons learned) into reports
for clients.
On the basis of findings from the evaluation
reports on which this cross-section evaluation
is based, and from the corresponding offers
submitted to the BMZ by the GTZ, the system
behind and coherence of assigning DAC
markers is seen to be in need of improvement.
The appropriate assignment of markers is es-
sential for classifying and evaluating the re-
spective projects and programmes. It is there-
fore recommended that the assignment of DAC
markers be more systematically aligned with
the objectives systems formulated for the pro-
jects/ programmes. In order to reduce the ex-
isting discrepancies, the respective relevance
should be checked and reassessed as neces-
sary in connection with evaluations and ad-
justments to plans.
Regarding capacity development, it is recom-
mended that a more uniform definition be ob-
served for purposes of routine implementation
and evaluations.
The success factors identified in Section 3.1.1
for the relevance of a project or programme
should, as far as possible, be given full consid-
eration when adjusting ongoing pro-
jects/programmes and in the conceptual de-
sign of new ones. In the evaluators‟ opinion,
these factors could also be retroactively inte-
grated when adjusting ongoing pro-
jects/programmes.
9.2 Recommendations for steering
ongoing
projects and programmes
For ongoing projects and programmes, too, the
extent of agreement between the formulated
objectives systems and the respective underly-
ing classification of projects and programmes
according to the DAC markers should be sys-
tematically investigated. This could be done in
connection with PPRs. The decisive factor
should be a current assessment of the realistic
(achievable) results of the respective pro-
ject/programme, which usually deviates from
the assessment made when the offer was writ-
ten. Classification of the projects and pro-
grammes according to the DAC criteria and the
adjustment of indicators for recording the re-
sults should then be based on that assess-
ment.
Critical reflections on the actual results levels
and objectives dimensions of a given project or
programme are recommended. If any discrep-
ancies are noted between the underlying con-
ceptual designs and the 'realities of implemen-
tation', appropriate adjustments should be
made. It is recommended to attach special
priority to the consistent implementation of a
multi-level approach and to alignment with the
principle of sustainable development.
It is also recommended that the existing sys-
tem of indicators be simplified and supple-
mented with measurable results factors. Some
examples of comparatively uncomplicated
approaches were already given in the preced-
ing sections. Here, too, the monitoring systems
may need to be adjusted and possibly stan-
dardised accordingly. Next, in the opinion of
the evaluators, it is of decisive importance that
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 115
the project/programme reporting include refer-
ence to the results factors. It is recommended
that GTZ's sectoral structure request or moni-
tor appropriate reporting. Internal resources
(personnel, available best practices) should be
used to provide advice on adjusting and stan-
dardising monitoring systems. Here, the GTZ
knowledge management system may serve as
a basis.
As far as the capacity development strategies
of projects and programmes are concerned, it
is recommended that transparent capacity
development concepts be elaborated and har-
monised, and adjusted to the available re-
sources of the respective projects and pro-
grammes. Such concepts should be consis-
tently aligned with the existing/adjusted objec-
tives definitions and, if necessary, focus more
on system-building and networking interven-
tions in order to meet the requirements of ca-
pacity building in the partner structures. In this
area, too, it is recommended to make use of
internal technical expertise for delivering advi-
sory services.
In addition, all existing means should be em-
ployed to achieve further formalisation of exist-
ing cooperative relationships (German and
international DC) within the context of the pro-
jects and programmes, as this provides a very
valuable basis for enhancing both the pros-
pects of success (mainly in terms of efficiency)
and the results of the measures. The goal
should be to achieve more real 'cooperation'
beyond the signing of corresponding agree-
ments.
9.3 Recommendations for planning
new projects and programmes
The analysis and/or differentiation of target
groups and of the poverty, gender and envi-
ronmental dimensions should be systematically
taken into account before actually designing
and planning projects and programmes. In the
opinion of the evaluators, this calls for specifi-
cations to be made by the sectoral structure
and for adjustment of the planning steps. It is
also recommended in that context to ask more
insistently for the relevant studies. In this re-
spect, the recommendations from the individ-
ual evaluations can be summarised as follows:
Poverty analyses should identify target
groups and correspondingly target group-
appropriate activities that contribute to
poverty reduction;
Especially in the world's poorest coun-
tries, the main focus should be on poverty
reduction;
Poor sections of the population should be
more closely integrated into political proc-
esses;
The decentralisation of outputs should not
focus merely on economic aspects, com-
mercialisation and privatisation, but also
on sociocultural aspects and pro-poor
outputs;
Tariff studies should pay particular atten-
tion to socioeconomic aspects and record
both the people's willingness to pay and
the affordability of the services provided;
The needs of poor people should be ac-
counted for in tariff systems;
The minimum quantity of water available
to poor people should be assured by such
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 116
means as cross subsidies, even if that
would have negative implications for the
revenue-to-cost ratio of the respective
supply structure;
In order to create a knowledge base,
socio-economic monitoring should be in-
tegrated into the measure as an institu-
tionalised mechanism for measuring the
contribution to poverty reduction and to
the achievement of the MDGs;
Cooperation with national poverty reduc-
tion programmes should be further
strengthened;
Such pro-poor concepts as water kiosks
and EcoSan should receive more atten-
tion in the 'choice of technologies'.
Regarding the consideration of wastewater
management and sanitation services, together
with declining interventions in rural areas, it is
the evaluators' opinion that the attention being
paid to the activity areas 'wastewater man-
agement' and/or 'sanitation' and to rural areas
per se should be re-examined for new projects
and programmes. Both of those priorities are of
major significance for the orientation of Ger-
man DC.
Systematic incorporation of a multi-level ap-
proach into the conceptual design of new pro-
jects and programmes also appears important.
In this respect, it is recommended that lessons
learned from the reviewed evaluation se-
quence be utilised and that more attention be
paid to the 'deconcentration' of service delivery
at the meso level. The goal here should be to
pay more attention to the micro level in line
with pursuing a holistic approach and with
achieving results at the target group level. At
the same time, the realistic macro-level results
should also be determined.
In the formulation of objectives systems for
new projects and programmes, attention
should be paid to simplification (small number
of indicators) and to the relevance to results
(by recording measurable factors). It is rec-
ommended in this context to formulate only
such indicators as are actually connected with
the planned outputs. Agreements with partners
should also and simultaneously help generate
a common understanding and shared respon-
sibility.
A transparent capacity development strategy
should be formulated at the planning stage and
agreed on with the partners. In so doing, any
foreseeable need for follow-up assistance
should be taken into account and the availabil-
ity of required resources established (planning
security). In the opinion of the evaluators, the
areas 'networking' and 'system development'
should be more closely incorporated into the
capacity development service package of pro-
jects and programmes in order to achieve
higher levels of 'effectiveness' and 'impact'.
Harmonisation and coordination with German
and international DC actors should also take
place during the planning phase in the interest
of achieving complementarity and synergies.
To the extent possible, concrete agreements
and perhaps jointly conceived design and
planning should be targeted. Important areas
like defining the scopes of responsibility, steer-
ing and resource provision should be agreed in
appropriately formalised cooperation. This
could be founded on concrete service com-
mitments that the partner institutions also find
transparent.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 117
To the extent that cooperative projects or pro-
grammes are planned, the contributions and
results that the partners in cooperation can
actually provide should be realistically as-
sessed at the planning stage. It is recom-
mended that this include an assessment of
existing risks (such as delayed outflow of FC
funds, TC personnel assignments, and contri-
butions from DED, CIM, BGR, etc.). Such risks
can, if necessary, be offset by adjustments at
some later date. It is recommended to seek
dialogue with the BMZ in this respect.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 118
Annex 1
Terms of reference for the synthesis report on water
I. Background and object of evaluation
The GTZ extended and optimised its evaluation system in 2005, the major innovation being the intro-
duction of independent evaluations on behalf of the GTZ. The Evaluation Unit commissions independ-
ent research institutes and consulting firms to conduct these evaluations in order to review the GTZ's
work from an external perspective. These institutes and firms in turn contract the services of one inter-
national and one national evaluator from the partner country in each case. Each year, the GTZ com-
missions a total of approx. 30 independent evaluations (interim, final and ex-post evaluations) in two
thematic priority areas. In 2008, the priority areas were 'decentralisation' and 'water'. Uniform evalua-
tion criteria and a uniform assessment grid enable comparison of the evaluations.
Each year, after the Evaluation Unit has examined and accepted the evaluation reports submitted by
the evaluators, it arranges for synthesis reports (cross-section evaluations) to be performed in each
priority area. These serve to promote institutional learning at the GTZ and to report to the German
Federal Ministry for Economic Cooperation and Development (BMZ). The results contribute to knowl-
edge management at the GTZ and can also be used for PR activities.
The object of the present synthesis report is:
16 evaluation reports from 2008 on the thematic priority area 'water':
8 interim evaluations (Africa supraregional, Algeria, Yemen, Jordan, Kenya, Peru, Zambia,
Viet Nam)
3 final evaluations (Ghana, Kosovo, Mexico)
5 ex-post evaluations (Ethiopia, India, Niger, Tanzania, Turkey)
II. Objectives of the synthesis report
The evaluation is intended to:
summarise the results of the 16 individual evaluations
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 119
identify recurring strengths and weaknesses or rather factors determining success and failure;
clarify whether identified deficits, if any, are the result of omission or of neglected documenta-
tion
identify cross-project lessons learned and recommendations
III. Tasks
To evaluate and process the 16 evaluation reports from 2008 in the thematic priority area 'water' and
to subject them to quantitative and qualitative comparison.
The questions to be answered result from the following report structure:
Contents
Executive summary
1. Introduction
1.1 Background, objective and object of the synthesis report
1.2 For the reader's guidance
2. Objectives and activity areas
Each of the evaluated projects and programmes is to be briefly summarised.
Key questions of relevance:
o Which are the main objectives pursued by the projects/programmes?
o Over the course of time, can trends be established in the objectives systems of the
evaluated projects and programmes?
3. Rating according to DAC criteria (See definition and key questions for DAC criteria in Annex 3
[GTZ Guidelines on Evaluating the Success of Projects and Programmes])
3.1 Relevance
3.2 Effectiveness
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 120
3.3 Impact
3.4 Efficiency
3.5 Sustainability
In each of these areas:
Rating (average and distribution) and weighting (average and distribution) of the criterion in the
reports
Reasons for weightings that deviate from the standard (factor 2)?
Were those reasons convincing?
Reasons for good and poor ratings (frequency!)?
Can 'patterns' be identified from them, i.e. factors determining success or failure?
For effectiveness and impact: At which levels are the results situated (institutional change, per-
capita water/sewage connections, water quality, wastewater disposal, legal framework, etc.)?
Are there any correlations between the ratings of individual DAC criteria?
Conclusions
3.6 Synopsis of 3.1 to 3.5
Overall rating (average and distribution) in the reports
In how many cases did (poor) ratings of individual criteria lead to downgrading of the overall rating?
The ratings assigned in the individual evaluations are binding for the analysis (sections 3.1 to 3.6). In
other words, the ratings must not be altered within the framework of this synthesis report.
4. Rating of cross-cutting development themes
4.1 Poverty reduction and MDGs
4.1.1 Conceptual problems
4.1.2 Mainstreaming in development strategies (in the field)
4.1.3 Poverty marker in the offers and its distribution
Poverty markers of development measures (distribution)?
To what extent do/did the development measures comply with these markers?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 121
4.1.4 Target-group differentiation and poverty analysis
Was the project design differentiated according to target groups, and was a poverty analysis
performed? Were poverty analyses used for the project design?
In how many development measures is/was an 'adequate' poverty impact achieved?
Did the project promote the participation of poor people in economic and political processes?
Does the project have positive results in terms of poverty reduction?
Does the project help to remedy structural problems identified by the national poverty reduction
strategy?
Were questions of equitable distribution (e.g. access to land?) addressed, and if so, how? (was
not part of the ToR)
How many people have benefited from the poverty impact and/or will do so in the future?
What are the reasons for achieving and not achieving poverty impacts? Can 'patterns' be identi-
fied, i.e. factors determining success or failure? Are there findings that are specific to given re-
gions?
Conclusions
4.2 Gender equality
What is the distribution of gender markers for the development measures?
To what extent do/did the development measures comply with these markers?
G-1 and G-2: Was the project design gender-responsive, and was a gender analysis (gender-
based analysis of the initial situation) available?
Was the G-0 marker awarded on the basis of a gender analysis? (Note: This became obligatory
for development measures at the end of 2005.)
Are there consistent examples of the mainstreaming of the gender aspect in 'water' projects and
programmes?
How many development measures have achieved an 'adequate' gender impact?
Do men and women make equal contributions to shaping the project/programme?
Do men and women derive equal benefit from the project/programme?
Does the project/programme help to reduce structural gender discrimination and thus to achieve
a positive change in the gender ratio?
Was one gender discriminated against or was this made up for by positive measures?
What is the ratio of direct to indirect results?
What are the reasons for achieving or not achieving gender impacts? Can 'patterns' be identi-
fied, i.e. factors determining success or failure? Are there findings that are specific to given re-
gions?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 122
Conclusions (also in terms of how gender-responsive aspects can be mainstreamed into 'water'
measures).
4.3 Effects on the partner's capacity to act (capacity development)
In how many development measures are/were (very) good / satisfactory / inadequate successes
achieved in terms of capacity development, and at which levels (individual, institutional, societal)?
Did the advisory services have an effect on the partner's capacity to act?
Did it prove possible for pilot approaches to be successfully multiplied or adjusted by the partner
structures?
Did the development measures have an effect on the design of national policies and laws, and
on the stakeholders' ability to shape such policies and laws? And on interaction between the
state, the economy and civil society?
What are the reasons for successful/unsuccessful capacity development? Can 'patterns' be
identified, i.e. factors determining success or failure? Are there findings that are specific to given
regions?
Conclusions
5. Concept of sustainable development
To what extent and how frequently did/do the development measures, i.e. the GTZ's working
method, reflect the concept of sustainable development (holistic, process-oriented and value-
oriented approach)?
Holistic approach: Linkage of economic, social and ecological objectives dimensions; of sec-
toral, organisational and policy advice; of the micro, meso and macro levels
Process-oriented approach: help for self-help: establishing transparency of actors' interests;
promoting interaction between the state, civil society and the private sector
Value-oriented approach: Promoting democracy, the rule of law, human rights; gender equality;
good governance; social and ecological market economy
Conclusions
6. Sectoral assessment
6.1. How do the German interface organisations work together and engage in
task-sharing ('joined-up DC')?
Additional explanations
Who is involved in the provision of water-sector advisory services? To what extent are planning and
implementation performed jointly? Do the combined efforts of the German interface organisations yield
the anticipated synergies in the sector? Can expedient task-sharing take place, or does obstructive
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 123
overlapping occur? How is interface management effected? How does cooperation rate? How does
the partner see this? How much value can be added by cooperative projects/programmes? Where has
FC investment actually provided added value to the development measure, and where would the de-
velopment measure have remained unsuccessful without it? Are the FC flanking measures comple-
mentary, or to what extent do their contents overlap? To what extent could the content of the FC flank-
ing measures also have been implemented by TC? Which concrete contribution has TC made towards
consolidating the sustainability of FC investments?
6.2. How are the development measures embedded in the activities of the international donor
community?
Additional explanations
How far has cooperation between donors progressed? Which coordination measures have been taken
regarding advisory content and task-sharing? Is a sector-wide approach in place? Which concrete
impacts is this having on the work of the GTZ? Does this lead to common strategies, steps of planning
or perhaps even cooperation in the implementation of advisory services? What is the donor commu-
nity's assessment of ongoing cooperation? What do the partners think of it?
6.3. To what extent has it been possible to implement the multi-level approach in the provision
of advisory services, i.e. to utilise synergies by providing the services simultaneously at
the international and transnational levels, at the national level (ministries) and at the re-
gional and local levels?
The multi-level approach is being increasingly included in the planning of TC projects. Its purpose is to
design national policies, strategies and laws in a solution-oriented and activity-oriented way and, con-
versely, to gain impetus for the reform or revision of national framework conditions via local or regional
implementation. 'Multi-level approach' also means greater harmonisation and coordination efforts. In
complex development measures, the various levels are sometimes split up among various German
implementing organisations and, primarily in Africa, numerous international development measures
which, ideally, should help develop the respective national water sector.
How is the multi-level approach being realised in the project, and which exchange mechanisms have
been introduced between the advisors at the different levels? How is utilisation of the results at the
respective advisory level ensured by the other levels? How can experience and findings from local
processes, pilot measures and partial solutions be transferred to the national level and mainstreamed
in a systemic overall context?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 124
6.4. How are GTZ's knowledge-management methods being applied to the updating and qualita-
tive improvement of the range of advisory services?
How did the switch to sector networks or staff conferences lead to the dissemination of advisory ser-
vice contents and instruments?
Which concrete ideas or incentives are available for further developing the range of knowledge man-
agement instruments in the water sector?
How, and in which concrete cases, have improved learning processes been introduced into the part-
ner structures?
7. Contract and cooperation management
Which 'modes of delivery' were appropriate and why?
Was TC the appropriate instrument? (Differences between anchor countries and poor coun-
tries?) Are preconditions for programme-based approaches (e.g. SWAp's) in place?
Which particular strengths and weaknesses of project steering occurred (frequently), and what
are your conclusions?
How was cooperation with the partner institutions rated? Which particular strengths and weak-
nesses occurred (frequently)? Can any recommendations be made?
How were the cooperative relationships and the quality of cooperation in DC/IC rated? (coop-
eration and delimitation between FC/TC, joined-up DC, donor coordination and the role of the
GTZ, requirements stemming from the Paris Declaration)
Which particular strengths and weaknesses occurred (frequently)? Do you have any recom-
mendations about how to deal with the weaknesses?
8. Summary of conclusions and lessons learned
9. Recommendations
(from the above sections 3. - 7.)
Total length of report: approx. 70 pages
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 125
Annex 2: (Standard) Terms of reference for individual evaluations
I. Background
Independent evaluations are conducted either during the term of a development measure, on its com-
pletion, or in the form of an ex-post evaluation between two and five years after the promotion has
ended. The evaluations are managed by the Evaluation Unit. As a rule, the unit commissions inde-
pendent research bodies, institutes affiliated to universities or consulting companies to perform the
evaluation. These in turn place an international evaluator and a local evaluator under contract, who
jointly evaluate the development measures. In addition to their own sector-specific expertise, the local
evaluators are also able in particular to add the perspective of the respective partner country.
Each year, a total of 30 independent evaluations are performed in various regions. This annual ran-
dom sampling addresses certain thematic priority areas in order to subsequently formulate compara-
tive and overarching key points. The 2008 priority areas are decentralisation and water. The evalua-
tion described here is part of the thematic priority area water.
II. Object of the evaluation
[Brief description of the context in the country]
[Brief description of the development measure including the prior history and any continuation within
the framework of a programme, etc.]
III. Objective of the evaluation
The evaluation aims to rate the success of the development measure. The assessment is based on
five blocks of questions.
a) Assessment according to international evaluation criteria
The success of the development measure is to be described and assessed on the basis of predefined
evaluation criteria (based on facts and figures): relevance, effectiveness, impact, efficiency and sus-
tainability (see also Annex 3: Guidelines on evaluating the success of projects/programmes).
b) Assessment of the development measure with regard to poverty reduction and millennium
development goals
The assessment of the development measure should include an appraisal of the extent to which it
verifiably contributes to poverty reduction and achieving the MDGs.
Key questions of relevance:
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 126
Was the conceptual design of the development measure differentiated according to target
group, and was a poverty analysis performed?
Did the development measure promote the involvement of poor people in economic and politi-
cal processes?
Does the development measure have positive impacts on poverty reduction?
Does the development measure help overcome structural problems identified in the national
poverty reduction strategy?
c) Assessment of the development measure with regard to the promotion of gender equality
The assessment of the development measure should include an appraisal of the extent to which it
verifiably contributes to the promotion of gender equality.
Key questions of relevance:
Is the conceptual design of the development measure gender-responsive, and was a gender
analysis performed?
Do men and women contribute equally to the conceptual design of the development measure?
Do men and women draw equal benefit from the development measure?
d) Assessment of the development measure with regard to the promotion of sustainable de-
velopment
The assessment of the development measure should include an appraisal of the extent to which it
verifiably contributes to the promotion of sustainable development.
Key questions of relevance:
Does the development measure take a clearly holistic approach (linking of economic, social
and ecological objectives dimensions; technical, organisational and policy advisory services;
micro, meso and macro levels)?
Does the development measure take a process-oriented approach (help for self-help; estab-
lishing transparency of actors‟ interests; promoting interaction between the state, civil society
and the private sector)?
Does the development measure take a value-oriented approach (promotion of democracy,
rule of law, human rights; good governance; social and ecological market economy)?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 127
e) Sector-specific performance assessment of the development measure
The performance assessment of the development measure based on the aforementioned criteria must
include, in particular, a factually substantiated assessment on the basis of specific thematic questions.
Additional explanations of the various questions should be included in italics.
These are:
BMZ sector strategy, regional and country strategies
1. To which extent are sector strategies, regional and country strategies helpful for combining
and innovatively structuring advisory services for the water sector?
Additional explanations
Do these papers provide new, content-specific impulses? If so, which? Are they helpful for partner
dialogue? If so, how? Do they perhaps limit content-related sectoral development? If so, with respect
to which themes? Do the sector strategies make statements about gender equality that enter into the
advisory services? 'Sanitation' for example: This theme is mainstreamed in most strategy papers as
well as in the sector strategy. Does this make it easier to broach the subject with the partners? 'Irriga-
tion', for example: This branch of advisory services is not (or no longer) included in many country
strategies. Does that force the advisory inputs to move in a less effective direction (considering that
agriculture accounts for the major share of all water consumed)? Or does this amount to a practical
thematic focus? Why?
2. Do the 'water' development measures contain the 'Integrated Water Resources Manage-
ment (IWRM)' approach called for by, among others, the BMZ water sector strategy? How is
this approach or strategy implemented in concrete terms?
Additional explanations
IWRM demands attention to various sectors that have to do with water. This serves to uncover con-
flicts of use and identify common strategies, the objective being to maximise the (social, economic,
ecological) benefits. This, in turn, requires the involvement of various user groups and the monitoring
of negotiation processes. In concrete water-sector terms, this means that not only the actors them-
selves, e.g. those concerned with drinking water supply, but also other water users, e.g. the agricul-
tural, energy and/or industrial sector(s), be involved in the dialogue. The same applies to those who
will be affected by the results of water use (downstream riparians / ecosystems / tourism / fishing
grounds / fisheries, etc.). Are poor people involved as well? Which distribution effects does the devel-
opment measure have? Which good and which less successful examples are to be found in the GTZ
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 128
'water' development measures, and how was IWRM actually implemented? In other words, were any
institutions established and supported, or any methods and procedures initiated and established that
provide anchorage for the IWRM negotiation processes at the national or regional level? Were women
involved in these IWRM negotiation processes? How were the needs of female water users accounted
for within the IWRM? How could other sectors and/or water users be included in water resources
planning? Where and how, if at all, did the development measures encounter limits? Does the devel-
opment measure also address sub-sectors of relevance to the country's water sector (e.g. water sup-
ply, wastewater management, rural, urban, water resources management, irrigation)? How is experi-
ence transferred between the sub-sectors, within the development measure itself, and within the
scope of support for the policy discussion in the partner country? Which synergy effects exist between
the contents and results of the advisory / extension services in the sub-sectors?
Cooperation and task-sharing
3. How are the development measures embedded in the activities of the international donor
community?
Additional explanations
How far has cooperation between donors progressed? Which coordination measures have been taken
regarding advisory content and task-sharing? Does a sector-wide approach exist? Which concrete
effects does this have on the work of the GTZ? Does this lead to common strategies, steps of planning
or perhaps even cooperation in the implementation of advisory services? How is the progressive co-
operation being rated by the donor community? How do the donors rate it?
4. How is cooperation and task-sharing being handled by the German interface
organisations (‘joined-up development cooperation’)?
Additional explanations
Who is involved in providing advisory services in the water sector? How much joint planning and im-
plementation is taking place? Do the anticipated synergies emerge through the combined efforts of the
German interface organisations? Can expedient task-sharing be performed or does obstructive over-
lapping occur? How is interface management effected? How is cooperation rated? How do the part-
ners rate it? How much value is added by cooperative projects/programmes? Is the personnel and
financing constellation in accordance with the objectives of the development measure? Where have
investments in FC created genuine added value in the development measure, and where would the
development measure have been unsuccessful without it? Are the FC flanking measures complemen-
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 129
tary, or to which extent does content-related overlapping take place? To which extent could the FC
flanking measures have been implemented by way of TC? Which concrete contribution has TC made
toward securing the sustainability of FC investments?
Characteristic features of GTZ:
Direct contribution, capacity development, multi-level approach
5. Have the characteristic features of GTZ advisory services (direct contribution and capacity
development) been successful or not?
Additional explanations
The work of GTZ is characterised by 'direct contribution' as a form of advisory services and by its fo-
cus on 'capacity development'. The latter means that processes of change are supported by appropri-
ately adopted and diverse measures in order that people, organisations and societies can mobilise,
maintain, adjust and expand their capacities for sustainable development. What were the most impor-
tant existing framework conditions for successful capacity development in the individual service pack-
ages of the development measure? Which approach was successful with regard to interaction be-
tween stakeholders and reconciliation of conflicting interests? Where are the blockages? Where and
how did the development measure visibly contribute (via capacity development) towards good govern-
ance? Did the development measure, within the framework of capacity development, build capacity for
action for achieving gender equality as a major prerequisite for good governance and sustainable de-
velopment?
6. To what extent has it been possible to implement the multi-level approach in the provision
of advisory services, i.e. to utilise synergies by providing the services simultaneously at
the international and transnational levels, at the national level (ministries) and at the re-
gional and local levels?
Additional explanations
The multi-level approach is being increasingly included in the planning of TC projects. Its purpose is to
design national policies, strategies and laws in a solution-oriented and activity-oriented way and, con-
versely, to gain impetus for the reform or revision of national framework conditions via local or regional
implementation. 'Multi-level approach' also means greater harmonisation and coordination efforts. In
complex development measures, the various levels are sometimes split up among various German
implementing organisations and, primarily in Africa, numerous international development measures
which, ideally, should help develop the respective national water sector.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 130
How is the multi-level approach being realised in the project, and which exchange mechanisms have
been introduced between the advisors at the different levels? How is utilisation of the results at the
respective advisory level ensured by the other levels? How can experience and findings from local
processes, pilot measures and partial solutions be transferred to the national level and mainstreamed
in a systemic overall context?
Results of GTZ development measures for water
7. What are the essential results of the evaluated German DC development measure in the
water sector?
Additional explanations
How can the results of the development measure in the water sector be summarised? At which levels
are the results produced (institutional change, per-capita water and sanitation connections, water qual-
ity, wastewater disposal, legal framework, etc.)? What can be said about the cost-benefit ratio for the
achievement of certain results (e.g. cost of each sustainable water and/or sewer connection, cost of
institution promotion, cost of regulation, etc.)? Which contributions are being made to the MDGs (MDG
7 targets water supply & wastewater management; MDGs 5+6 health; MDG 1extreme poverty and
hunger)? (See item 2.3: overarching development results / Millennium Development Goals)
Which income groups are (predominantly) being reached)? To what extent are poor people / the dis-
advantaged / women being reached? Transboundary water management: How can/could it be
achieved that the adopted approach to institutional strengthening of transboundary water management
has direct, positive effects on the people? How are such results documented and communicated?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 131
Supply and demand side of service packages
8. On which themes should German DC in the water sector focus its future advisory services?
Additional explanations
Have there been any major shifts in the contents of the requested or required advisory services?
Which water-sector themes are generating heightened demand for advisory services and support, now
and in the future? (Since actors presently operating in the water sector were interviewed during the
evaluation, this question can also be posed for concluded development measures.)
Successful methods/instruments and knowledge management
9. Which innovative and/or replicable methods and/or instruments are being recommended
for dissemination and/or for broader application?
Additional explanations
Which concrete methods and/or instruments are available for dissemination in the region or in GTZ
water-development measures? (e.g. use of certain workshops, training modules, OE methods, etc.)
10. How are GTZ knowledge-management methods being used for updating and
improving the quality of the advisory services?
Additional explanations
How and in which concrete cases did the switch to sector networks or staff conferences lead to the
dissemination of advisory services and/or instruments? Which concrete ideas or incentives are there
for the further development of knowledge management tools for the water sector?
IV IMPLEMENTATION
a) Responsibilities
The Evaluation Unit is responsible for planning and managing GTZ's annual independent evaluation
programme. The subcontractor is institution X, who places an international expert and a national
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 132
expert under contract for the purpose of this evaluation. The subcontractor is responsible for conduct-
ing the individual evaluation, for quality assurance and for reporting on the evaluation.
b) Inputs
The subcontractor shall provide the following inputs:
peruse and evaluate relevant documents
draw up an 'inception report' containing the e-Val results and the facts, figures and data from
the beginning of the project/programme (as found in the appraisal report, baseline surveys,
and benchmarked indicators). Write the draft version of the inception report in German for
consultation with the Evaluation Unit, but draw up the final version – to serve as a working ba-
sis for the team of evaluators – in the lingua franca of the country in question. Submit the in-
ception report at least three weeks prior to the evaluator‟s departure. In special, well-founded
cases, an 'inception phase' for the country of assignment which precedes the evaluation may
be planned.
draw up the terms of reference for the national expert in the partner country
select and engage the national expert
have the international expert attend a one-day preparatory seminar about fundamental as-
pects of evaluation at the GTZ
prior to performance of the evaluation, have the selected international expert participate in lo-
cal preparatory meetings with representatives of the Evaluation Unit, the responsible regional
group and the Planning and Development Unit (P+D)
prepare, organise and perform on-the-ground surveys
meet with the German embassy and the local GTZ office, if possible prior to commencement
of the mission
document the preliminary results of evaluation and recommendations for the final presentation
in the project country
present and discuss the preliminary results of evaluation with the partners and local GTZ par-
ticipants prior to the international expert‟s departure.
draw up the report (containing the results determined by, and the recommendations made by
the international and national experts) in German (or, alternatively, by arrangement with the
Evaluation Unit also in English) within five weeks of return
attend an evaluation meeting at the GTZ
assess the national expert
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 133
draw up the final version of the evaluation report (comprising not more than 45 pages of sum-
mary and chapters 1 through 6, plus front pages, table of contents, list of abbreviations and
annexes; executed in MS Office 2003 as a PDF file; see specimen format for reports) in Ger-
man (or, alternatively, by arrangement with the Evaluation Unit, also in English) within three
weeks of the evaluation meeting, and generate a brief report (summary and tabular overview
of the main report, including an overview of the facts and figures of relevance to the assess-
ment of the project/programme results) in German and in the lingua franca of the given coun-
try
provide quality assurance for the reports and their foreign-language versions (inception report,
evaluation report and brief reports)
provide quality assurance for the organisation and implementation of the evaluation, as well as
technical follow-up assistance for the evaluators.
The GTZ shall provide the following inputs:
make the necessary documents available
hold preparatory talks with the subcontractor/institution and the evaluator, and host (including
organisation) a one-day preparatory seminar for the evaluators
announce and introduce the evaluation to the relevant partners and important contacts in the
country of assignment
provide (on request) logistical support through the local GTZ office (reimbursement of costs by
the commissioned institution in accordance with the contractual agreement)
organise and hold the evaluation meeting with the involvement of the responsible sectoral and
regional divisions and, in the case of cooperative projects/programmes, the other development
cooperation organisation
approve the inception report and the evaluation report (incl. foreign-language executive sum-
mary).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 134
Annex 3: Guidelines on evaluating the success of projects/programmes
1 Preliminary remarks
This document is a guide to evaluating the success of projects/programmes implemented by GTZ. It
contains the central evaluation criteria, the key questions devised to examine the criteria, and an as-
sessment grid devised to obtain an overall rating.
The evaluation criteria, which were developed in a working group operating under the name
“Evaluierung aus einem Guss”117
and are binding for its members, are based on the five criteria
agreed within the OECD-DAC2:18
relevance, effectiveness, efficiency, overarching development-policy
results (impact) und sustainability.
Furthermore, key questions were devised which represent important points of reference for assess-
ment of the evaluation criteria.
The rating of a project/programme according to the individual evaluation criteria is given on the basis
of a six-point scale (exception: a four-point scale is used for the sustainability criterion).
Explanation of terms:
Development measure: The term “development measure” used in this document encompasses the
following terms: project, programme, program-based approaches, and development intervention.
Results: The term “results” used in international debate and in this document includes the outputs, the
use of outputs, the direct and indirect results and the highly aggregated results.
17 A group of the Evaluation Departments of BMZ (lead), GTZ, KfW, DED and InWEnt created to harmonise and coordinate
evaluation activities.
18 OECD-DAC: Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development
(OECD), Paris
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 135
These guidelines are also based on the rationale and concepts of the GTZ results model:19
The GTZ results model
Basis of the assessment
The evaluation of a development measure requires systematic examination of the underlying cause /
effect-hypotheses. This examination is carried out in the following stages:
The quality of the results chain of the development measure is investigated and as-sessed
(including rationale for the assessment).20
This includes, for example, examining the plausibil-
ity of the cause / effect-hypotheses and the level to which the objectives are to be achieved.
(formulation of objectives and indicators)
If the cause / effect-hypotheses and the objectives of the development measure are judged to
be consistent, they are taken as the basis for the evaluation of the pro-ject/programme.
19 See also “Guidelines on Offers”, GTZ November 2006 4 The objectives and indicators defined in the offers are binding and
are applicable here, not the information given in any other project documentation.
20 The objectives and indicators defined in the offers are binding and are applicable here, not the information given in any other
project documentation.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 136
If the cause / effect-hypotheses and the objectives of the development measure are judged
not to be consistent, the evaluators develop their own hypotheses, objectives and indicators
which appear to them to be more appropriate. These then form the basis for evaluating the
success of the project/programme. General remarks on assessment
All that is assessed is the development-policy effectiveness of the project/programme as defined in the
evaluation criteria.21
What is assessed is primarily the situation as it can be measured or established at the time of the
evaluation. Exceptions are explained in Section 2 in the descriptions of the relevant evaluation criteria.
Results that are difficult to estimate at the time of the evaluation must be plausibly inferred and de-
scribed using proxy indicators or by verifying central assumptions about the results chain.
2 The five evaluation criteria
2.1 Relevance Are we doing the right thing?
The extent to which the objectives of the development measure match the needs of the target groups,
the policies of the partner country and partner institutions, the global development goals and the Ger-
man Government‟s basic development-policy orientation.
Key questions for assessing relevance22
21 The overall success of the development measure is not to be equated with the quality of the work performed by GTZ and its
implementation partners. For example, good management on its own does not necessarily lead to overall success.
22 The relevance of the project/programme is assessed as of the time at which the conceptual approach was formulated or
adjusted. A concept adjustment that was not carried out, despite being necessary, is also relevant to the assessment, how-
ever.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 137
To what extent is the development measure suitable for addressing central development is-
sues for the partner country and target groups (differentiated according to gender, ethnic
group, parties to a conflict), and specifically poor population groups?
To what extent is the development measure in compliance with the (sector) policies and
strategies of the partner country (national plans, PRSP etc.) and the partner institutions?
To what extent is the development measure in compliance with international themes, stan-
dards, conventions?
Does the project or programme have a high or low priority for the responsible institutions in the
partner country or for civil society? How is this apparent?
Does the development measure comply with the client‟s basic development-policy orientation?
23
Which of the cross-cutting issues are relevant? How far are these taken into consideration in
the design of the project or programme?
To what extent is the development measure targeted at poverty reduction and the MDGs?
23 Currently these are: poverty reduction, fostering gender equality, participatory development and good governance, environ-
mental protection and conservation of natural resources, crisis prevention, combating drug abuse, rural development, and
conservation of tropical forests. See also BMZ homepage: http://www.bmz.de/en/index.html
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 138
To what extent does the project or programme fit into the country strategy, priority area and
programme definition, and the sector strategies?
To what extent is the project or programme in line with GTZ‟s concept of sustainable devel-
opment, i.e. to what extent does it contribute – through a holistic project approach, through
process orientation and through value orientation – to ensuring that the results of the pro-
ject/programme and the developments in the partner country can be sustainable?24
2.2 Effectiveness
Are we achieving the objectives of the development measure?
The extent to which the intended direct results (objectives) of the development measure are being
achieved (comparison of actual situation with targets).
Key questions for assessing effectiveness25
To what extent are the objectives of the development measure being achieved (comparison of
actual situation with targets on the basis of the defined indicators)?26
Are any limits critical to
success for achieving the objectives being undershot or overshot?
What are the decisive reasons why the objectives are or are not being achieved?
To what extent do the political partners (lead executing agency in the partner country and
BMZ) and the implementing organisations (national implementation partners and GTZ) have a
positive or negative influence on the achievement of objectives?
What unintended positive and negative direct results have occurred? How should these be
assessed in the overall context? What is the response to these?
24 See also GTZ homepage: http://www.gtz.de/en/top-themen/15534.htm
25 When evaluating ongoing projects or programmes, the appraisers assess the effectiveness in terms of the degree to which
the objectives have been achieved at the time of evaluation, i.e. whether the project/programme is within the objectives corri-
dor. In final and ex-post evaluations, the effectiveness is assessed on the basis of the degree to which the objectives were
achieved at the end of the project or programme.
26 The assessment is made on the basis of the objectives and indicators that are considered appropriate; see also the explana-
tions on the “Basis of the assessment” on Page 2 of this document.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 139
2.3 Overarching development results (impact)
Are we contributing to the achievement of overarching development results?
The extent to which the project or programme contributes to achieving the intended over-arching re-
sults and produces other indirect results.
Key questions for assessing overarching development results (impact)
What (positive and negative) changes at the level of indirect results can be observed in the
wider sectoral and regional environment of the development measure?
Which of these changes can be plausibly attributed to the project/programme at various levels
(population, sector, institutions and regulations)? For example: 27
How far are important barriers to development being structurally eliminated or reduced
through the project/programme?
How are the living conditions or development opportunities changing?
How is the project/programme strengthening the ability to solve problems of the target
groups(of the project/programme and others), intermediaries and institutions?
What contributions is the project/programme making towards achieving overarching
development goals (e.g. Millennium Development Goals and implementing the Millen-
nium Declaration, structural reduction of poverty - e.g. promotion of pro-poor growth,
pro-poor governance)?
What results are being achieved in terms of other cross-cutting issues: gender, envi-
ronmental protection and conservation of natural resources, participation and govern-
ance, crisis prevention and conflict sensitivity?
What contributions can the development measure realistically make to the achievement of
these indirect results (expected level to be achieved by the project/programme)?
27 Depending on the individual circumstances, the examples listed here for indirect results may also be direct results, in which
case they have to be taken into account under the “effectiveness” criterion. In order to decide whether a result is direct (“ef-
fectiveness” criterion) or indirect (“impact” criterion), the question has to be answered whether achievement of the result can be
attributed causally to the development measure (direct result) or whether the development measure is making plausibly explica-
ble contributions to the achievement of the result (indirect result).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 140
How is the actual contribution of the development measure to the achievement of these indi-
rect results assessed?
What are the decisive reasons why indirect results are or are not being achieved?
To what extent is the effectiveness of the development measure positively or negatively af-
fected by other policy fields, strategies or interests (German ministries, bilateral and multilat-
eral development partners)? What consequences has the project/programme drawn from this?
What broad impact is being achieved: e.g. suitable as a model, structural changes, replication
of approaches and scaling-up?
2.4 Efficiency
Are the objectives being achieved cost-effectively?
A measure of the degree to which the resources invested in a development measure are appropriate
compared to the outputs and results achieved.
Key questions for assessing efficiency
What resources is the development measure using for the various forms of modes of delivery
(long-term experts, short-term experts, procurements, operating and administration costs,
training courses, local subsidies and financing contracts, other contributions)?
To what extent is the structure of the development measure (e.g. project, programme struc-
ture, program-based approaches) appropriate for achieving the outputs and results cost-
effectively? What structure might make it possible to do this more efficiently?
To what extent are the objectives and outputs/activities of the project or programme coordi-
nated with or complementary to those of others, or designed for task-sharing (joined-up Ger-
man Development Cooperation, program-based approaches such as SWAp, basket funding or
budget support)?
To what extent is there adequate coordination between donors? In what ways is this encour-
aged or hindered?
To what extent is the composition of the modes of delivery - personnel concept and personnel
assignment, provision of materials and equipment, training, funding - cost-effective, i.e. how
far are internal resources being used efficiently?
What alternatives are there to the modes of delivery employed, and which (if any) would be
more efficient?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 141
To what extent have the outputs been produced cost-efficiently (cost-benefit ratio)? What al-
ternatives are there, and which (if any) would be more efficient?
To what extent are the direct and indirect results achieved efficiently in terms of microeco-
nomic, macroeconomic and sector-specific standards?
Are the outputs and results achieved at the correct time and within a reasonable period?
2.5 Sustainability
Are the positive results durable?
A measure of the probability that the positive results of the development measure will continue beyond
the end of assistance.
Key questions for assessing sustainability
What period is appropriate/realistic for the continuation of the results, and which critical mini-mum
requirements for success in this period (expectations of sustainability) are appropriate for the pro-
ject or programme?
What approaches, instruments, methods or concepts are lastingly used, institutionalised or further
developed by the target groups, partner institutions or other actors? How is this done?
How will the results for the target groups, partner institutions and partner country continue be-yond
the end of assistance? How far can the (direct and indirect) results of the project/programme be
maintained, or is an improvement or deterioration to be expected?
To what extent are the requirements for sustainability met?
To what extent are (organisational, personnel, financial, economic) resources and capaci-
ties available in the partner country (in the longer term) for maintaining the results
achieved? To what extent is the required ability in place to adapt to changing frame-work
conditions and solve associated problems?
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 142
How should the four dimensions of sustainability28
(economic, political, social and ecologi-
cal) and their risks and opportunities be assessed?
How do these dimensions of sustainability interact? Under the specific conditions, to what
extent is the result balanced, stable and capable of modification in the longer term?
What are the key risk factors for longer-term sustainability of the results? How is the evolution of
these factors assessed?
Operationalisation of the four dimensions of sustainable development to assess sustainability:
Economic: how are the achieved outputs and results secured at the microeconomic or
macroeconomic level? How stable or capable of modification are they in terms of the
pace of economic growth (local, regional, national, global)?
Political: how is the project or programme contributing to a fair and peaceful balance
of interests? To what extent are changes in the political culture and changes in behav-
iour, attitudes and awareness apparent in the target groups and partner institutions?
To what extent and in what way is ownership on the part of target groups and institu-
tions necessary and existent? To what extent is the legal framework in place that is
necessary for changes? To what extent is the political will for change identifiable?
Social: how is the project or programme contributing to greater equality of opportunity,
social justice, improved access to social services and resources?
Ecological: in what way is long-term ecological viability ensured? What risks are there
of lasting negative environmental effects emerging in the long term?
3. Assessment grid
3.1 Assessment of the individual evaluation criteria
28 In specialist debate it is common to distinguish between three dimensions: economic, social and ecological sustainability. To
provide a more differentiated description, in this document social sustainability is broken down into political sustainability (socie-
tal level) and social sustainability in the narrower sense (individual level).
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 143
A six-point scale is used to rate the project or programme according to four of the criteria, namely
“relevance”, “effectiveness”, “overarching development results” (impact) and “efficiency”. The scale is
as follows:
1 very good rating, significantly better than expected
2 good rating, fully in line with expectations, no significant defects
3 satisfactory rating, falling short of expectations but with positive results dominant
4 unsatisfactory rating, significantly below expectations, and negative results dominate despite
identifiable positive results
5 clearly inadequate rating: despite several positive partial results, the negative results clearly
dominate
6 the project/programme is useless, or the situation has deteriorated on balance
A rating of 1-3 indicates a positive assessment, a rating of 4-6 an assessment that is not positive.
The “sustainability” criterion is rated on the following four-point scale:
1 (very good sustainability)
The overall success of the project/programme (positive to date) will continue unchanged or
even increase with a high degree of probability.
2 (good sustainability)
With a high degree of probability, the overall success of the project/programme (positive to
date) will only minimally decrease but will overall remain significantly positive (normal situation
– to be expected).
3 (satisfactory sustainability)
The overall success of the project/programme (positive to date) will decrease significantly but
remain positive with a high degree of probability. This also applies if the sustainability of a pro-
ject/programme is assessed as inadequate up to the point of evaluation, but will evolve posi-
tively with a high degree of probability, so that the project/programme will achieve positive
overall success.
4 (inadequate sustainability)
The overall success of the project/programme is inadequate at the time of evaluation and
there is a high degree of probability that it will not improve. This also applies where the sus-
tainability has previously been rated positively but there is a high degree of probability that it
will decline seriously and no longer meet the requirements for level 3.
Overall rating
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 144
The overall rating is calculated from the assessment and a weighting of the five individual criteria
which is determined for each specific project. The rating is given on a six-point scale:
1 very good rating, significantly better than expected
2 good rating, fully in line with expectations, no significant defects
3 satisfactory rating, falling short of expectations but with positive results
dominant
4 unsatisfactory rating, significantly below expectations, and negative results domi-
nate despite identifiable positive results
5 clearly inadequate rating: despite several positive partial results, the negative re-
sults clearly dominate
6 the project/programme is useless, or the situation has deteriorated on balance.
An overall rating of 1-3 shows that a project or programme was successful, a rating of 4-6 shows it
was unsuccessful. However, projects/programmes can only be rated as “successful” if the direct
results (effectiveness), indirect results (impact) and sustainability are rated at least “satisfactory”
(3). 29
A decision is made and justified for each criterion whether it is “particularly important”
(weighting 3), “important” (weighting 2) or “less important” (weighting 1) in the specific con-text of
the project/programme. In the absence of a special reason which makes it more or less important,
a criterion is held to be “important” (weighting 2). Ratings and weightings are always expressed in
whole numbers, rounded up or down in accordance with the usual rules of mathematics. 30
29 13 In exceptional cases a rating of 4 for sustainability is acceptable for a “successful” project/programme if this has been
planned from the start and was inevitable for the project/programme and still appears acceptable at the time of evaluation be-
cause of the great development significance of the project/programme. In such instances the weighting for the “sustainability”
criterion must be “1”.
30 The Evaluation Unit has developed an Excel macro for calculating the overall rating; only the values for the rating and weight-
ing of the individual criteria need to be entered in the macro, which then calculates the overall rating.
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 145
lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
Annex 4: Rating of individual criteria and overall rating of 15 evaluated projects/programmes
Country Project /
programme Total Relevance
Effect-
iveness Impact Efficiency
Sustain-
ability Total
Africa
suprareg.
NWI 2 1 2 2 3 2 good
Algeria PDEA 3 4 3 3 4 3 satisfactory
Yemen WSP 2 2 2 1 2 3 good
Jordan WMIA 2 1 2 3 2 2 good
Kenya WSRP-KEN 2 2 2 2 2 3 good
Peru PROAGUA 2 1 2 3 2 2 good
Zambia WSRP-ZMB 2 2 2 3 2 3 good
Synthesis report – Independent evaluations in 2008 in the thematic priority area 'water' 146
Viet Nam WDPC 3 1 3 3 3 3 satisfactory
Interim evaluations 2.3 1.8 2.3 2.5 2.5 2.7 good
Ghana EVORAP 3 3 2 2 3 3 satisfactory
Kosovo WSWDP 2 2 2 3 2 3 good
Mexico GICA 3 2 3 3 3 3 satisfactory
Final evaluations 2.7 2.3 2.3 2.7 2.7 3.0 satisfactory
Ethiopia IWSTR 4 2 3 3 4 4 unsatisfactory
India IGWDP 2 1 2 3 3 2 good
Niger PHV/MI 4 3 4 3 4 4 unsatisfactory
Turkey QADI 4 3 3 4 3 3 unsatisfactory
Ex-post evaluations 3.5 2.5 3.0 3.3 3.5 3.3 satisfactory
All evaluations 2.7 2.0 2.5 2.7 2.8 2.8 satisfactory
Deutsche Gesellschaft für Technische Zusammenarbeit (GTZ) GmbH – German Technical Cooperation – Dag-Hammarskjöld-Weg 1-5 65760 Eschborn/Germany T +49 61 96 79-1408 F +49 61 96 79-801408 E [email protected] I www.gtz.de