2013 Indonesia: Ex-post Evaluation of WASH programme
Author: Preethi De Silva
"With the aim to continuously improve transparency and use of evaluation, UNICEF Evaluation Office manages the "Global Evaluation Reports Oversight System". Within this system, an external independent company reviews and rates all evaluation reports. Please ensure that you check the quality of this evaluation report, whether it is “Outstanding, Best Practice”, “Highly Satisfactory”, “Mostly Satisfactory” or “Unsatisfactory” before using it. You will find the link to the quality rating below, labelled as ‘Part 2’ of the report."
The Water Sanitation and Hygiene (WASH) Programme in six Provinces of Eastern Indonesia began implementation in June 2007 to June 2012. As the Government of Indonesia set a target of halving the proportion of people without sustainable access to safe drinking water and basic sanitation by 2015, WASH became a major priority.
One of the key reasons behind the increased attention for WASH is that there were an enormous number of people in Indonesia without access to safe water and sanitation facilities (about 50 million and 100 million people respectively ) with the well-known negative consequences for health and well-being, especially of children and women. As an example, the second largest killer of Indonesian children under five is diarrhea which is largely due to poor water and sanitary conditions.
The Programme was built on the government’s National Policy for Development of Community-based Water Supply and Environmental Sanitation and was expected to support the GoI, NGOs and communities to accelerate improvements in the WASH sector. The Goal of the Programme was to improve health, well-being and livelihoods in the project areas through improving hygiene practices and access to safe water and sanitation. The Programme consisted of 4 components with result targets:
> Component 1: Improved hygiene practices and access to safe water and sanitation, and in about 180 villages in 25 districts by the year 2010 targeting 320,000 direct beneficiaries.
> Component 2: Improved hygiene practices and access to water and sanitation among students and teachers of 500 primary schools in 25 districts and 5 urban areas by the year 2010.
> Component 3: Improved hygiene practices and access to water and sanitation among 70,000 slum inhabitants of five cities and towns by the year 2010
> Component 4: Strengthening institutional capacity of provincial and district governments to better plan, budget, implement and monitor WASH Programmes
The evaluation is planned to inform mid-term review of the country programme of UNICEF and Government of Indonesia taking place mid 2013.
The purpose of the evaluation is to draw the lessons learnt and identify factors contributing to the success or failure of planned results, to assess sustainability and draw concrete recommendations for improvement for the current and future WASH programmes. The evaluation will assess the relevance, effectiveness, efficiency and sustainability/replication of the program.
The primary users of the evaluation findings are GOI/National Development Planning Agency (BAPPENAS), Ministry of Education, Ministry of Health, Ministry of Public works and other relevant ministries, Government of the Netherlands and UNICEF Indonesia. Secondary users include regional and headquarters level in UNICEF and its partners.
The consultancy commenced with desk review and the Evaluation Consultant was guided by key evaluation questions given in the Terms of Reference. An Evaluation Matrix was prepared with sub-questions under each evaluation criteria. Three methods were used for collecting information & data; secondary sources such as bi-annual progress reports and other related technical reports, primary data from one-to-one meetings with key stakeholders and direct observations from field visits.
Subject to criteria like - (i) type of safe water system, (ii) continued presence of UNICEF in the location, (iii) availability of ex-programme staff, (iv) travel time/logistics and (v) level of programme success and failure - three provinces out of the six programme provinces were selected as study area. Two districts and four villages were taken in each province, except in Papua province where due to logistical constraints it was not possible to travel to Jayawijaya district and thus only two villages were taken from Jayapura district.
At the national level, meetings were held with BAPPENAS WASH Secretariat, the Embassy of Royal Netherlands, Ministry of Health, Community-led Total Sanitation/Sanitasi Total Berbasis Masyarakat (CLTS/STBM) Secretariat and Ministry of Home Affairs (Annex 4). At sub-national levels, meetings were held with Provincial/District Development Planning Agency (BAPPEDA) offices, health, education and public work departments in districts and provinces. In addition, a meeting was held with Head of Operations of CARE International Office in Makassar. In villages visited in five districts depending on the time available, interviews were held with heads of villages, heads of elementary schools, village water committees. Group discussions were held with school teachers, school children, women and men beneficiaries.
Findings and Conclusions:
> Relevance. The Programme is relevant as it was aligned with National Policy for Development of Community-based Water Supply and Environmental Sanitation and Target 10 of the MDGs.It is incorporated an equity approach in line with UNICEF’s renewed focus on equity in both design and implementation
> Effectiveness. People gained access to sustainable community-based water supplies through pipe water systems, rain water tanks and dug wells. There is increased knowledge on good hygiene practices, under the three components. Communities started constructing their own toilets in almost all villages. The elementary schools in the villages benefitted by receiving child-friendly WASH facilities and knowledge on hygienic practices among students. The student groups have adequately knowledgeable of hygiene practices and their appearance and body language reflected status of their health and commitment for practicing good hygiene practices.
> Efficiency. It is difficult to assess. Procurement delays due to central procurement by UNICEF prolonging construction period.
> Equity. The programme selected most disadvantaged districts, villages and urban slum areas in six provinces. All children in the school were given training without discrimination. Separate latrines were allocated for male and female students. Both male and female villagers took part in construction work.
> Sustainability. In urban slum areas pipe water systems were well built and sustained ownership and efficient management. The district departments of public works are currently replicating this community participation approach. Slow sand filters are also being currently replicated by the District Department of Public Works. However, rural pipe water systems and school sanitation facilities is threatened due to poor maintenance. There was no after-care or post-project programme for rural pipe water systems, therefore village water committees were not strengthened enough to administer and manage it.
1. Recommendations for Follow – up of the Programme
a) Successful, partial successes as well as failures intervention has to be documented in order to better understand the contributory factors for such success and failures for planning and better decision-making in the future by UNICEF.
b) Enhance Planning and M&E Capacity of AMPL Pokjas
c) Develop database management in districts and provinces in WASH sector
d) Establish Exit Strategy for Facilitators
e) Construct stock-taking Current Operational Status and Routine Monitoring of Performance of Pipe Water Systems
f) Begin Legalization of Village Water Committee/User Organization for receiving additional financial assistance from the Community Development Fund
g) Use of Education Block Grants (BOS) funds for Maintenance of School Sanitation Facilities.
h) Conduct Testing Water Quality to ensure supply of safe water, free from E.Coli Bacteria.
2. Recommendations for Similar Projects in the Future
a) Time bound realistic targets be set in STBM planning.
b) Prepare a Project Implementation Plan involving all stakeholders to ensure transparency
c) Establish project M&E system with required baselines and clarity and focus on planned results
d) Undertake a Project Appraisal or Evaluability Assessment before funds are committed, and an independent evaluations on completed projects.
e) Re-introduce Log Book at households for facilitating continuous and close monitoring, following up and close supervision in implementing STBM activities.
f) Use of Modified Kirkpatrick Model for evaluating training and capacity building programmes with follow up coaching and mentoring processes until expected results are achieved.
g) Plan and implement an "After-care Phase” or “Post-Project Operation Phase” for empowering and supporting village water committees and ownership by communities.
h) Introducing health and hygiene education as a subject in school curriculum since elementary level.
The importance of community participation approach adapted by the Programme for water supply was recognized by the government at sub-national level and the district departments of public water is following community-based approach in implementation of government annual work plans in WASH
Sand Filters introduced under the Programme to remove Iron from water proved to be a very successful intervention in Luwu Utara district and replication of the technology with government funds in the district with community-based approach is currently underway. Water supply systems in urban areas were a successful intervention under the Programme with a high degree of sustainability in Makassar and Kupang provincial capitals of South Sulawesi and NTT provinces respectively.
UNICEF favored decentralized procurement by district BAPPEDA offices over central procurement by UNICEF. This was due to the fact that as there had been delays in providing materials for construction affecting delayed implementation, as reported to the Evaluation Mission in South Sulawesi province by provincial and district BAPPEDA.
Full report in PDF
PDF files require Acrobat Reader.