Experiences in Impact Evaluation:The PEMA Perspective
Mandate
PEMA (Centre for Project Evaluation & MacroeconomicAnalysis) was established in January 2003 as anoperating unit of the Ministry of InternationalCooperation
While a basic mandate is impact evaluation, this mandate hasbeen extended during the past three years to:
a) Help upgrade the M&E capacities in key ministries
b) Coordinate the M&E reports prepared by various U.N. agencies inEgypt to assess progress made annually in achieving theobjectives under the United Nations Development AssistanceFramework (UNDAF)
c) Contribute to OECD efforts to survey progress towards better aideffectiveness
PEMA undertakes evaluations of development projectsin order to:
Ascertain their impact on target beneficiaries and thesustainability of such impact
Draw lessons learnt from evaluation and put these atthe disposal of policy and decision makers to assistthem in taking measures to improve the efficiencyand effectiveness of assistance received
PEMA has completed the evaluation of 31projects in various sectors
Current staff of the Centre compromisestwelve researchers and the executive director
Steps for undertaking impactevaluation
Once a project is selected, the following steps areundertaken:
Desk study of documents
Propose evaluation questions and methodology
Data gathering (fieldwork)
Content analysis
Draft report writing
Roundtable meeting is held to receive stakeholders’ comments
Final report is sent to stakeholders and to Minister ofInternational Cooperation, whereby follow-up actions take place
Findings on impact evaluationsin various sectors
Educational Sector
Girls’ access to primary (basic) education is enhanced by providingcomprehensive packages of community services
Projects implemented for improving teaching methods in primaryschools are not sustainable due to weak monitoring mechanisms
Vocational Education:
Businessmen (private sector) support is crucial for the success andsustainability of vocational education programs, particularly for fundingtraining in specialized areas.
Many of the graduates of vocational schools do not pursue a vocationalcareer as they enter universities to obtain academic degrees
Small and micro-enterprisedevelopment
Building the capacities of NGOs and CDAs is crucial for creatingefficient systems for operating micro-credit loans
High charges of loans and disregard of the grace period for loanrepayment impede the sustainability of SMEs
Insufficient follow-up surveys to determine the impact of loanson beneficiaries, and to identify measures that could furtherenhance sustainability and effectiveness of loans
Insufficient coordination between funding agencies results inthe implementation of several SME projects in the same areas,thus reducing demand for loans
Health Sector
Embedding project activities into the existing operationalstructure of health system is essential for ensuring sustainability
Systems for collection and analysis of data on health servicesare required for effective supervision and delivery of services
Highly centralized system of distribution of medical resources(medical staff and medical supplies) results in ineffectivedistribution
Low salaries and inadequate financial incentives lead toshortage of medical staff and health workers
Agricultural Sector
Insufficient public facilities discouragesbeneficiaries from settling in reclaimed areas
Insufficient irrigation water impedesapplication of modern farming practices insome of the reclaimed areas
Limited financial and technical capabilities offarmers associations after project termination
Lessons learned from impact evaluations
More emphasis should be given to project design:project objectives; outcomes; means of verificationand implementation strategy
Some projects do not set technical and financialmechanisms required for sustaining projectoutcomes
In some projects, responsible bodies do not establishnew units or appoint permanent staff to implementand monitor project activities
Capacity building activities should be relevant andbased on the needs of trainees
Setting criteria for selecting trainees and ensuringapplication of criteria are important for ensuringeffectiveness of capacity building activities
Executing bodies should have more flexible systemsof work by encouraging trained personnel to applywhat they learned from the project
It is crucial to establish budget lines within ministries forcapacity building activities
Bad management of project's time leads to implementation ofproject activities in the remaining time of the project. Therefore,activities are implemented at "less than acceptable" level.
A correct budget estimate for project activities is essential toavoid deficit or loss of funds
How do target beneficiaries perceive of the project and itsoutcomes is an important indicator of project effectiveness