Baseline and Final Evaluation (Combined) Cons

Combined Baseline and Final Evaluation Statement of Work

“Integrated Multi-sectoral Response to Critical Needs of IDPs and Returnees in Northern Ethiopia” / “Strengthening community resilience through addressing the critical multi-sectoral needs of returnees, IDPs and host communities in Ethiopia”

1. Evaluation Purpose

The purpose of this evaluation is to assess the performance and effectiveness of the project activities undertaken throughout the current award from USAID’s Bureau for Humanitarian Assistance (BHA) to achieve project outcomes and to establish baseline measurements for the new award. Endline measurement of outcome-level indicators will be integrated as part of the performance assessment. Endline results in project sites to be continued under the new award will also be considered as baseline measurements. The evaluation plan will also include data collection in new project sites under the new award to collect baseline measures in the new implementation areas. The evaluation team will also examine unintended consequences of the project interventions to inform future emergency assistance in Northern Ethiopia.

The overall objectives of this evaluation are

• To evaluate the project in terms of relevance and effectiveness, with a strong focus on comparing achievements against project targets,
• To collect baseline measurements under the new award,
• To identify key successes, gaps, and constraints to be addressed in future humanitarian programming and
• To document good practices within the current programs and to generate evidence-based lessons learned that can inform and strengthen the strategies of both ongoing and future programs.

1. Background Information

Activity Name (current award)

Integrated Multi-sectoral Response to Critical Needs of IDPs and Returnees in Northern Ethiopia

Implementer

Plan International USA

Period of Performance

July 22, 2021 – June 30, 2024

Active Geographic Region

Tigray, Amhara, Afar

Activity Name (new award)

Strengthening community resilience through addressing the critical multi-sectoral needs of returnees, IDPs and host communities in Ethiopia

Implementer

Plan International USA

Period of Performance

May 20, 2024 – May 19, 2025

Active Geographic Region

Tigray, Amhara, Oromia (Borena)

1. Evaluation Type

This is a mixed-methods, performance evaluation, primarily focused on assessing the extent to which outcomes were achieved and collecting baseline measurements for the new award. The evaluation will also examine the project’s implementation processes in terms of their gender responsiveness, inclusiveness and relevance to the local context.

1. Evaluation Questions

2. Performance: To what extent have the activity’s interventions adhered to planned implementation (e.g., schedules, participant targeting, service delivery, and outputs) and achieved intended outcomes?
3. Effectiveness: To what extent has the intervention appropriately assisted women, men, boys, girls, people with disabilities and other vulnerable groups affected by the conflict in Northern Ethiopia? How did the project prioritize and engage these groups in design and implementation?
4. Challenges and action taken: What were the key barriers or challenges to program implementation? How were they addressed?
5. Baseline status: What are the baseline levels of key performance indicators in participating communities under the new project award?

The evaluation team will aim to highlight best practices and lessons learned under questions 2 and 3, in terms of engagement strategies that worked well and innovative solutions to barriers that could be replicated and scaled for future interventions.

1. Evaluation Methods & Limitations

This evaluation will use mixed methods, including project document review, participant-based surveys, direct observations, key informant interviews (KII) and discussions with male and female focus groups.

Project Documents and Records to be Reviewed

As a first step in the evaluation, the following project documents will be reviewed for background, to assess performance, and to inform further refinement of primary data collection tools to address information gaps:

• Project award document (description of the activity)
• Performance indicator tracking table (PITT)
• Project reports (baseline, internal monthly progress, field visit, post-distribution monitoring, semi-annual performance)
• Detailed implementation plan
• Distribution records
• Accountability data from the project’s community feedback mechanism
• Health facility data from facility records and national Health Information Management System
• Regional Bureaus Reports (Health Bureau, Water Bureau, and Women and Children Affairs Bureau)

Quantitative Data Collection

The quantitative data will be collected using a household questionnaire for both child protection, WASH, health and nutrition indicators. The tool will be administered using an electronic mobile phone platform (Kobo Tool Box). The mobile application is chosen to ensure the quality of the data collected, improve data integrity, reduce human error during data entry, and ensure data security. The survey will include demographic questions to determine if a household has children 0-5 months, 6-23 months. If the household has a child in this age group, the enumerator will request to interview the mother of these children, as she would be best placed to provide accurate responses to questions related to indicators N08 and N09. The demographic questions will also screen for women of reproductive age. These women would then respond to the sub-questionnaire on dietary diversity to measure indicator N10. All other questions related to WASH and health indicators may be answered by the head of household or any representative of the household above the age of 18. The respondents for the child protection household questioners will be children within different age categories of 5-9, 10-14, and 15-18 years old with the questions and interview length adjusted for younger age groups.

Sampling strategy for child protection indicator CO2a: Two objectives influenced the planned sample size: achieving an acceptable margin of error (MOE) at project level but having no less than 20 observations in each stratum to make the data useful at regional level. A total sample size of 558 was thus established at the 95% confidence level, distributed over the three regions proportionally based on the size of each region’s client base. Each regional sample is equally distributed over six age-sex strata, namely boys aged 5-9 years of age, girls aged 5-9 years, boys aged 10-14 years, girls aged 10-14 years, boys aged 15-18 years, and girls in this age bracket. Borena zone of Oromia, which was previously under a separate BHA award will be integrated under the new award, with child protection activities continuing in one Woreda only. The sample size for this integrated zone for child protection baseline data collection will be finalized in discussion with the consultant.

Sampling strategy for WASH, nutrition, and health household survey: In the interests of efficiency, cost, and time, all questions related to the indicators under these sectors will be consolidated into one multi-stage, cluster sample using one survey tool with different sections for the household-level WASH data, the individual level nutrition data, and for the one question on health messages for which the respondent was alternated between an adult male in the household and an adult female in the household, given that the target population for the related activities is the general adult population. The downside to this approach is the risk of not achieving a satisfactory number of the individuals of interest for the nutrition questions, namely infants aged 0-5 months, young children aged 6 – 23 months, and women of reproductive age, 15 – 49 years given that they represent only approximately 1.5%, 6%, and 24% of the general Ethiopian population. The approach for dealing with this challenge is described below.

An overall sample size of 800 was calculated for results at the 95% confidence level using the following formula:

n = 4 (p)(1-p)DEFF/MOE2

Where: p = probability for the proportion of interest (established as 0.5)

DEFF = design effect (established as 2)

MOE = margin of error (established as 0.05)

The DEFF factor is required to compensate for the intra-cluster effect inherent in multi-stage cluster sample design.

The following steps are then followed to distribute the sample:

1. The sample size is distributed first over the three regions, proportional to size.
2. A mapping exercise is conducted to identify the zones and woredas in which WASH, nutrition, and health activities overlap.
3. Lists of the targeted kebeles and health facilities within the selected woredas, together with their population statistics, will be obtained to select two or three clusters with probability proportional to size. This will be done by cumulating the populations of the listed target kebeles and, after determining the number of clusters to be selected, an online random number generator was used to randomly select the first cluster.
4. The final step in the sampling process is to select households within the selected kebeles. This step will be assigned to supervisors to perform, working from kebele population registers, supplemented where needed by the records of Health Extension Workers (HEW). Supervisors, with the support of kebele leaders and HEWs will identify households with at least one woman of reproductive age and an infant 0-5 months or a young child aged 6-23 months. From these kebele lists, supervisors will randomly select the households for interviews through systematically sampling from the registers using the desired sample size to calculate the sampling interval.

Note that Afar Region is removed and Oromia Region (Borena zone) added under the new award. However, project activities in Borena will only cover child protection. Therefore, Borena does not need to be considered in the sampling plan for this multisector survey, and Afar data will only be considered in the endline analysis of the current award and excluded from the new baseline analysis.

Qualitative Data Collection

Qualitative data will be collected through focus group discussions (FGD) and key informant interviews (KII). The FGDs will be held with children and adolescents participating in child-friendly space (CFS) activities. The focus groups will be designed to measure indicator CO2.b (percent of children and adolescents participating in child-friendly space (CFS) activities who report an improved sense of safety and well-being at the end of the program). Selected HF staff, Government Bureau representatives and community leaders will serve as the key informants and be interviewed using a KII guide. The guide for each group of respondents focused on the overall performance of the project. The questions will include a critical reflection that allows respondents to mediate on both project success and challenges and to capture new knowledge and actions for the future. FGDs and KIIS will focus first on what worked well within the project and participants’ suggestions for improvements. These elements will be translated into a Strengths and Opportunities, Weaknesses and Threats (SWOT) Analysis which will inform and enhance the sustainability and development of future phases of the project. For example, strengths and opportunities could be leveraged and scaled in future iterations of the project, while weaknesses and threats could be addressed through adaptations to future project designs to mitigate those challenges.

Qualitative sampling: The sampling for the FGD will be a purposive selection of children in the age groups 5-9, 10-14 and 15-18 years old who have been enrolled in CFS and attended at least 8 CFS sessions to ensure diverse perspectives from children who have actively participated in the programs. The FGDs will have 8-10 participants, with separate focus groups for girls and for boys for each of the age groups. The discussion length will be shorter for the younger groups. The KIIs will be purposefully selected participants from health facilities, project staff and local stakeholders—such as community leaders, focal points, volunteers supporting implementation—based on their involvement in child protection, WASH, health and nutrition project activities. For feasibility, the FGDs and KIIs will take place in the same kebeles sampled for the quantitative surveys.

Analysis Requirements

Quantitative data will be analyzed with MS Excel and SPSS. The analysis will include descriptive statistics and comparison of results from baseline and endline. Qualitative data will be analyzed using qualitative software, such as NVivo or Dedoose. The analysis will use codes to identify and extract key themes, and illustrative (deidentified) quotes will be integrated into the presentation of results to complement and contextualize the quantitative findings. All results will be disaggregated by sex, age group, and region. The evaluation findings will be organized and presented by evaluation question, sector, and according to the reporting format stipulated by the BHA M&E guidelines.

1. Evaluation Timeline and Deliverables

The evaluation activities will be conducted from May 27 to August 11, 2024. Responsible parties are noted for the consultant; Plan USA Monitoring, Evaluation , Research and Learning (MERL) staff; Plan International Ethiopia (PIE) MERL and Country Office (CO) staff

1. Evaluation Findings Dissemination and Use

The final evaluation findings will be shared with the donor, Plan International’s country office, Plan International’s US office, local representatives in program implementation areas and other potential stakeholders, such as clusters or humanitarian agencies working in the same areas.

The method of dissemination may include the publication of the report in repositories or online libraries, such as ReliefWeb, Plan Internationals’ online knowledge management platform, and USAID’s Digital Data Library. In addition, the project team will hold results dissemination and reflection events both internally with staff in the organization (either in-person, online, or hybrid) and externally, with representatives and stakeholders at the regional, woreda and kebele levels as feasible.

Evaluation reflection events, particularly those held internally with staff, will focus on how the results can inform future programming, such as scaling up successful strategies, addressing weaknesses, refining program design, allocating resources effectively, enhancing sustainability, and improving monitoring and evaluation frameworks.

1. Evaluation Team Composition

This evaluation will be led by an external consultant with logistical support from Plan Ethiopia on the data collection activities on the ground.

Roles and responsibilities of the consultant

The consultant will lead the technical aspects of the evaluation, including

• Document review
• Refinement of data collection tools and sampling strategy
• Develop/update guidance for data collection and data quality considerations
• Provide technical support in facilitation of enumerator training
• Review pilot test data and support any further adjustments needed for tools
• Review incoming data during collection and provide progress reports and data quality issues to the field teams
• Data cleaning and analysis
• Report writing
• Presentation of results in an online/hybrid validation meeting with core project team (time permitting)

Roles and responsibilities of Plan International

Plan Ethiopia will provide the following support for the final evaluation:

• Review and propose adjustments to data collection tools
• Update digital tools according to any revisions needed
• Develop handout for enumerators to support local translation of key terms
• Hire, train and manage enumerators
• Logistics for enumerator training, pilot testing and data collection
• Providing relevant and clear information to the consultant for background, planning, regular updates during data collection and debriefing for data cleaning and interpretation of results
• Prepare local government stakeholders and community representatives for facilitating access and mobilizing communities and providing guidance to the team for implementing the evaluation activities on the ground
• Review report and provide feedback to consultant for any revisions needed
• Arrange logistics and invitations for participants to results validation and dissemination meetings.

Key qualifications for the consultant

The evaluation consultant should have the following skills and experience:

• A minimum of 5 years of evaluation experience, including substantive involvement in reviews at program and/or outcome levels in related fields with international organizations.
• Familiarity with the policies, programs and operating styles of the local and national governments that are participating in the project
• Understanding and sensitivity to the diversity of cultures, traditions and languages;
• Excellent proficiency in spoken and written English.
• Advanced knowledge and experience on gender and safeguarding issues
• Experience working with communities using participatory approaches
• Solid methodological and research skills

PI241502357

Apply Here

How to apply

Apply Here