AGECS Endline Evaluation in Mombasa and Kilifi

Introduction

The Aga Khan Foundation (AKF) is an agency of the Aga Khan Development Network (AKDN), a group of private not-for-profit, non-denominational international development agencies, institutions, and programmes that work primarily in the poorest parts of Africa and Asia to improve the quality of life of communities in remote and resource poor areas. In East Africa, AKF works with partners to improve the quality of life by promoting and developing innovative solutions to development challenges in Kenya, Tanzania, and Uganda. Reflecting the complex and multi-faceted nature of development, AKF programmes in the region encompass Education, Health and Nutrition, Early Childhood Development, Agriculture and Food Security, Climate Resilience, Economic Inclusion, and Civil Society. AKF is implementing a 33-month AGECS project which will promote gender equality by providing capacity strengthening for CSOs working on gender equality and women’s empowerment, including targeted support to women’s organizations.”

About the AGECS Project

Advancing Gender Equality through Civil Society (AGECS) is a 33-month (2022-2025) project implemented by the Aga Khan Foundation, with support from Global Affairs Canada (GAC), involving civil society partners in five countries in Africa -Uganda, Kenya, Tanzania, Mozambique, and Madagascar. In Kenya, AKF will partner with SAUTI YA WANAWAKE PWANI and Dream Achievers Youth Organization (DAYO) to implement AGECS project activities in Kisauni, Nyali, Likoni, Changamwe, Jomvu, and Mvita sub-counties in Mombasa County while in Kilifi county the focus will be in Kilifi South, Kilifi North, Malindi, Magarini and Ganze sub counties. The project seeks to implement innovative gender targeted and transformative initiatives that address barriers to gender equality in terms of community participation, decision-making opportunities, and access to resources and services for women, girls, boys, and men in the specific project geographies.

Purpose of the end-line evaluation

This end-line study aims to assess the overall relevance of the interventions in achieving the programme’s planned objectives. The consultant will be required to collect the end-line data on the indicators of the programme to measure the performance of the programme against the baseline study. The evaluation is expected to be conducted in both AGECS sub projects AKF will partner with SAUTI YA WANAWAKE PWANI and Dream Achievers Youth Organization (DAYO) to implement AGECS project activities in Kisauni, Nyali, Likoni, Changamwe, Jomvu, and Mvita sub-counties in Mombasa County while in Kilifi County the focus will be in Kilifi South, Kilifi North, Malindi, Magarini and Ganze sub counties. will be done in Mombasa and Kilifi in the listed sub-counties, all led by the same consultant or consultancy team.

Objectives of the End-line Evaluation:

Instructions: This section has been pre-populated with a general objectives statement for all AGECS project baselines. The projects may adjust these objectives or add objectives if relevant.

This consultancy’s overall objective is to conduct an endline evaluation of the Confidence and Competence project against the OECD DAC evaluation criteria: relevance, coherence, effectiveness, efficiency, impact, and sustainability. This will encompass:

1. Comparative analysis of endline data against baseline and midline data on output and outcome indicators
2. Critical assessment of the project design and delivery, identifying key challenges and success
3. Do-no-harm checks for any unintended/negative consequences of the intervention
4. Learnings and recommendations for future projects that can be drawn from this project.

Evaluation criteria and questions
The report should be structured around and answer the following evaluation questions:

• Relevance
  To what extent did the intervention objectives and design respond to beneficiary needs?
• To what extent did the intervention objectives and design align with local government policies and priorities?
• How has the project considered gender sensitivity both in the design and its implementation of activities?

Coherence

• To what extent did the project harmonize and coordinate with relevant actors in the implementation area?
• To what extent did the intervention add value while avoiding duplication of effort?

Effectiveness

• To what extent did the project achieve its intended outputs and outcomes, as defined by its performance indicators?
• To what extent did project outcomes differ for men and women?

Efficiency
Was the project completed on time and on budget?

Impact

• What are the most significant changes in members’ lives because of project interventions?
• To what extent did the intervention address gender gaps, and what remaining aspects need to be considered further?
• Did the intervention have any unintended and/or negative consequences?

Sustainability

• How likely is it that the positive effects or impacts of the intervention will continue beyond the project lifespan?

Evaluation Scope

The consultancy firm will conduct the following tasks for the assignment:

• Review the project documents (e.g., the proposal, logic model, MEL Plan, indicator tracking table, baseline and midline datasets and reports, case stories, and budget).
• Prepare an inception report, including the detailed methodology, approaches, and data collection tools closely aligned to the baseline and midline tools and field data collection plan.
• Hiring and training enumerators, pilot study, and report.
• Data collection and data quality assurance.
• Field visits to observe the project interventions on the ground for lessons learned, key achievements, and challenges.
• Field data collection on the effectiveness of project interventions (against output and outcome indicators) and strategies through individual interviews, and consultation like KIIs and FGDs with key stakeholders, community groups, and/or target beneficiaries.
• Present the key findings and recommendations of the end-line evaluation in a meeting or workshop with the project team

Table 1: List of Outcome Indicators to be assessed by sub-project and geographical coverages

Areas: Mombasa & Kilifi County

• % of AKF-supported CSOs with improved performance
  Data will be collected separately by AKF through the Organizational Performance Index (OPI) tool and shared with the consultant for analysis and reporting.
• # of policy, advocacy, and service delivery recommendations articulated by women that are integrated into ward-level or sub-county and county plans and budgets
  Document review of County plans and budgets, Key informant interviews with policymakers.
• # and % of women supported by the sub-project who start income-generating activities
  Survey with women trained/supported by the sub-project.
• # of trained women demonstrating enhanced business management knowledge and skills
  Pre- and post-training tests.
• % of women and girls participating in skills-building programs who are employed or self-employed
  Survey with women trained/supported by the sub-project.
• % of women who actively participate in community groups and networks
  Sample survey with the project population.
• Monetary value of support raised from local donors by CSOs
  Financial document review.
• The extent to which members of the project population value the work of CSOs
  Sample survey with the project population.
• # of women demonstrating enhanced knowledge and skills in effective legislative engagement
  Pre- and post-training tests.
• # and % of adolescent girls and young women (AGYW) supported by the sub-project that are linked to local markets and financial opportunities
  Survey with women trained/supported by the sub-project.
• % of community members (male and female) demonstrating gender-equitable behaviors towards adolescent girls’ and boys’ household roles
  Sample survey with the project population.
• % of effective referrals for Sexual and Gender-Based Violence (SGBV) and Adolescent and Youth Sexual and Reproductive Health and Rights (SRHR) services made through community sensitizations
  Document review from previous projects.
• % of community members citing benefits of women participating in decision-making spaces
  Sample survey with the project population.

3.2 Target groups to be covered: This evaluation needs to cover the project beneficiaries i.e. the trained community members, groups, and other stakeholders involved in the project.

The evaluation must cover the progress and activities done in the implementation i.e. providing a clear assessment of progress against the target of the results statements as stated in the project log frame.

Evaluation Methodology

4.1 Research designs

The end-line evaluation will use both qualitative and quantitative methods. Data will be collected from the project location in Mombasa and Kilifi counties. The selected consultancy firm will prepare the study methodology, adapt the midline data collection tools prepare further tools where necessary, and prepare the sampling frame to meet the objectives of this study. The methodology and relevant tools will be adjusted in consultation with The Aga Khan Foundation, pilot-tested, and finalized before the study.

4.2 Data

All primary data collected during the evaluation must facilitate disaggregation by gender of the head of household, location, and age. Data triangulation is expected for this evaluation; the secondary data available is expected to provide additional insight into the triangulation of the collected data. A range of project documentation will be made available to the selected firm that provides information about the project design, implementation, and progress, including the baseline and midline datasets for comparative analysis. The evaluation firm is required to adhere to the Aga Khan Foundation East Africa data protection and privacy policy throughout all project valuation activities, minimizing the collection of any non-essential personally identifiable information and ensuring secure storage and transfer of data. In addition, the consultancy firm is expected to explore any personal and professional influence or potential bias among those collecting and/or analyzing data and mitigating them ethically.

Ethical Standards and Safeguarding:

It is expected that the evaluation will be conducted under the guidance of ethical considerations, which includes an adherence to the principles of openness of information given; sensitivity about gender, inclusion and cultural contextual contexts; reliability and independence of findings and conclusions as well as confidentiality of information and data protection. Informed consent must be provided by all respondents. Aligning with the content of the scope of work and specific objectives shared for this study, the consultant will determine the study methodology, the data collection tools and propose the team set up for data collection to be endorsed by the project team. The consultant will determine the appropriate sample sizes considering the geographical areas, targeted groups, and the homogeneity/heterogeneity of the target population.

Moreover, the consultant will make extensive use of the available project reports including, monitoring and evaluation data collected by project staff in the Counties (baseline and outcome assessment data), project narrative reports, and documented case stories. The consultant is expected to review the above background documentation as part of the desk review phase of the study. A strong part of the information will come from the field data collection. The evaluation will be conducted using participatory methods to collect qualitative data, triangulated for validation. The consultant and his/her team will collect primary data through direct interviews and focus group discussions (FGD) with project beneficiaries and other stakeholders. The consultant will work in close collaboration with M&E officer from Aga Khan Foundation (AKF), under close collaboration with Project Coordinator for AGECS project.

Deliverables and timelines

The main deliverables and tentative timelines for this Endline evaluation are as follows:

• Final inception Report: Presentation of a finalized inception report for validation– 10th January 2024

• Draft baseline report: Submission of a draft baseline report and successful completion of validation workshop – 27th February 2025

• Final baseline report: Completed study report (the report should be disaggregated at Sub- County level for understanding the sub-counties relevant situations), including quantitative and qualitative study findings, completed dummy tables, data visualizations, and analysis of findings–25th March 2025

  Study datasets: Raw data and complete datasets in English. The Consultant will also be expected to provide a fully ‘cleaned-up’ dataset in an agreed file format (e.g., XLS, SPSS) accompanied by the code used to carry out analysis and a variable codebook.

  Transcribed transcripts: Complete, edited transcriptions and translations in English of all focus group discussions and key informant interviews conducted.

  Budget: The Budget expected will cover from data collection to the final endline report.

Key competencies

• Team lead should possess at least a master’s degree in Gender Studies, demography/ population studies, or Statistics and with essential training in Monitoring and Evaluation. Any training in public policy is an added advantage.
• Other team members should ideally possess multidisciplinary qualifications across gender, social science, social work, and related qualifications.
• Strong experience carrying out surveys and associated qualitative data collection, particularly focusing on Gender or related thematic areas.
• The lead should have experience of conducting evaluations on gender and women studies for at least 6 years.
• Demonstrated experience in designing and implementing studies, including proven experience in sound sampling, mixed methods approach (quantitative and qualitative), tool development and enumerator training.
• Demonstrated experience in employing Open Data Kit (ODK) or other computer assisted personal interviewing software.
• Demonstrated experience in data management, including designing and managing Gender Sensitive projects and capable of handling large datasets for study purposes.
• Country knowledge and experience in conducting research and impact evaluation studies in Coast region, both urban and peri-urban areas and especially the Counties applied for.
• Demonstrated experience safeguarding and working with vulnerable populations.
• Advanced proficiency in English is mandatory. Knowledge of additional languages used in the project area applied for is an asset.