End-Term Evaluation of the Nadoum Programme

End-Term Evaluation of the Nadoum Programme: Utilizing Outcome Harvesting to Assess Impact and Achievements

Background

Since 2019, the MENA H Coalition began the implementation of a three-year Global Fund grant titled “Sustainability of HIV and Key Population Services in the MENA Region”. The overall goal is to improve the provision of HIV services to people living with HIV (PLHIV) and key populations in the MENA region, focusing on improving regional and national ownership and sustainability of these programs. In 2020, the Global Fund invited the MENA H Coalition to submit a Validation of Continuation funding request which would extend the period for the achievement of this goal and strategic objectives through to the end of 2024.

The continuation of the grant over the 2022-2024 period is guided by the following six strategic objectives (SO):

SO1: Increase resource mobilization, technical and management capacity for Key Population (KP) organizations;

SO2: Develop, document, support and promote use of innovative and sustainable community led service delivery models;

SO3: Advocate for increased domestic financing for quality HIV prevention, treatment and care services for KPs though public health systems, community systems or through social contracting;

SO4: Reduce structural barriers, including stigma and discrimination against KPs and improve their access to health services and retention in care;

SO5: Document human-rights violations against key populations and provide ways of responding to them; and

SO6: Support processes and policy reforms towards the scale-up of comprehensive prevention programs for people who use drugs.

The grant proposes to make this contribution through achieving six outcome-level results by 2024:

• Increased technical and operational capacity for KP CSOs and networks at national and regional levels to advocate for and to secure their health and rights;
• Innovative approaches to service delivery that incorporate partnerships, integration, technology and community-led approaches;
• Diversified funding base and funding modalities for KP services (including through increased domestic investment and social contracting) to ensure their sustainability;
• Increased social protection for KPs across MENA to reduce or remove structural barriers to access, uptake and retention in HIV and other services;
• Improved promotion and protection of human rights for KPs; and
• Stronger regional and national level programme to promote HIV prevention and treatment for people who use drugs (whose HIV-related vulnerability remains a main driver the HIV epidemic across MENA).

Background to Evaluation

The implementation of Nadoum will be completed by December 2024. Frontline AIDS intends to commission an external consultancy group (or consultant) to conduct an external end-term evaluation. This evaluation will serve as an accountability tool towards the donor and grant stakeholders. It will be also utilized to advocate for increased funding for MENA, specifically focusing on grants that enable the reduction of structural barriers. Given the confirmation from the Global Fund that the grant will be discontinued by December 2024 and that the possibility of a catalytic grant is not feasible, it is crucial to understand the implications of this funding gap.

The primary audiences for this evaluation are the Global Fund, grant implementing partners, and prospective donors.

Purpose of the evaluation

The end-term evaluation aims to analyse Nadoum’s contribution to observed changes (positive, negative, intended, and unintended, direct, and indirect). It will develop stories of change that provide insights into the processes contributing to the outcomes and assess the relevance of the outcomes from the perspectives of national civil society organizations.

Evaluation Questions

• What are the positive and negative, intended, and unintended – outcomes and how do they demonstrate significance towards sustainability of services for KPs?
• What did Nadoum do that contributed to these outcomes?
• What are the indications that the changes Nadoum contributed to, will sustain beyond the lifetime of the programme?
• Did Nadoum invest in where the need is greatest?
• In which areas of the ToC did Nadoum’s inputs/strategies not lead to outcomes?
• What are the gaps/priorities that need to be made on the basis of our understanding of these outcomes that will support the advocacy for continued catalytic funding?

The evaluation questions will be further refined with the consultant/consultancy team.

Evaluation approach

A mixed-method approach:

• Participatory stakeholder-focused: gaining an understanding of program’s performance by eliciting the perspectives of key program stakeholders and beneficiaries, drawing on their direct experiences of the program or their interpretation of the program based on available data (degree to which the program contributed to or achieved change and types of changes experienced through their direct involvement with the program)
• Social Justice: producing and using evaluation findings to reduce identified inequalities. Findings to be employed in advocacy, social policy development, and to inform strategies for social change. i.e. What (mechanisms) work (outcome) for whom and under which circumstances (context). Focus on the identification of those groups the program benefitted and those it did not.
• Systems/Developmental: the grant operates in complicated and complex environments. Relationships and dynamics are non-linear. There is heavy reliance on multiple components implemented by multiple partners with no linear causality able to be established between the program’s methods and its effects. Predictability of outcomes is not easily established. Developmental evaluation approach is recommended responding to the realities posed by evaluating complicated and complex programs such as Nadoum.

Methodology

The methodology will be based on outcome harvesting, which retrospectively identifies emergent outcomes by collecting examples of what has changed in actions, relationships, policies, and practices. Outcome harvesting will allow the examination of how and why those changes were brought about and how Nadoum has contributed to them.

• A desk-based literature review: the lead consultant will be asked to carry out an initial assessment of existing progress update reports, review partner workplans, MTE report, and review existing outcome statements and outcome mapping. A total of 40 outcomes are mapped. We have been identifying outcome leads and developing them into outcomes.
• Key Informant Interviews (KIIs): Interviews with implementing partners will be held to identify outcomes and provide context, explaining how and why certain results were achieved (or why not). A criteria-based selection to ensure that key informants are selected based on their knowledge, involvement, and relevance to the program.
• Sensemaking workshops will be held in each country, ideally face to face, where possible. During the inception phase, Frontline AIDS will agree with the consultant on the modality of implementation and their presence in all or some of the workshops. The consultant is expected to design collaborative sessions aimed to interpret and make sense of the collected outcomes, enabling stakeholders to understand the implication, patterns, and significance of these outcomes in a comprehensive manner. Providing a platform for stakeholders to validate the outcomes collected, ensuring that the data accurately reflects reality and is supported by multiple sources of evidence. While developing actionable recommendations in each country to transition from Nadoum.

Whendesigning the evaluation’s data collection tools and analyzing and interpreting the data collected during the evaluation, the consultants will be encouraged to reflect on the BOND principles for assessing the quality of evidence[1]. From this perspective, good quality evidence means:

1. Voice and Inclusion: the perspectives of Key Populations and Adolescent Girls and Young Women are included in the evidence, and a clear picture is provided of who is affected and how, with attention paid in particular to those groups most marginalized due to their vulnerability, HIV status, and criminalization;
2. Appropriateness: the evidence is generated through methods that are justifiable given the nature of the purpose of the enquiry;
3. Triangulation: the evidence has been generated using a mix of methods, data sources, and perspectives.
4. Contribution: the evidence explores how change happens, the contribution of the intervention and factors outside the intervention in explaining change; and
5. Transparency: the evidence discloses the details of the data sources and methods used, the results achieved, and any limitations in the data or conclusions.

Outputs

• Inception report including tools and instruments to be used throughout the evaluation process (mapping, interviews, sensemaking workshops, etc).
• Draft and Final evaluation reports. Annexes should include the terms of reference, a list of people and organisations interviewed; a list of documentation and materials reviewed; a timeline of the evaluation process, data collection instruments used, and a database of all finalized outcomes harvested.
• Executive summary of the final evaluation (of no more than 5 pages): this may be published on the Frontline AIDS website so should be readable as a standalone document.
• PowerPoint visual presentation of final evaluation findings.

Management and governance of the evaluation

This evaluation will be directly managed by the lead consultant who will assume overall responsibility for the deliverables. Senior Advisor: M&E and Program will act as the first point of contact for the team of consultants/consultant and will be responsible for overseeing the implementation of the evaluation. In addition to the Senio Advisor, the Support Officer: MEL will support the consultant in various stages of the evaluation, including: facilitating workshops and developing outcomes statements.

Frontline AIDS will support all stages of the evaluation process including providing relevant documentation, assisting in the organisation of data collection (providing contact details, ensuring availability of interviewees and relevant data), providing feedback on drafts of all agreed outputs, including the methodology.

The evaluation will be guided by key programme and M&E Frontline AIDS’ staff working on Nadoum. They will:

• Review and refine objectives, tools and methodologies in consultation with the consultant/consultancy team.
• Provide feedback on draft inputs presented by the team of consultants.
• Ensure a management response to the evaluation is written and recommended actions are assigned to named individuals to implement.

Profile of consultancy team

The successful bidder will have one or more members who meet the following criteria:

Essential:

• Experience in undertaking evaluations using Outcome Harvesting.
• Sensitivity and understanding of needs of marginalized and vulnerable communities.
• Experience working in an international development context.
• Understanding of participatory evaluation approaches
• Ability to systematically analyse and present complex data and information.
• Excellent communication and facilitation skills
• Excellent written and spoken English and French
• Ability and commitment to deliver the expected results within the agreed period of time.

Desirable:

• Substantial experience in conducting evaluations of HIV and/or human rights programmes.
• Based in, or from, the countries where the programme is operating.
• Has knowledge of the HIV sector in the MENA
• Arabic comprehension.

[1] https://www.bond.org.uk/ngo-support/evidence-principles-download