Final evaluation of Public Policy Information Monitoring and Advocacy (PPIMA) IV Programme in Rwanda

1.Introduction

Norwegian People’s Aid (NPA) Rwanda is seeking to hire the services of a qualified consultancy firm to conduct a final evaluation of the fourth phase of its Public Policy Information Monitoring and Advocacy (PPIMA) programme in Rwanda.

1.1 Overview and rationale

The Public Policy Information, Monitoring and Advocacy (PPIMA) is a programme implemented by Norwegian People’s Aid (NPA) and funded by the Norwegian Agency for Development Cooperation (Norad), the Swedish International Development Cooperation Agency (Sida), Swiss Development Cooperation (SDC) and Danish Ministry of Foreign Affairs through its Project Office in Rwanda. PPIMA is a civil society support programme aimed at strengthening Rwandan civil society organisations (CSOs) and citizens to participate in and influence the formulation and implementation of national and local policymaking, planning and governance. As a programme, PPIMA started in 2009 and has been implemented through four phases which cumulatively span for almost fifteen years. Being implemented currently is PPIMA IV, from 2020 to end of 2024.

1.2 PPIMA IV summary and anticipated outcomes

The programme’s intended goal is to ensure that Rwandan citizens live in a democratic society with equitable social justice; that all citizens of Rwanda, including the vulnerable groups, have equal access to their economic, political, and social rights.

The 2020-2024 PPIMA Programme Phase IV is being implemented in collaboration with 20 local CSO partners across 19 Districts[1] and continues to focus its efforts on strengthening CSO partners internal governance systems, their ability to advocate, influence and conduct research and form policy while promoting citizen participation and priorities through a Community Score Card (CSC) model. The programme also aims to promote accountability among decision makers and specifically encourage them to consult their constituents to inform decisions, be responsive to citizens’ concerns, including a gender perspective in planning and budgeting and address the specific needs of vulnerable and marginalized groups.

The following outcomes are anticipated during PPIMA Phase IV:

Outcome 1: Partners are more effective in their public policy and advocacy work to influence and mobilize in recognition that an empowered civil society is a fundamental pillar of sustainable democracy.

Outcome 2: Supporting citizens’ engagement in decision-making processes, knowledge of their rights, and access to justice and conflict prevention and management mechanisms.

Outcome 3: Local and national government have formulated policies and plans that reflect the concerns raised by citizens and gender equality considerations.

2. Objectives of the assignment

The main objective of the commissioned final evaluation is to assess and determine the fulfilment of PPIMA IV objectives and intended outcomes against the OECD DACC six evaluation criteria namely relevance, coherence, effectiveness, efficiency, impact, and sustainability.

More specifically, the final evaluation will seek to:

1. Establish whether and how PPIMA IV has been doing the right thing vis-a-vis the situation it initially sought to act upon in the unique governance context of Rwanda including in responding to the critical needs and interests of different target groups and stakeholders (relevance) as well as determine how well the programme is designed and thus best fitted to address the situation (coherence);
2. Assess and confirm progress made towards achieving the overall PPIMA IV objectives and anticipated outcomes (effectiveness) at district and national levels across all three programme outcomes and in line with the Theory of Change;
3. Assess the sustainability of the PPIMA outcomes and progress towards achieving the desired impact (Impact and sustainability);
4. Assess Value for Money (VfM) performance compared to the original VfM proposition on top of the timeliness with which the outcomes were delivered against the original plan (efficiency);
5. Identify programme and non-programme related explanations for success(es) and failure(s) that could be “translated” into more effective, efficient and sustainable programme interventions in future;
6. Examine NPA and partners’ capacity to sustainably generate, share and use learning across the organization and network to strengthen adaptative implementation and programming capacity and hence improve results delivery at different levels;
7. Provide a set of concrete and actionable recommendations for improvement of the future similar programming.

3. Scope of work

The scope of work will include but is not limited to:

1. Set up, mobilize and manage a relevant team (core team and support team) that will deliver the assignment based on a prior matching of the experts with the critical skillsets needed to successfully deliver this assignment;
2. Review every relevant available documentation (including MEAL frameworks) either by NPA, partner CSOs and other sources that have documented PPIMA IV work in order to identify and understand contributions to change making;
3. Review and consider, to the extent possible, the emerging findings from the ongoing and or upcoming PPIMA-related analytical exercises namely the PPIMA Comprehensive Outcome Harvesting, the Community Score Card (CSC) Impact study for the sake of enriching the design of the evaluation framework and or minimise duplications;
4. Design a comprehensive evaluation framework in accordance with the best practices in conducting evaluative studies while ensuring a strict yet adapted application of OEAC DACC six evaluation criteria. In so doing and without limiting the creativity of the consultant with the design of the evaluation framework, a minimum consideration should be given to the below orientation by NPA:

• Ascertain PPIMA relevance and coherence, with a focus on the extent to which PPIMA’s strategy is aligned and responds to stakeholders and beneficiaries’ needs, rights, and priorities and the programme’s responsiveness to the context and quality of its (programme) design.
• Determine how effectively the Community Score Card (CSC) and Community/Women Safe Spaces processes enable to achieve the inclusion of people with disabilities, marginalized groups and women;
• Whether and how the governance structure of the management of PPIMA contributes to achieving the objectives in promoting civic participation, gender equality, environment and access to justice;
• The effectiveness of partners capacity strengthening approaches and interventions;
• NPA’s partnership and coordination approaches and coordination to deliver results;
• How effective have PPIMA IV been to generate new research and or use existing evidence in advocacy to influence decision making processes;
• Any unintended effects, both positive and negative that have occurred as a result of the programmes interventions;
• The extent to which the programme has been effective and adequately captured and explored the significance of the full range of results and outcomes, i.e. producing the results anticipated;
• The programme delivery efficiency levels and how the least costly resources possible have been used to produce programme results;
• How the PPIMA IV outcomes and impact are being sustained and are to be sustained beyond the programme’s timeline with a focus on the Community Score Card, and to what extent do local leaders, CSOs and citizens reflect ownership of PPIMA;
• The extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects at both local and national levels including at personal level for those who benefited from it;
• Whether the observed PPIMA impact addresses the ultimate significance and potentially transformative effects of its interventions;
• Whether and how well the learning accumulated along the implementation is used or not for adaptation of both implementation and or programming.

1. Prepare a comprehensive inception report, serving as evaluation execution methodology report, which includes among others very clear and highly relevant key guiding evaluation questions meant to operationalise every specific objective of the evaluation;
2. Submit, as part of the inception report, sufficiently detailed data collection instruments (both in English and Kinyarwanda) to be used during data collection;
3. Prepare the necessary documentation to support the application for the final evaluation pre-authorisation by the Rwanda Governance Board (RGB) and the research visa by the National Institute of Statistics of Rwanda (NISR) respectively, as required;
4. Collect relevant primary data, ensure full transcription into textual data of audio-visually recorded data and avail soft copies to NPA;
5. Sort-out, process and analyse data collected from the field and determine emerging findings;
6. Produce a well written, straight to the point evaluation findings report;
7. Produce other relevant materials for ease of dissemination of the evaluation results as needed by NPA (refer to the list of key deliverables);
8. Present the final evaluation results to selected audience.

4. Relevance of the assignment and potential use of the findings

The findings and recommendations from the final evaluation of PPIMA IV will be used to:

1. Draw conclusions about how well the programme has overall performed;
2. Learn about what works versus what does not work with this nature of intervention and eventually feed into better-adapted future programming;
3. Feed into the planning and execution of relevant policy advocacy processes geared towards improved conditions and framework processes for digital media development in Rwanda.

5. Preferred methodology

It is expected that the evaluation team will propose an overall evaluation design/plan to address the evaluation questions and a plan for collecting and analysing data. The evaluation design shall be based on qualitative techniques to address the evaluation questions. Key informant interviews, focus group interviews, in-depth review of PPIMA documentation and reports, discussions with PPIMA staff and feedback from partners and beneficiaries are among the techniques that the evaluation team shall consider when addressing the evaluation questions.

The plan shall include a detailed work plan, schedule and an overview of questions, a presentation of the method and a basic layout of the following considerations:

• Personal interviews with NPA and CSOs partners, staff responsible for managing and delivering PPIMA;
• Personal interviews with PPIMA donors namely Sida, SDC, DPO and Norad;
• Interviews with community animators, paralegals, governance focal points (Beneficiaries)
• Interviews with implementing actors and citizens (Beneficiaries);
• Interviews with government departments and agencies both at the local and national level to explore the reach and engagement of government actors across select Districts, RGB, MINECOFIN, MINALOC and MINIJUST;
• Interviews with community leaders and, in particular, individuals in communities who engage and do not engage in the community scorecard initiatives;
• Interviews with service delivery agencies at community, sector, district and national levels to discuss their role in and views of the CSC and other PPIMA processes;
• Where appropriate, illustrations of good (or less good) practice/outcomes should be presented;
• Quantitative methods where feasible;
• Any additional information or procedures to obtain views and feedback on the program that the reviewer feels necessary to accomplish the tasks outlined in these Terms of Reference;
• Field trip to at least ten (10) different partners and ten districts (for consultation with programme beneficiaries and key actors).

6. Key deliverables and timelines

The execution of the commissioned assignment will run from September to November 2024, and the final evaluation report should be submitted by 15th November 2024. It is in overall anticipated that the execution of this assignment will take up to a maximum of 32 cumulative days distributed per key deliverables described below. Interested service provider (consultant) must include the work plan and time schedule for executing this assignment in their technical proposal (offer), separately submitted from the financial offer.

Inception report and evaluation design (7 days): The evaluation team will review the relevant project documents, including considering the emerging findings of the ongoing Outcome Harvesting and the Community Score Card (CSC) Impact Study and produce an inception report or background report that addresses what the evaluation team has learned based on program documents.

The inception report will, at minimum, include the design of the evaluation with the following details: i) a detailed evaluation design matrix; ii) draft questionnaires and other data collection instruments or their principal features; iii) the list of potential interviewees and sites to be visited; iv) identified limitations to the evaluation design, and v) an updated schedule. The inception report should be submitted within two weeks from contract signing. Any revisions to the questions in the inception report should be documented in writing in the evaluation report and are subject to pre-approval by the quality assurance team of NPA.

Pre-Field work briefings (2 days): Following the signing of the contract and before undertaking fieldwork, the evaluation team will hold a series of in-house briefing sessions as per need to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. These series of sessions will involve NPA programme team and partners. Donors will be consulted and involved as appropriate.

Evaluation Briefing/Presentation (5 days): Drawing on the emerging findings, the evaluation team is expected to hold a series of briefing sessions and preliminary presentations after field work and following the submission of the first draft. The draft final evaluation report presentation will discuss the summary of findings and recommendations with Sida, SDC and Norad within not more than 15 business days following the conclusion of fieldwork.

Draft Evaluation Report (15 days): The submission date for the draft evaluation report will be determined in the evaluation work plan, considering the number of days required to produce the draft evaluation report and address all comments is estimated to be a maximum of 15 days.

Final Evaluation Report (3 days): The evaluation team will take no more than 3 business days to respond/incorporate the final comments from Sida, SDC, DPO and Norad. The evaluation team leader will then submit the final report.

7. Supervision and quality assurance

The successful consultancy firm will have the primary responsibility for ensuring high quality standards of the evaluation process and related deliverables. The firm will functionally report to NPA Programme Team on basis of a contact and communication tree to be later unveiled during the contracting process.

8. Desired profile of the service provider

The ideal local or international consultancy firm will need to prove that they have been into the consultancy business for at least five consecutive years and that they are specialised in evaluative studies on governance matters in Africa and preferably in Rwanda. To this end, they shall submit, as part of their offer, at least two most recent certificates of good completion and at least two contact details of references who know the quality of their work and can therefore testify to their technical and ethical competence.

International companies must demonstrate that they have previous work experience in Rwanda and prove their understanding of local context. It is mandatory, however, that the international firms must identify and partner with an experienced local firm or local expert (s) who have good understanding of Rwandan governance context. To this end, a teaming or partnership agreement/statement dully signed by all parties must be incorporated as part of their application.

For local consultancy firms, their application must be accompanied by a certificate of registration, valid RRA tax clearance and RSSB compliance certificate. The international consultancy firms must submit a proof of valid business registration issued by a competent authority in the country of incorporation and other relevant certifications (e.g. taxes and social security clearances) by competent authorities either in Rwanda or in their country of registration.

The bidding company (firm) will provide a team of experts comprised of a minimum of two key positions among others: (i) a Team Leader who is a renowned expert in programme/projects evaluation and ii) a specialist in Governance, Civic Participation and Capacity Building. All team members filling the two key positions are expected to have strong understanding of working on voice empowerment and accountability processes involving citizens, civil society organisations, governments, and development partners (DPs) in sub-Saharan Africa. Prior experience in Rwanda is desirable and mandatory only for the team leader. They will also be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest.

The Team Leader (TL) will need to be physically present in Rwanda during the evaluation period and will be ultimately responsible for the following:

• Overall management of the evaluation team,
• Coordinating the implementation of the evaluation,
• Assigning evaluation responsibilities and tasks.
• Authoring the final evaluation report in conformity with these ToRs,
• The TL must be an experienced evaluation expert with a documented/evidence track record of 5 years of experience in similar consultancies and evaluations,
• Experience and understanding of network approaches to international development, civic participation and advocacy work; ideally including field experience and demonstrating a sound understanding of gender and other social inclusion issues,
• S/he should have a strong background in the governance and capacity building field,
• S/he should be fluent in English,
• S/he should have at least an advanced degree in Social Sciences, Development Studies, Economics, Statistics, Political Science, or Public Administration.

The Governance and Civic Participation expert will be tasked to:

• Support the TL in the overall management of the evaluation team and the final products in conformity with this Statement of Work,
• S/he should be familiar with the administration and institutional capacity,
• S/he must possess excellent writing and interpersonal skills and be familiar with experience in governance and civil society programmes as well as Sida, SDC, DPO and Norad programs, objectives, and reporting requirements,
• S/he should have experience in designing and implementing FGDs. A Master’s or Bachelor’s degree in Social Sciences, Political Science, Public Administration, or a related field is required to ensure that all areas of technical expertise required for the evaluation are effectively covered. Fluency in English is required and Kinyarwanda desirable as is significant prior work experience in Rwanda.

9. Procurement method

This tender follows a competitive tendering method and is open to both local and international consultancy companies that have the desired profile as described under section 8.

10. Evaluation criteria and scoring

A quality-based selection method that prioritises the soundness of the technical offer will carry more weight over the pricing indicated in the financial offer. NPA reserves the right though not to award the tender to the best technical offer in case no middle ground about the pricing negotiations can be reached or in case the entity emerging as best offeror did not pass the due diligence.

Technical offer:

Description

Quality soundness of the technical offer in terms of the proposed methodology and how it relevantly addresses all the study objectives as well as the different elements of scope of work.

Weightage

40%

Description

Suitability of expertise of the proposed core team

Weightage

15%

Description

Experience of implementing similar work with development organizations (15%)

Weightage

15%

Description

Sufficiently detailed financial proposal in EUR (consultant fees, clear breakdown of activity costs, travel costs, etc.)

Weightage

30%

11. General Terms and Conditions

All rights reserved to Norwegian People’s Aid. NPA reserves all right to reject all applications not fulfilling the requirements associated to this request. NPA will not be responsible or liable for any cost and time allocated by applicants to the preparation and submission of Expressions of Interest. NPA reserves the right to engage other companies or individual consultants if required.

Any conflict of Interest involving an applicant must be fully disclosed to NPA. When failed to discose any conflict of interest, the application will be disqualfied and no award will be made in such case.

How to apply

A technical offer not exceeding 10 pages, annexes excluded, and a separate budget have to be virtually submitted to NPARwandaprocurements@npaid.org with the mention “PPIMA IV Final Evaluation Offer’’ under the subject heading before or not later than the 5th September 2024 at 4:00 p.m. Kigali local time.

Please note that the technical offer needs to clearly demonstrate how the different objectives of this assignment will be addressed methodologically (including the sequencing of the qualitative versus quantitative methodology) and how the different elements of the scope of work will be handled.

Financial offer (taxes inclusive) with detailed budget items, units and unit costs as well as budget notes relevant for each budget line will also be submitted separately via the same email address mentioned above. All relevant costs including travel and accommodation must be included in the budget for the consultancy as NPA will NOT cover any separate costs related to this activity. The available budget for this evaluation is 65,000EUR. Financial offers beyond this budget ceiling will be rejected.

NPA will hold prior discussions with short listed interested consultants/firms to provide further clarification to the ToR in order to ensure quality final offers. Short listed consultants/firms will be contacted by NPA for a meeting/ interview prior to a final revision of the technical proposal and budget. Applicants will be notified about their bidding outcome before 15th of September 2024.

Please note that the technical offer should, in addition to demonstrating the understanding of this intended work, include a critique of the actual Terms of Reference, and make technical suggestions about what needs to be improved and how with a view to ensuring high-end quality products.