ADB's Evaluation Architecture and Country Evaluations Guidelines

1. ADB s Evaluation Architecture and Country Evaluations Guidelines ADB s Evaluation Architecture and Country Evaluations Guidelines Mala Hettige Mala Hettige Senior Advisor Senior Advisor Independent Evaluation, ADB Independent Evaluation, ADB Presentation at the meeting of Sub-Regional Development Institutions Johannesburg, 27 April 2014

2. Asian Development Bank (AsDB) Asian Development Bank (AsDB) Established in 1966 in Manila, Philippines Established in 1966 in Manila, Philippines 67 shareholding member countries 67 shareholding member countries Provides $ 21b in development assistance Provides $ 21b in development assistance About 760 projects About 760 projects 5 regional department and PS department 5 regional department and PS department Managed by MDG, 6 VPs Managed by MDG, 6 VPs 3000 staff in HQ and 29 RMs + 3 rep. offices 3000 staff in HQ and 29 RMs + 3 rep. offices Administrative Budget about $600 million Administrative Budget about $600 million

3. Independent Evaluation @ AsDB Independent Evaluation @ AsDB Abides by OECD s 4 dimensions of independence Abides by OECD s 4 dimensions of independence DGE reports to the Board through the Development Effectiveness Committee (DEC) DGE reports to the Board through the Development Effectiveness Committee (DEC) Reports public upon DG-IE approval, discussed under Chair, DEC Reports public upon DG-IE approval, discussed under Chair, DEC DGE appointed/removed by DEC 5 year non-renewable term. DGE appointed/removed by DEC 5 year non-renewable term. Work Program and separate budget approved by Board Work Program and separate budget approved by Board DGE has authority over staff recruitment DGE has authority over staff recruitment Staff subject to backward and forward conflict of interest policy Staff subject to backward and forward conflict of interest policy

4. Major Evaluation Products Major Evaluation Products Annual Reports Annual Reports Country Assistance Program Evaluations Country Assistance Program Evaluations Corporate Evaluation Studies Corporate Evaluation Studies Impact Evaluation Study Impact Evaluation Study Thematic Evaluation Studies Thematic Evaluation Studies Project/Program Performance Evaluations Project/Program Performance Evaluations Technical Assistance Program Evaluations Technical Assistance Program Evaluations Real- time Evaluation Studies Real- time Evaluation Studies Project Validation Reports Project Validation Reports Other Evaluation Reports Other Evaluation Reports 4

5. Accountability and Learning Accountability and Learning 5

6. Evaluation Information System (EVIS) Evaluation Information System (EVIS) A database of evaluation lessons A database of evaluation lessons Provides users easy access and search capability Provides users easy access and search capability Lessons by sector, theme, topic, country etc. Lessons by sector, theme, topic, country etc. EVIS is the first system of its kind EVIS is the first system of its kind Accessible through http://evis.adb.org Accessible through http://evis.adb.org 6

7. Management Action Record System (MARS) Management Action Record System (MARS) Tracks ADB Management actions on recommendations Tracks ADB Management actions on recommendations IED Validates actions taken by Management IED Validates actions taken by Management Reports this validation to the Board Reports this validation to the Board Computerized and available to staff and Board Computerized and available to staff and Board Departments/offices update progress on actions Departments/offices update progress on actions

8. Country Evaluations and ADB Operations Country Evaluations and ADB Operations Country Partnership Strategy Project Identification /Preparation Appraisal / Approval Implementation Country Evaluation 8

9. History of Country Evaluations in ADB History of Country Evaluations in ADB CAPEs (Country Assistance Program Evaluations) 1998 first CAPE Peoples Republic of China 1998 first CAPE Peoples Republic of China 2002 Mongolia 2002 questioned choices made 2002 Mongolia 2002 questioned choices made 2005 Bhutan had recommendations + criteria ratings 2005 Bhutan had recommendations + criteria ratings 2006 Indonesia added ratings by sector 2006 Indonesia added ratings by sector 2007 India had contribution to themes+ value added 2007 India had contribution to themes+ value added 2007 China had Strategic and Institutional positioning 2007 China had Strategic and Institutional positioning 2008 Sri Lanka considered the implications of conflict 2008 Sri Lanka considered the implications of conflict 2008 Philippines considered political economy factors 2008 Philippines considered political economy factors 2009 Cambodia top-down + bottom up assessment 2009 Cambodia top-down + bottom up assessment 2010 Lao existing guidelines applied 2010 Lao existing guidelines applied 2014 Tajikistan assessment of objectives- ongoing 2014 Tajikistan assessment of objectives- ongoing

10. Evolving Guidelines for CAPEs in AsDB Evolving Guidelines for CAPEs in AsDB From 1998 to 2006 no guidelines First Country Evaluation Guidelines 2006 Revised Guidelines issued 2010 CAPE/ validation of a country self evaluation mandatory prior to approval of a CPS Now doing a few revisions to the guidelines Also preparing validation guidelines

12. Background: GPS on Country Evaluations Background: GPS on Country Evaluations GPS for CSPE established in 2008 Based on OECDDAC evaluation principles Draws on the 2007 ECG Review of CSPE A core GPS criteria allows comparability An optional GPS criteria helps accountability and learning

13. Application of GPS: of Country Evaluation Application of GPS: of Country Evaluation Name of study differs and number differs Coordinated with the upcoming country strategy Substituted with validations or learning products About 9-12 months, 25-75 pps, $ 250+, Lending/non lending, about 10 years (2-3 cycles) Application of principles, approach taken differs Overall Rating and Evaluation Criteria followed by about half. But try not to emphasized much

14. Objectives : Accountability Board on results over a period Learning - engage with stakeholders on future CPS Timing : to feed into the next strategy Coverage : Long enough to assess results and Recent to ensure findings are relevant Key Aspects of CPSE Key Aspects of CPSE

15. GPS for Country Strategy and Program Evaluation GPS for Country Strategy and Program Evaluation GPS for CSPE established in 2008 Based on OECDDAC evaluation principles Draws on the 2007 ECG Review of CSPE A core GPS helps allows comparability An optional GPS helps accountability and learning

16. Core Criteria Relevance Effectiveness Efficiency Sustainability Impact Optional Criteria Positioning Coherence Institutional Development MDB Performance Borrower Performance Partner Coordinatio n Evaluation Criteria Evaluation Criteria

17. Relevance Relevance GPS : Relevance refers to the degree to which MDB strategy and program were consistent to critical needs and long-term plans of the country

18. Relevance > Strategic Relevance Relevance > Strategic Relevance Can include optional criteria on positioning how well did MDB respond or anticipate Evolving development challenges Priorities of the government Build on MDB s comparative advantage Consider support from other development partners Also can include coherence extent to which there were measures to Foster internal + external synergies within MDB Foster external partnerships that promote cost- effective division of labor through complimentarities

19. Efficiency Efficiency GPS: Efficiency refers to the extent to which the design and delivery of assistance were most cost effective .

20. Effectiveness Effectiveness GPS: Effectiveness refers to the extent to which the assistance instruments achieved the intentions and objectives set .

21. Sustainability Sustainability GPS: Sustainability refers to the likelihood that actual and anticipated results will be resilient to risks beyond the program period.

22. Impact Impact GPS: Impact refers to an MDB s contribution to long-term changes in development conditions .

23. Optional Criteria Institutional Development MDB Performance Client Performance Partnership Performance Other concerns By sector? What about cross sector impacts? By objectives? How to attribute and measure? How to account for unintended impacts?

24. IED Website www.adb.org/evaluation

25. Thank You! Thank You! https://www.facebook.com/adbevaluation evaluation@adb.org www.adb.org/evaluation https://twitter.com/adbevaluation http://www.youtube.com/evaluationatadb http://www.scribd.com/adbevaluation/ http://www.soundcloud.com/adbevaluation/ Follow Us