SlideShare uma empresa Scribd logo
1 de 105
Page 1 of 105
Final Evaluation of the
Afghanistan Justice Training
Transition Program
Prepared by Evaluation Team: Conor Foley and Orsolya Székely
March 2016
Page 2 of 105
ACKNOWLEDGEMENTS AND DISCLAIMER
This is an independent, external evaluation report carried out by the above authors into the Justice
Training Transition Program(JTTP),implemented by the International Development LawOrganization
(IDLO). The contents and conclusions of this evaluation report are based solely and strictly on the
opinions of the authors, and should in no way be seen as reflecting those of IDLO, its program staff or
donors.
The evaluation team would nevertheless like to thank the staff of the JTTP,IDLO in Headquarters and
in Afghanistan, particularly in the JTTP M&E unit, who not only put up with our intrusive presence
and questioning, but actively and generously assistedour work. They provided us with invaluable feed-
back and additional guidance on the production of our initial Inception Report, promptly furnished us
with all necessary documents and facilitated us in making contacts and arranging interviews in
Afghanistan and Rome. They also maintained and updated a daily interview schedule for us, which
maximized the number of interviews that we were able to conduct. IDLOorganized our accommodation
and board during the field mission, provided us with transport, interpreters and security clearance for
each external interview, and took care of our personal security. IDLO staff also made themselves
available for interview outside office hours and at weekends, which maximized our time available to
speak to Afghan government and justice sector officials during the official working week. The JTTP
translation team in Kabul also worked extremely hard to translate over 200 questionnaires distributed
during the focus group process. The JTTP finance team in Rome prepared a useful summary of the
program’s budget and expenditure up to December 2015, which helped structure our discussion of this
issue.
The evaluation team owe an enormous debt of gratitude not only for the support provided, but for the
unfailingly polite, constructive and friendly manner in which it was given. Afghanistan has a much
deserved reputation for the hospitality of its people. It is a difficult, dangerous and stressful place for
the staff of international organizations to work in at the moment, and is rightly considered a hardship
post for those who are based there. But we were made to feel welcome and supported by IDLO’s
national and international staff every moment that we were in the country.
Needless to say, any errors of fact or interpretation are solely the responsibility of the evaluation team.
Page 3 of 105
TABLE OF CONTENTS
Acknowledgements and Disclaimer ............................................................................................... 2
Table of Contents ......................................................................................................................... 3
Program Geographical Scope in Afghanistan.................................................................................. 5
Acronyms .................................................................................................................................... 6
Executive Summary...................................................................................................................... 8
1 Evaluation Purpose, Scope, Methodology and Limitations ......................................................12
1.1 Purpose and Scope....................................................................................................... 12
1.2 Evaluation Questions to be addressed............................................................................ 13
1.3 Evaluation Questions ................................................................................................... 14
1.4 Data Sources and collection methods ............................................................................ 15
1.5 Management and logistical support ............................................................................... 17
1.6 Limitations of the evaluation ........................................................................................ 17
1.7 Conflicts of Interest...................................................................................................... 18
2 History and Overview of the Program....................................................................................19
3 Main Evaluation Findings .....................................................................................................21
Relevance............................................................................................................................... 21
3.1 Evaluation Question 1: To what extent was the program theory of change able to respond to
the changing needs and context of the Afghan justice sector,including changes to the political and
security context?..................................................................................................................... 21
3.2 Evaluation Question 2: How able was the implementing organization to respond to the
identified needs/priorities?....................................................................................................... 24
Effectiveness.......................................................................................................................... 27
3.3 Evaluation Question 3: Was the JTTP successfulin delivering the expected outputs with a
focus on increased quality at the same time?............................................................................. 27
3.4 Evaluation Question 4: Was JTTP able to identify innovative approaches and implementing
mechanisms to facilitate the delivering of the expected outcomes and outputs despite the challenges
of Afghanistan? ...................................................................................................................... 33
3.5 Evaluation Question 5: What were the key internal and external factors that contributed to
the program successes / hindered its effectiveness? ................................................................... 36
3.6 Evaluation Question 6: To what extent did funding, staff, time and other resources
contribute to achieving the results?........................................................................................... 40
3.7 Evaluation Question 7: Did the Program deliver its outputs in a cost-efficient and timely
manner? ................................................................................................................................. 44
Impact.................................................................................................................................... 48
3.8 Evaluation Question 8: What results did JTTP produce according to the planned logical
framework, and did it go beyond the originally planned results?................................................. 48
3.9 Evaluation Question 9: Have there been any lessons learnt in terms of result delivery which
could be transferred as a knowledge pool to other programs in the future? .................................. 55
Page 4 of 105
3.10 Evaluation Question 10: What communication strategies and other tools were put in place to
enhance program visibility? ..................................................................................................... 57
3.11 Evaluation Question 11: Will the results produced by capacity development of justice
professionals likely to sustain in the future and what are the necessary conditions for the benefits
achieved to be consolidated?.................................................................................................... 59
3.12 Evaluation Question 12: To what extent are the systems, capacities, and facilities transferred
to in-house training departments going to be retained within the institutions without external
support (organizational sustainability)? .................................................................................... 62
3.13 Evaluation Question 13: Has gender been mainstreamed and aligned into the JTTP activities
especially with regard to the recommendations of the mid-term review?..................................... 64
4 Conclusions .........................................................................................................................67
5 Recommendations ................................................................................................................68
6 Appendices..........................................................................................................................69
Annex 1: List of Documents Reviewed..................................................................................... 69
Annex 2: List of Interviews ..................................................................................................... 76
Annex 3: Focus Group Discussions .......................................................................................... 79
Annex 4: Phases of Work....................................................................................................... 85
Annex 5: Conduct of Focus Group Discussions Guide.............................................................. 86
Annex 6: Focus Group Questionnaire ...................................................................................... 88
Annex 7: Notes for conducting an interview with provincial officials ........................................ 89
Annex 8 IDLO Terms of Reference For the Evaluation, 15 December 2015................................ 91
Annex 9 Evaluation Matrix...................................................................................................... 97
Page 5 of 105
PROGRAM GEOGRAPHICAL SCOPE IN AFGHANISTAN
Page 6 of 105
ACRONYMS
ACLEA Advanced Continuing Legal Education for Afghanistan
AIBA Afghanistan Independent Bar Association
AIHRC Afghanistan Independent Human Rights Commission
ANA Afghanistan National Army
ANP Afghanistan National Police
AGO Attorney General’s Office
CID Criminal Investigation Division
CLE Continuing Legal Education
CLRWG Criminal Law Reform Working Group
CMS Case Management System
CPC Criminal Procedure Code
CSI Crime Scene Investigation
ETC Education Training Committee
DAC
OECD
Development Assistance Committee of the Organization for Economic Co-operation and
Development
DOWA Department of Women’s Affairs
EVAW law Elimination of Violence Against Women law
EUPOL European Union Police Mission
FGD Focus Group Discussion
FRU Family Response Unit
GDET General Directorate of Education and Training
GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit
GTC General Training Command
HOOAC High Office of Oversight and Anti-Corruption
IANDS Interim Afghanistan National Development Strategy
IDLO International Development Law Organization
IGO Inter-Governmental Organization
IIPLD Institute International de Paris le Defense
ILFA International Legal Foundation for Afghanistan
ILF International Legal Foundation
INL U.S. State Department Bureau for International Narcotics and Law Enforcement Affairs
ISISC International Institute of Higher Studies in Criminal Science
ITA International Training Advisor
GiROA Government of the Islamic Republic of Afghanistan
JSSP Justice Sector Support Program
JTTP Afghanistan's Justice Training Transition Program
LAD Legal Aid Department
LAOA Legal Aid Organization of Afghanistan
M&E Monitoring and Evaluation
MOWA Ministry of Women’s Affairs
MTT Mobile Training Team
MOA Memoranda of Agreement
MOI Ministry of Interior
MOJ Ministry of Justice
MOU Memorandum of Understanding
NAD Needs Assessment Department
NGOs Non-Governmental Organizations
NJP National Justice Program
NJSS National Justice Sector Strategy
NLA National Legal Adviser
Page 7 of 105
NRC Norwegian Refugee Council
OCAT Organizational Capacity Assessment Tool
OVI Objective Verifiable Indicator
PHQ Police Head Quarters
PPP Procurement Policies and Procedures
R&R Rest and Recreation
SIGAR Special Inspector General for Afghanistan
TCUB Training Capacity Building Unit
TIMS Training Information Management System
ToC Theory of Change
ToR Terms of Reference
ToT Training of Trainers
UNAMA UN Assisted Mission to Afghanistan
UNDP UN Development Programme
UNODC UN Office on Drugs and Crime
UNDSS UN Department of Security Sevices
US United States
USAID US Agency for International Development
USIP US Institute for Peace
VAWU Violence Against Women Unit
Page 8 of 105
EXECUTIVE SUMMARY
Afghanistan faces huge ongoing challenges as an increasingly violent insurgency threatens government
control over ever larger parts of the country. There is now a very real danger that the precipitate
withdrawal of international support could reverse the gains of recent years. Nevertheless, there are
some grounds for optimism and some areas where international investment eventually paid dividends:
the capacity-building of Afghan criminal justice professionals is one of these.
This report presents the results of a final evaluation of the Justice Training Transition Program (JTTP),
funded by the US State Department Bureau for International Narcotics and Law Enforcement Affairs
(INL) and implemented by the International Development Law Organization (IDLO) in partnership
with the Government of Afghanistan. This evaluation covers almost the whole period of the JTTP’s
existence from its establishment in January 2013 to the final completion of the extended program in
March 2016. The purpose of the evaluation is to:
 Assess the extent to which the JTTP was able to achieve its intended objectives;
 Provide IDLO and INL useful information to inform/validate a follow-on program design;
 Identify best practices and lessons learned in the JTTP’s theory of change and implementation
that can be valuable for future capacity development and transition programs.
The evaluation team used the following interconnected data collection methods: (i) document review;
(ii) field mission; (iii) interviews; (iv) focus groups and a questionnaire form distributed at these; and
(v) observations. Whereverpossible, the evaluation teamtriangulated this data using a range of sources.
Its findings have been structured around the five internationally-accepted evaluation criteria set out by
the Development Assistance Committee of the Organization for Economic Co-operation and
Development (DACOECD):Relevance,Effectiveness,Efficiency, Impactand Sustainability. The main
findings of this evaluation are as follows:
Relevance
The JTTP’soverall goal is to increase the trust of the citizens of Afghanistan in the formal state criminal
justice sector. It provides direct capacity-building support activities – through training and coaching –
to individual Afghan legal professionals within four targetinstitutions: the Supreme Court, the Attorney
General’s Office, the Ministry of Justice and Ministry of Interior, while working to build the capacity
of these institutions to continue these activities themselves. This corresponds with the Afghan
government’s own goals, the objectives of its main donors and IDLO’s global strategic priorities. The
evaluation team believe that the capacity-building support provided to individual Afghan legal
professionals was relevant and appropriate to their needs. While its transition strategy was somewhat
delayed it was appropriate to the challenges that Afghanistan faces. Indeed, these challenges are
precisely why programs such as the JTTP are so necessary and relevant. The transition strategy was
designed and implemented in full consultation with the target institutions and took account of their
needs and capacities. Memoranda of Understanding (MoUs) have now been signed with all four
national justice institutions, which clarify the transition related co-operation for the period of the JTTP
and potentially beyond, should another project continue to be implemented with the same target
institutions. IDLO proved capable of implementing the JTTP and brought added value when compared
to the previous implementing organization. The decision in the JTTP’s initial design to focus
exclusively on the official state criminal justice system, however, significantly reduced the program’s
scope in addressing how justice is actually administered and perceived in Afghanistan.
Effectiveness
The JTTP became increasingly effective at improving the quality of its individual capacity development
support and at developing co-ordination and outreach mechanisms with both the national Afghan
Page 9 of 105
institutions and other international stakeholders, aimed at a sustainable transition. Reporting, and
monitoring and evaluation systems have been developed, which have been used for effective
communication with the donor, strategic planning and measurementof the program’s results. The JTTP
has also become increasingly effective at working with the target justice sector institutions to define the
assistance that it can provide towards establishing and supporting operational in-house training
departments. This has included helping each institution to establishing governance arrangements and
helping to recruit and train instructors for the units once these have been approved.
The JTTP is one of the largest justice sector capacity-building project in Afghanistan and this scale,
combined with the strong emphasis on sustainability, has led IDLO to develop some highly innovative
approaches during the course of the program. The evaluation team believes that the JTTP has put
considerable effort into improving the material, content and delivery of its courses. Its Curriculum Unit
has continued the development of this material and produced individualized training manuals to hand
over a well-developed and designed package of training materials to the national justice sector
institutions. The recipient institutions, for their part, also appear to have increasingly recognized the
value of the program in addressing their own capacity gaps. The JTTP has transformed itself from a
relatively passive deliverer of legal training services, against set quantitative targets, to a far more
engaged and pro-active program that is actively attempting to transform the justice sector in
Afghanistan. It exceeded its achievements in both the numbers of people that it trained and coached
and by increasing the number of training opportunities offered. It also developed a pro-active strategy
of outreach and engagement with both national and international institutions.
Efficiency
The JTTP budget makes it by far the largest program that IDLO has ever implemented, although it is
relatively small in the overall context of the investment by the international community in rule of law,
justice sector reform and governance in Afghanistan. The JTTP obviously has a low number of direct
beneficiaries, if these are measured solely in terms of individual graduates of its capacity-building
support activities. If its theory of change is accepted, however, its number of indirect beneficiaries
could be multiplied to include everyone who has an interest in seeing a prosperous, fair and stable
Afghanistan. The JTTP has benefited from a close relationship with its donor, INL, which has a good
institutional memory of the program and has provided strong oversight of it. The JTTP has had a
significant under-spend, which enabled a no-cost program extension of nine months and still left over
money for a follow-on program. The under-spends seemto have beencausedby a combination of some
delays in fulfilling some activities on time – principally due to problems associated with recruiting and
retaining international staff – some cost-efficiencies implemented by the program, and some evidence
that the original budget may have been too generous in some areas. The evaluation team believe that
IDLOcould usefully explore how to maximize the organizational and thematic synergies betweenthese
programs as part of an overall effort to establish a more unified profile of its work in Afghanistan. The
evaluation team is satisfied that IDLO was able to scale up its organizational capacity to implement the
JTTP and the extra capacity that it has developed in the process means that it is now capable of credibly
taking on further large programs, both in Afghanistan and elsewhere.
Impact
The JTTP has pioneered the development of methodology to evaluate the effectiveness of its capacity
development support activities on Afghanistan’s justice sector institutions and the quality of justice that
these deliver. This appears to enable the JTTP to evaluate the impact of its activities at an institutional
level. Preliminary data shows significant improvements in the quality of justice delivered by the
graduates of its training and coaching programs and also to identify where ‘capacity gaps’ still exist.
During all interviews and FGDs it was repeatedly stated that the criminal justice system is improving
in Afghanistan and that its legal professionals are using the knowledge that they have acquired from the
JTTP to improve the quality of their work at both an individual and institutional level. Although its
transition strategy started late there has been significant progress with all the target institutions. A
Lessons Learned Brief (LLB) was in the process of being drafted during the present evaluation, which
should enable IDLO to develop a detailed and evidence-based theory of change that can be used both
Page 10 of 105
for program development and evaluation purposes in the future. The JTTP’s transition strategy
contained clear benchmarks to indicate progress in each institution. This shows that progress in the
Supreme Court and Ministry of Interior hasbeen considerable and the progressin the Ministry of Justice
could be regarded as satisfactory. Implementation within the AGO is behind schedule, however,
although this was mainly for externalfactors,including delays to the appointment of a permanent AGO
that were beyond the capability of the JTTP to control. The four institutions also had quite different
levels of capacity at the start of the JTTP and this clearly influenced progress under Outcome Two.
There are many other lessons that can be learned from the program’s implementation, in particular how
a sustainable transitional training program canbe developed if it is not clearly linked to the management
structures, human resources and performance appraisals systems of the institutions to whom it is
intended to subsequently transfer the training. The JTTP succeeded in scaling up to implement its
largest ever program in the most dangerous and insecure environment in which it has ever operated.
Sustainability
The JTTP has developed clear criteria for selecting candidates for participation in its training courses
and coaching, good pre and post course testing of the knowledge gained and follow-up methodology to
measure changes in work behavior and the impact this has had on the relevant institutions. While
there is no guarantee that the capacity development achievements of the JTTP will be fully sustainable
over the long-term, it does seem reasonable to conclude that the program has met the objective set out
in its original proposal. There remains a clear question mark over the sustainability of its transition
arrangements given both the newness of the institutions and the parlous state of the government of
Afghanistan’s finances. Afghanistan is almost entirely reliant on international donors for continuing
financial support. Given its other domestic priorities, it is unrealistic to expect the government to fund
the establishment of the training units envisaged under Outcome Two without external donor support,
but it is vital the government takes ‘ownership’ of this process through a serious budgeting process.
Although the target institutions can be given budgets to conduct training operations they still need to
develop the capacity to manage, design, conduct, and evaluate institutional trainings. The evaluation
team, therefore,believes that a successor program to the JTTP is needed to continue to provide support
to the new training units. IDLO was in the process of discussing a follow-up program to the JTTP,
while this evaluation was taking place. The evaluation team hopes that this may include some of the
JTTPs most obvious strengths: an impressive curriculum, a developed and sophisticated M&E system,
a team dedicated to maintaining links with the target institutions and some capacity to deliver particular
training to target groups of Afghan legal professionals – particularly at an inter-institutional level.
Conclusions
The evaluation’s overall conclusion is that the JTTP was relevant to the context in which it was
implemented and that its activities were generally conducted in an effective and efficient manner. Its
individual capacity-building activities aimed at supporting Afghan legal professionals have performed
a valuable service that was appreciated both by the Afghan national justice institutions and by the
participants in the training themselves. The JTTP became increasingly effective at working with the
target justice sector institutions to define the assistance that it can provide towards establishing and
supporting operational in-house training departments. It also became more proactive in looking for
linkages between its work on strengthening the capacity of the justice sector and those organizations
that are involved in monitoring its performance in relation to justice and human rights. The evaluation
team believes that the JTTP’s activities have had a positive impact, which could be sustainable in at
least the medium term given some continuing external support.
Recommendations
The key overall recommendations arising from the above findings are as follows:
1. IDLO should build on the strengths of the JTTP in its follow-up program, which should
focus on helping the Afghan justice sector institutions to deliver their own training
Page 11 of 105
programs, ensuring quality control of this training and promoting greater transparency and
legal clarity within the Afghan justice system.
2. IDLO should actively consider how to increase synergies between this program and other
programs that it is implementing in Afghanistan.
3. IDLO should also actively work to increase its linkages and synergies with other similar
programs working in the area of justice sector reform and capacity-building.
4. The new program should maintain the capacity to deliver inter-institutional specialized and
tailored courses on particular legal topics as requested by Afghan institutions or other
stakeholders.
5. The new program should have the capacity to enable IDLO to participate in debates about
legal reform, including developments in parliament and at the Supreme Court.
6. The new program should have the capacity to continue monitoring and evaluating the impact
of capacity-building development on partner institutions and to provide some support to
address ‘capacity gaps’ where necessary.
7. The new program should conduct proactive outreach with organizations that are currently
monitoring the performance of the Afghan justice sector.
8. IDLO should draw on its Lessons Learned Brief (LLB) in Afghanistan to develop a detailed
and evidence-based theory of change that can be used both for program development and
evaluation purposes in the field of developing justice sector capacity to organize continuing
professional development.
9. IDLO should draw on the methodology used to conduct its Levels 3 and 4 Assessments in
other programs.
10. IDLO should also consider what other lessons can be learned from its experiences
implementing the JTTP in the particular circumstances of dangerous and insecure situations
such as Afghanistan and continue to develop organizational policies and procedures that take
account of such situations while providing field offices with tools tailored to their particular
situations.
11. IDLO should ensure that it develops a communications strategy for all future program
activities, carefully tailored to the program and country in which it is to be implemented.
12. IDLO should develop a comprehensive risk-management strategy into its programming
methodology.
13. IDLO should continue to ensure that gender equality and gender mainstreaming are fully
integrated into all its existing program activities and reflected in all relevant program
documents, in line with its gender strategy.
Page 12 of 105
1 EVALUATION PURPOSE, SCOPE, METHODOLOGY AND
LIMITATIONS
1.1 PURPOSE AND SCOPE
This evaluation covers almost the whole period of the Justice Training Transition Program (JTTP)’s
existence from its establishment in January 2013 to January 2016 – two months before the final
completion of the extended program in March 2016. The purpose of the evaluation is to:
 Assess the extent to which the JTTP was able to achieve its intended objectives;
 Provide IDLO and INL useful information to inform/validate a follow-on program design;
 Identify best practices and lessons learned in the JTTP’s theory of change and implementation
that can be valuable for future capacity development and transition programs.
A mid-term evaluation (MTE) was carried out between July and October 2014, which noted the JTTP
was broadly ‘on track’ towards achieving its targets under the program’s first objective, capacity
development and training, but that it had been very late in drafting a transition strategy for handing over
the training activities to national justice sector institutions in a sustainable manner. 1
The MTE
concluded that the JTTP’s Transition Plan was ‘basically sound as a working document’ but required
‘buy in and national ownership’. The MTE did not believe that this could be accomplished by July
2015, the original date for the completion of the program. It recommended the strategy be revised with
new timelines and increased staff resources devoted towards this transition. It further recommended
that ‘IDLO should, as a matter of priority seek to negotiate a no-cost contract extension for the JTTP
with INL to early 2016.’ It also recommended that: ‘IDLO and INL should begin discussions about a
follow up project starting in 2016. This should focus on helping the Afghan justice sector institutions
deliver their own training programs and ensuring quality control of this training. The project should
also conduct pro-active outreach with organizations that are currently monitoring the performance of
the Afghan justice sector.’2
These proposals were broadly endorsed by IDLO’s senior management team.3
IDLO submitted a no-
cost contract extension request until 31 March 2016, which foresaw ‘completing all training activities
and close provincial offices by January 2016; and continue to work on the Transition in early 2016 to
support the target institutions towards the next Afghan government budget cycle.’ The JTTP also
created a new unit with dedicated focal points specifically dedicated to supporting the transition
process.4
While the legal capacity building component of the JTTP (Outcome One) was extensively covered by
the MTE, the effortsof transferring this training and coaching capacity to the Afghan justice institutions
(Outcome Two), synthesized in the Transition Plan, is assessed here for the first time. This final
evaluation also addresses the JTTP’s impact, which was omitted from the terms of reference of the
MTE. It seeks to highlight innovative measures,good practices and lessons learned that may be useful
to the JTTP and INL in discussing the design of a follow up program as well as providing IDLO with
information that may help the design of other similar programs elsewhere in the world.
As the MTE noted, evaluating any justice sector capacity-building program is inherently difficult,
because the results and impact of these efforts are not immediately visible and may take years or even
decades to become apparent. These difficulties are particularly pronounced in Afghanistan, given the
1 Conor Foley and Katerina Stolyarenko, External mid-term evaluation ofAfghanistan Justice Training Transition
Program (JTTP) (2013-2014),November 2014, p.8. Executive Summary [Hereinafter MTE, 2014].
2 Ibid.
3 IDLO, Implementation Tracking Matrix of the Management response to the mid-term evaluation ofthe
“Completing the transition in Afghanistan:Justice Training Transition Program (JTTP)”, DRAFT, 25 May
2015 [Hereinafter IDLO Management Team Response,2015].
4 Ibid.
Page 13 of 105
weaknesses of the rule or law, widespread corruption and a growing anti-government insurgency.
Nevertheless, this final evaluation sought evidence of changes in the work-related behavior of the
JTTP’s graduates and also analyzed what positive and negative factors contributed to changes in
performance in the different institutions that are being supported.
Some aspects of the individual capacity development support provided by the JTTP were criticized
during the MTE, by both representatives of the target institutions and graduates of its coaching and
training programs. This evaluation re-visits some of these criticisms and the extent to which they were
subsequently addressed by the JTTP, but its main focus is on the sustainable transition of the support
provided to the Afghan justice sector institutions themselves.
1.2 EVALUATION QUESTIONS TO BE ADDRESSED
The findings of this evaluation have been structured around the five internationally-accepted evaluation
criteria set out by the Development Assistance Committee of the Organization for Economic Co-
operation and Development (DAC OECD): Relevance, Effectiveness, Efficiency, Impact and
Sustainability.5
After detailed discussion with IDLO during the Inception Phase of this evaluation,
these were sub-divided further into a total of 13 evaluation questions that are set out below. A detailed
Matrix of the Evaluation Methodology, including questions, indicators, data sourcesand collection tools
is contained in Appendix Nine.
5 International Development Law Organization, A guide for course/project managers, assistants, training focal
points,data entry staff on Implementing the IDLO approach to evaluating training,April 2010.
Page 14 of 105
1.3 EVALUATION QUESTIONS
Relevance
Evaluation Question 1: To what extent was the program theory of change able to respond to the
changing needsand context of the Afghan justice sector,including changesto the political and security
context?
Evaluation Question 2: How able was the implementing organization to respond to the identified
needs/priorities?
Effectiveness
Evaluation Question 3: Was the JTTP successful in delivering the expected outputs with a focus on
increased quality at the same time?
Evaluation Question 4: Was JTTP able to identify innovative approaches and implementing
mechanisms to facilitate the delivering of the expected outcomes and outputs despite the challenges
of Afghanistan?
Evaluation Question 5: What were the key internal and external factors that contributed to the
program successes / hindered its effectiveness? 

Efficiency
Evaluation Question 6: To what extent did funding, staff, time and other resources contribute to
achieving the results?
Evaluation Question 7: Did the Program deliver its outputs in a cost-efficient and timely manner?
Impact
Evaluation Question 8: What results did JTTP produce according to the planned logical framework,
and did it go beyond the originally planned results?
Evaluation Question 9: Have there been any lessons learnt in terms of result delivery which could be
transferred as a knowledge pool to other programs in the future?
Evaluation Question 10: What communication strategies and other tools were put in place to enhance
program visibility?
Sustainability
Evaluation Question 11: Will the results produced by capacity development of justice professionals
be sustainable in the future and what conditions are necessary for the benefits achieved to be
consolidated?
Evaluation Question 12: To what extent are the systems, capacities, and facilities transferred to in-
house training departments going to be retained within the institutions without external support
(organizational sustainability)?
Cross-cutting
Evaluation Question 13: Has gender been mainstreamed and aligned into the JTTP activities
especially with regard to the recommendations of the mid-term review?
Page 15 of 105
1.4 DATA SOURCES AND COLLECTIONMETHODS
The evaluation team consisted of two international contractors who were assisted during the Field
Mission by four national contractors: Although all four national contractors had previously worked for
the JTTP, care was taken to ensure that they remained rigorously independent of the program for the
purposes of the evaluation. All four were only used in provinces where they had not previously worked
for IDLO and all received prior training and instruction on how to conduct interviews and focus group
discussions (FGDs) in an impartial manner.
The evaluation team used the following interconnected data collection methods: (i) document review;
(ii) field mission; (iii) interviews; (iv) focus groups and a questionnaire form to be distributed at these;
and (v) observations. Wherever possible, the evaluation team triangulated this data using a range of
sources.
Document Review
A full list of all documents reviewed is contained in Appendix One. The evaluation team also drew on
other published reports on the state of the justice system in Afghanistan and its adherence to
international human rights standards, as well as critical evaluations of efforts to strengthen its capacity,
such as the current program. The evaluation team also reviewed gender related documents relevant to
the JTTP’s progress since the mid-term evaluation.
Field Mission
The evaluation team conducted a field mission to Afghanistan in January and February 2016, which
covered five out of the seven provinces where the JTTP had been implemented through its provincial
offices: Kabul, Nangarhar,Balkh, Heratand Badakhshan. Asdiscussed above, four national contractors
were hired by the team in order to maximize both the number of interviews and FGDs and the
geographical scope of the mission. The security situation precluded a visit to Kunduz, by either national
or international contractors,and it was also decided not to visit Bamyan as the winter snow can lead to
flights in out of the province being cancelled, which could have left a team member marooned there.
Initial interviews and the first FGD were conducted in Kabul at the start of the field mission and then
the international contractors visited Herat and Balkh respectively, accompanied by two national
contractors. They conducted more interviews and FGDs in these provinces and then returned to Kabul
where they conducted more interviews, observed training course and attended other meetings. The two
national contractors proceeded onto Nangarhar and Badakhshan where they conducted interviews and
FGDs, using a standard questionnaire that the evaluation team had previously designed and tested. Two
other national contractors assisted in the administrative organization of two FGDs in Herat and then
travelled to Kabul where they conducted more FGDs using the same format. The two international
contractors participated in and observed some of the FGDs conducted in Kabul to ensure consistency
of their delivery and conduct.
Six of the JTTP provincial offices were closed between November 2015 and January 2016, before the
evaluation team could visit them, but the team were able to visit regional training centers,established
by the Ministry of Interior (MoI), interview relevant officials from the four partner institutions, conduct
FGDs with direct program beneficiaries and also try to seek some independent assessment of the
program’s results, relevance, effectiveness and sustainability. Meetings were conducted with the
regional offices of the Afghan Independent Bar Association (AIBA) and UN Assistance Mission to
Afghanistan (UNAMA) Judicial Affairs Officers who provided valuable insights into how the justice
system was functioning at the provincial level. The evaluation team also attended a Board of Donors
meeting convened by UNAMA and observed some training classes as they were being conducted.
The schedule of the evaluation team’s visits is set out below:
Page 16 of 105
Dates January – February 201 Province
28 - 30 January. All Kabul
31 January – 4 February. One national and one international Balkh
31 January – 4 February. One national and one international Herat
5 - 10 February. One national Badakhshan
5 - 10 February. One national Nangarhar
4 – 14 February. Two international and two national Kabul
10 – 14 February. All Kabul
Interviews
The evaluation team conducted one-to-one interviews with almost 100 people. A full list of
interviewees is contained in Appendix Two. A standardized interview question sheet is contained in
Annex Seven. Interviews were conductedface-to-face whereverpossible and those thatwere conducted
by Skype are identified accordingly. The interviewees included representatives of the following
organizations:
 IDLO personnel in Headquarters and Afghanistan, including the JTTP and IDLO’s other two
programs in Afghanistan
 Representatives from the Ministry of Interior, Attorney General’s Office, Supreme Court,
Ministry of Justice;
 Ministry of Women Affairs;
 the Afghan Independent Bar Association;
 Training staff from the target institutions;
 US State Department (INL), USAID and the US Institute of Peace;
 UNAMA, UNDP,and UN Women;
 Other international organizations working in the field of access to justice, gender justice and
justice sector capacity building in Afghanistan, including JSSP and other relevant
organisations;
 NGOs working directly on justice related issues or analyzing the Afghan justice sector.
Focus Group Discussions
The evaluation team also went to considerable lengths to organize as many FGDs with JTTP graduates
aspossible. Participants in the FGDs included: graduates of the AdvancedContinuing Legal Education
for Afghanistan (ACLEA) and ACLEA-Plus Courses, graduates of Continuing Legal Education (CLE)
Course, and the staff of the training units and seconded trainers. Simple questionnaire forms were also
distributed at the start of each FGD and collected at the end. A sample questionnaire and the Guidance
for the FGDs are contained as Appendices Five and Six of this report.
The aim of the FGDs was to explore whether and how the training, and coaching support provided to
individual legal professionals (graduates) also benefited the performance of their department and
institutions as a whole and what other capacity outcomes can be identified at the individual level as a
result of the JTTP intervention and engagement strategy. FGDs were also organized during the MTE
and provided valuable information about the quality and conduct of the training and coaching provided
by the JTTP. For this evaluation, the emphasis of the guiding questions was less on the participants’
satisfaction with the training that they received and far more on questions such as changes in work-
related behavior of the graduates, and what positive and negative factors they believe contribute to
improving the performance of their respective institutions and the sustainability of the capacity-building
support once it is taken over by the Afghan national institutions themselves.
A total of 14 FGDs were organized involving just over 200 participants. As well as providing useful
critical feed-back on the perceived benefits of the training and capacity development by those who
Page 17 of 105
received it, under Outcome One, these FGDs provided a statistically significant sample of both the
impact of the Programin terms of improving the functioning of the Afghan justice sector and reasonably
well-informed observers about the sustainability of this support once responsibility for this passes to
the national institutions themselves as envisaged in the JTTP’s transition strategy under Outcome Two.
Information from the FGDs was therefore used to triangulate some findings.
1.5 MANAGEMENT ANDLOGISTICAL SUPPORT
The international contractors were contracted for a total of 42 days each. This broke down as seven
days to draft and finalize the Inception Report, twenty days to conduct the field mission, ten days to
produce the first draft of the Final Report and five days to finalize the report after receiving comments
from IDLO. The phases of work of the evaluation is set out in Appendix Four.
The evaluation process was managed by IDLO’s JTTP M&E Unit, in close coordination with the
Program Team, ensuring that all information was provided and requests for feedback to the evaluation
were responded to in a timely manner. The JTTP M&E Unit reviewed the Evaluation Questions,
Evaluation Matrix and data collections tools, which were contained in anInception Report and provided
comments on these. The evaluation team submitted a first draft copy of their Final Report on 7 March
2016. IDLO provided a single set of consolidated comments on the draft on 17 March. Upon receipt
of these written comments, the evaluation team agreed to correct all documented factual errors and
inaccuracies and make changes related to the report’s structure, consistency, analytical rigor, validity
of evidence, and requirements in the TOR. After making the necessary changes, the evaluation team
undertook to submit the Final Evaluation Report for IDLO approval.
The JTTP Team in Afghanistan assisted with arrangements for the evaluation including travel and
appropriate logistic support and security. This included liaising with local stakeholders to schedule
meetings and providing interpretation services. After an initial discussion with key program personnel
in-country the evaluators submitted a detailed plan for the FGDs, interviews, and visits at the start of
the mission. IDLO undertook to respect the operational independence of the evaluation team to
organize their own methods of working without external interference. The evaluation team accepted
that its members were fully bound by IDLO’s security procedures and accepted that decisions of the
JTTP Chief of Party were final on all issues relating to security and logistics.
IDLO kept INL informed and involved at key times during the process, seeking input to draft
deliverables as appropriate. The evaluation team conducted one interview with two INL staff in Kabul
and one telephone interview with an INL staff member in Washington.
1.6 LIMITATIONS OF THE EVALUATION
The biggest limitation of the evaluation was the compressed time within which it was carried out, due
to delays in contracting the international contractors and the fact that both of these had taken on other
work and personal commitments by the time they were hired. In consultation with IDLO,the evaluation
team mitigated these limitations by hiring four national contractors to help with data collection and
organizing their time to maximize the number of interviews and FGDs that could be carried out in the
field. This problem exacerbated a number of more general limitations under which the evaluation was
conducted. The size and diversity of Afghanistan and the challenges faced by its justice sector,meant
that even the original time-scale within which the field mission was proposed to take place was quite
limited. The continually worsening security situation and the possibility of winter snow could have
made the logistics of organizing the field mission and – in particular – visiting the provinces very
difficult. The evaluation was also drawing on interviews from a range of sources, which are by their
nature subjective. Cultural factors could also have made it difficult to rely on tools such as self-
assessment questionnaires to provide quantitative and qualitative evidence that training of Afghan legal
professionals has actually improved their effectiveness.
Page 18 of 105
The evaluation team sought to mitigate these limitations and minimize possible biases through
triangulation of methods and data. This was achieved by interviewing a range of stakeholders at
different levels from a variety of institutions and reviewing a wide range of documents. The evaluation
team also reviewed independent reports on the state of the justice system in Afghanistan as well as
critical evaluations of efforts to strengthen its capacity, such as the current program. The team also
actively sought to interview people who could provide an independent perspective on these issues.
In the event, the weatherwascomparatively mild during the field mission and although there were some
problems with internal flights, this did not excessively disrupt the data collection. While there were a
number of security incidents and terrorist attacks during the time in which the field mission was being
conducted, the evaluation team was able to carry out its full schedule of external meetings and there
were no lock-downs or major disruptions of their work. Triangulation of data was relatively straight
forward because the evaluation team received very consistent information from different sources and
the evaluators were impressed at the open and honest accounts that they received from a wide range of
interviewees, including in the justice sector institutions and the JTTP staff themselves. These were
frank about the difficulties and challenges that they faced and prepared to be self-critical about failures,
while maintaining an underlying optimism about the overall progress and achievements accomplished.
1.7 CONFLICTS OF INTEREST
The two international contractors are independent and have no prior involvement with the
programming-setting, design, or overall management of the JTTP. Neither of the international
contractors on the evaluation team has ever worked for IDLO,although one has evaluated two previous
programs implemented by it. As discussed above the four national contractors do have prior
connections to both IDLO and the JTTP,but care was taken to ensure that they maintained an impartial
and objective approach to the data collection tasks assigned. This report was written solely by the
international contractors and its findings are their own responsibility alone.
Page 19 of 105
2 HISTORY AND OVERVIEW OF THE PROGRAM
The ‘Justice Training Transition Program’ (JTTP) is funded by the US State Department Bureau for
International Narcotics and Law Enforcement Affairs (INL) and implemented by the International
Development Law Organization (IDLO) in partnership with the Afghan government. The program
aims to increase the confidence of Afghan citizens in the justice sector. It focuses on achieving two
outcomes: (a) Beneficiary Afghan justice sector professionals achieve increased capacity and
competencies in delivering justice according to Afghan law; and (b) Afghan justice institutions are
capable of managing the sustainable implementation of training programs. It provides five types of
capacity-building support to Afghan legal professionals:
 Advanced Continuing Legal Education for Afghanistan (ACLEA) – classroom teaching of 10
subjects (now reduced to six) designed to increase legal and technical knowledge among
Afghan judges, prosecutors, defense attorneys, and investigative police officers from the
Criminal Investigation Division (CID) of the Afghanistan National Police (ANP). The course
requires 240 hours of intensive classroom sessions;
 Continuing Legal Education (CLE) outreach seminars, which address practicalchallenges that
justice professionals face during their practice;
 Needs–based mentorship sessions in which JTTP legal experts provide on–the–job technical
support to justice professionals; and
 One hundred and twentyhours of follow up advancedtraining for ACLEAgraduates(ACLEA–
Plus).
 A set of paper, electronic and multimedia training materials developed and distributed to
participants.
The program was established in January 2013 and was originally due to close in July 2015, but this was
extended by an agreement between IDLO and INL and it will now close at the end of March 2016. The
MTE provided a brief overview of previous attempts to strengthen the Afghan justice system, dating
back to the United Nations (UN) conference in Bonn, Germany, after the fall of the Taliban in October
2001.6
The Bonn Agreement specified that a Judicial Commission be established to rebuild the
domestic justice system ‘in accordance with Islamic principles, international standards, the rule of law
and Afghan legal traditions.’7
Hamid Karzai,who had beenappointed head of aninterim administration
at Bonn, was elected President of Afghanistan in 2004 and re-elected in 2009, serving two full terms in
office.
Although customary (non-state) justice mechanisms still account for a high proportion of dispute
resolutions in Afghanistan, the capacity of the major justice institutions have considerably strengthened
in the last decade,and there is some evidence that public confidence in official institutions has increased
in recent years.8
Annual surveys by the Asia Foundation, for example, show a rising willingness of
people to report on crimes and violence, and seekredressthrough the state system.9
Anew Constitution
was adopted in 2004, which sets out the hierarchy of domestic law, provides for a three-tiered judiciary
and sets out the functions and powers of the other main justice institutions.10
Since that date, a number
of significant new laws have been adopted or reviewed and drafted, including: a Criminal Procedure
Code; the Laws of Organization of the Courts, of the Office of the Attorney General and of the Ministry
6 MTE, 2014. See also Sheila Reed (Team Leader), Conor Foley and Hamayoun Hamed, Evaluation Report on
IDLO Italian-funded projects (2005 – 2008) for legal and judicial reform in Afghanistan,2008.
7 The Agreement on Provisional Arrangements in Afghanistan Pending the Re-establishment of Permanent
Government Institutions, II Legal framework and judicial system, [Hereinafter Bonn Agreement, 2001], Article
2.
8 For an overview see: Lenny Linke, Dispute Resolution in Afghanistan,Presentation to the Rule of Law Board
of Donors Meeting,undated
9 Afghanistan in 2015:a survey of the Afghan people,Asia Foundation,2015
10 2004. Constitution of the Islamic Republic of Afghanistan,Article one hundred and sixteen.
Page 20 of 105
of Justice; the Police Law; the Elimination of Violence Against Women Law; a Defense Attorney Law;
and the ‘Advocates Law’ of 2007, which established the Afghanistan Independent Bar Association
(AIBA).
A number of different international organizations have developed justice sector training and capacity-
building programs over the last decade. Since 2008, considerable financial support has been given to
justice sector reform and capacity-building. Training by different actors initially developed in an ad
hoc mannerand sufferedfrom lack of coordination.11
Various attemptshave been made to improve this
but it still remains fragmented. Tensions also arose due to questionable judicial appointments by the
Karzai government, which made some international organizations reluctant to work with some Afghan
justice institutions.12
Presidential elections in 2014 eventually resulted in an agreement to form a power-sharing
administration betweenthe two leading candidates, AshrafGhani, who became President,and Abdullah
Abdullah, who was made a Chief Executive Officer, with powers similar to that of a prime minister.13
The first round of the election took place in April 2014 and the second in June. The full details of the
deal were not agreed until September 2014 and this led to an effective paralysis of some parts of the
government for more than six months. The advent of the newadministration also led to senior personnel
changes in Afghan ministries and justice institutions which, as will be discussed below, has clearly
impacted on the JTTP’s transition strategy.
The MTE was positive about the quality of the training and coaching delivered by the JTTP. It noted
that while the delivery of some courses had been delayed and there were some areas where the training
could have been improved, it had generally been effectively implemented in terms of quantity and
quality. The MTE also found that the JTTP had been delivered in a cost-efficient manner, by reducing
the costs of international staff and associated security costs, while at the same time ‘nationalizing’ its
training team to its Afghan staff. It noted that the JTTP had developed useful internal monitoring and
evaluation mechanisms, but it did not at that time have the capacity to assess the impact of the training
in the targeted institutions. It stated that without such monitoring it was very difficult to identify the
direct and indirect, medium and long-term results of such projects, nor to analyze how they can
positively contribute to the goal of strengthening the Afghan justice system and restoring public
confidence in it. The MTE made a number of recommendations about how the JTTP could become
more proactive in looking for linkages between its work on strengthening the capacity of the justice
sector and those organizations that are involved in monitoring its performance in relation to justice and
human rights.
The MTE was considerably more critical about the progress achieved under Outcome Two of the JTTP.
It acknowledged that the extent of engagement in the target institutions varied from institution to
institution and progress had been made in some areas. It also noted that many of the delays were due
to a variety of factors, some of which were clearly beyond the JTTP’s control to influence, such as six
months of impasse in the government after the disputed elections in 2014. The MTE found that despite
these difficulties the JTTP had made considerable progress and a sustainable transition could still take
place ‘if the project is granted a contract extension and supported by a follow-up considerably scaled
down transition project.’ The MTE made a number of recommendations, which it was hoped would
positively assist the JTTP strengthen its activities in the second part of the program and to address some
of the systemic weaknessesofthe Afghan justice system. This evaluation will review the progress since
the MTE focusing, in particular, on the issues described in the scope and purpose section of this report.
11 J. Alexander Their, Re-establishing the Judicial System in Afghanistan, Center on Democracy, Development
and the Rule of Law, Stanford Institute for International Studies, September 2004.
12 International Crisis Group, Reforming Afghanistan’s broken judiciary,ICG, Asia Report N°195 – 17 November
2010.
13 Guardian, ‘Afghanistan presidential candidates strike power-sharing deal’, 20 September 2014.
Page 21 of 105
3 MAIN EVALUATION FINDINGS
RELEVANCE
3.1 EVALUATION QUESTION 1:TO WHAT EXTENT WAS THE PROGRAM THEORY
OF CHANGE ABLE TO RESPOND TO THE CHANGING NEEDS ANDCONTEXT
OF THE AFGHAN JUSTICE SECTOR, INCLUDING CHANGES TO THE
POLITICAL AND SECURITYCONTEXT?
The JTTP contains two objectives. The first of these is to provide direct capacity-building support
activities – through training and coaching – to individual Afghan legal professionals within four target
institutions: the Supreme Court, the Attorney General’s Office (AGO), the Ministry of Justice and
Ministry of Interior. The second is to assist Afghan justice institutions so that they will be capable of
taking on responsibility for building the capacity of their own personnel as part of a comprehensive and
sustainable strategy of Afghan institution-led continuing legal education. The underlying assumption
of the program is that this individual capacity development support can lead to demonstrable and
sustainable improvements in the administration of the formal justice sector in Afghanistan, which, in
turn, can ultimately lead to greater public confidence in this system.
All Judges in the country come under the Supreme Court for the purposes of training. The AGO
contains both Investigative Prosecutors and Trial Prosecutors – both of whom are supported by the
JTTP. The JTTP’s support in the Ministry of Justice is for Defence Lawyers in the Legal Aid
Department (LAD) and its support for the Ministry of Interior is for its Criminal Investigation Officers
(CID)within the police. The JTTP’sfirst objective (Outcome One)is to provide direct capacitybuilding
support for individual Afghan legal professionals. Its second objective (Outcome Two) is to develop a
transition strategy to assist the above institutions so that they will be capable of taking on responsibility
for building the capacity of their own personnel as part of a comprehensive and sustainable strategy of
Afghan institution-led continuing legal education.
As the MTE noted, the JTTP corresponds with both the Afghan government’s own goals and those of
the US Mission in Afghanistan, its largest donor. It is also in line with IDLO’s Global Strategy Plan14
and the findings of the IDLO survey and publication: Women in Afghanistan’s Justice Sector.15
In
interviews with representatives of the target institutions this evaluation team was repeatedly told that
the capacity development support that they had received from the JTTP was relevant and appropriate
to their needs.16
This view was also expressed in the FGDs, with many participants giving examples
of how they support that they had received had helped them to be more effective in their work. During
all interviews and FGDs during the current evaluation it was repeatedly stated that the criminal justice
system is improving in Afghanistan and that Afghan legal professionals are using the knowledge that
they have acquired from the JTTP to improve the quality of their work at both an individual and
institutional level. The evaluation team spoke to over 100 individual interviewees and over 200
participants in the FGDs. This can be seen as a statistically significant cross-section of Afghan legal
opinion and the view was expressed consistently by almost everyone.
The courses that the JTTP provides – which will be discussed further below – are based on detailed
needs-assessments and have been developed and modified to meet the specific needs of the target
beneficiaries. Since 2014, all courses have been reviewed and commented on by the institutions
14 International Development Law Organisation: Creating a Culture of Justice Strategic Plan 2013-2016
http://www.idlo.int/sites/default/files/Strategic%20Plan%202013-2016.pdf, accessed 20 February 2016.
15IDLO: Out of the shadows,onto the bench: Women in Afghanistan’s Justice Sector, p.6
http://www.idlo.int/publications/womens-professional-participation-afghanistans-justice-sector-challenges-and,
accessed 20 February 2016.
16 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
Page 22 of 105
themselves, to ensure relevance to the work of their professionals. This also serves to ensure these
institutions have the necessary curricular tools to deliver training on their own.
The JTTP theory of change assumed that the target institutions were willing to take over full
responsibility for the capacity development activities themselves and would allocate sufficient
resources in terms of personnel, funding and equipment within the original planned duration of the
program.17
This, however, proved over-optimistic for reasons that will be discussed further later in this
report. The JTTP subsequently modified this strategy and its logical framework was updated according
to the specifications of the transition strategy developed under Outcome Two. The transition strategy
was based on the assessed needs of the target institutions, was drawn up in full consultation with them
and was relevant to their needs and capacity. This was confirmed to the evaluation team in meetings
with representatives of the targetinstitutions and is also borne out by its program documents. The JTTP
now has ten outputs in comparison with the previous eight (five under Outcome One and three under
Outcome Two, respectively). A list of 30 indicators was developed as a part of the logical framework
reflecting both quantitative and qualitative results. 18
The defined indicators continue to reflect
adequately what the program is doing and they also measure progress made towards the achievements
of expected results.
The original logical framework was quite specific regarding the activities under Outcome One, but
rather imprecise under Outcome Two. The overall objective of the latter was clearly defined but there
was no roadmap, the indicators were very broad and no timelines were given. This should not
necessarily be seen as a criticism, however, as a case can be made that the more detailed strategy for
the transition neededto be developed during the JTTP’simplementation. Memoranda of Understanding
(MoUs) have now been signed with all four national justice institutions, which clarify the transition
related co-operation for the period of the JTTP and potentially beyond, should another project continue
to be implemented with the same target institutions.19
The first Monitoring and Evaluation (M&E) Plan was developed in June 2013 and an Evaluation
Strategy was finalized September 2013.20
This has helped the program to develop its overall strategy
in furtherance of the above objectives, taking into account the changing needs of the Afghan justice
sector, although it seems that it would have been helpful to have had these systems in place earlier in
the JTTP’sexistence. The M&E systemis connected to the US Government’s Afghanistan Performance
Monitoring Plan (PMP)21
and provides a system of regular reporting to INL. The system also provides
periodic and regular information to inform the JTTP’s own program management on progress made
towards intended objectives, as well as to identify deviations from these targets and support decision
making processes to strengthen the program’s effectiveness. The M&E system also fulfils donor
reporting requirements, which include biweekly, monthly and quarterly reports.22
The program’s M&E system also evaluates the effectiveness of the JTTP’s individual capacity support
activities, basedon the IDLO’sstandard evaluation approach to trainings basedon the Kirkpatrick four-
level model. As will be discussed further below, the JTTP evaluates not just the perceived quality of
the training and acquired knowledge, but also how trainees have changed their behavior, and applied
the knowledge they have received in their work. It has also begun to measure the impact of these
improvements on the quality of justice provided by the institutions themselves.
17 JTTP Logic Model July 2013 and reconstructed theory of change (undated)
18 JTTP Monitoring Evaluation Plan and Logframe of 21 April 2015.
19 Ministry of Interior January 2015, Ministry of Justice April 2014, Supreme Court undated,Attorney General’s
Office January 2015.
20 JTTP M&E Strategy, version 3, 12 September 2013. See also: JTTP M&E Framework, 24 February 2014; and
JTTP M&E System and Approaches,22 July 2014.
21 https://www.usaid.gov/afghanistan/performance-monitoring-plan, accessed 20 February 2016.
22 JTTP, Monitoring and Evaluation Concept Note,IDLO, undated,p. 1.
Page 23 of 105
The MTE noted that some of the JTTP assumptions on the outcome level did not ‘reflect all necessary
conditions which are essential for achievement of expected results.’ It cited as an example, ‘a political
commitment to improve the justice sectorsystem and create favorable conditions for fair administration
of justice, political will to fight corruption and allocate sufficient resources (personnel, funds and
equipment) for ongoing professional development’.23
These assumptions were clearly over-optimistic.
Afghanistan’s governance structures remain weak and became totally paralyzed during the 2014
election. The country’s Budget is parlous and its government is facing a fiscal crisis, while the security
situation is visibly deteriorating every year. However, it is precisely because of the challenges that
Afghanistan faces in relation to weak rule of law, corruption, intimidation and violence that makes
programs such as the JTTP so necessary and relevant.
A number of senior justice sector officials, stated that they felt the training and coaching provided by
the JTTP had tangibly improved not just the skills of individual professionals but the culture of the
institutions as well. These officials are also noticeably taking ownership of the transition process and
some candidly acknowledged that the delays had been the responsibility of their own institutions. 24
Afghanistan’s Deputy Minister for Justice,for example, stated that ‘the JTTP has done wonderful work
in helping us to change ourselves. Of course we really needan externalevaluation of our own institution
to assess whether we are improving our own services to the Afghan people!’ 25
The Head of
Afghanistan’s Supreme Court reportedly told IDLO staff in Rome that ‘you did not just give us a raft,
you helped us to build a bridge.’26
There are,however, two areas where this evaluation team believe that IDLO may wish to reflect on the
relevance of the JTTP’s activities when devising future programming. The first is the program’s
exclusive focus on the official state criminal justice system. Although the reasons for this are
understood in terms of program design and donor priorities, this evaluation team believes that ignoring
both customary law and civil law significantly reduced the JTTP’s overall relevance in seeking to
strengthen the capacity of the Afghan justice sector. It is widely accepted that around 80 per cent of
legal disputes in Afghanistan are settled through customary mechanisms: Shuras and Jirgas.27
The role
and importance of customary law increased in Afghanistan due to the break-down of the official
institutions of law and order during the conflict.28
While the problems with the customary system are
well-known,29
the evaluation team believes that the decision to ignore it completely when designing the
JTTP,significantly reduced the program’s scope in addressing how justice is actually administered and
perceived in Afghanistan.
Linked to this is the exclusive focus on criminal law, which, by ignoring civil law entirely, meant that
the JTTP was unable to address issues such as land and property rights, commercial law, and divorce,
custody and inheritance rights. Given that the most contact that most people have with legal systems
throughout the world is on such issues and they will generally judge the effectivenessof a justice system
by its ability to handle them, it seems strange to omit this from a program whose overall goal is to
strengthen public confidence in it. The omission is particularly regrettable in Afghanistan where the
division between criminal and civil law is more blurred and where some disputes, particularly related
to land and property, can encompass both civil and criminal issues.30
23 MTE, 2014, p.23.
24 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
25 Interview conducted in Kabul on 9 February 2016.
26 Interview conducted by Skype in Rome on 4 March 2016.
27 Lenny Linke, Informality and formal land administration, Working Paper, Policy advisory group on land –
Sub-working group on formal/informal land issues,UNAMA Rule of Law Working Group, no date.
28 Ali Wardak, ‘The Tribal and Ethnic Composition of Afghan Society’ in Afghanistan:Essential Field Guides to
Humanitarian and Conflict Zones (Second Edition), edited by Edward Girardet & Jonathan Walter, Geneva:
Crosslines Ltd, 2003.
29 Dr. Martin Lau, Afghanistan’s Legal System and its Compatibility with InternationalHuman Rights Standards,
The International Commission of Jurists, November 2002.
30 Conor Foley, A Guide to Property Law in Afghanistan,UNHCR/Norwegian Refugee Council, 2005.
Page 24 of 105
3.2 EVALUATION QUESTION 2:HOWABLEWAS THE IMPLEMENTING
ORGANIZATION TO RESPOND TO THEIDENTIFIEDNEEDS/PRIORITIES?
As discussed above, a number of different international organizations have provided considerable
capacity-building support to the Afghan justice sector. In 2005, INL established the Afghanistan
Justice Sector Support Program (JSSP), which is implemented by a private sector contractor
PAE.31
This has trained and mentored justice sector officials, helped to restructure criminal justice
institutions and improved court administration and facilities. The JSSP established six provincial
training teams located in Herat,Balkh, Nangarhar, Bamiyan, Kabul, Kunduz and Paktia, as well as a
Kabul-based mobile team. These ran courses that brought together different groups of Afghan legal
professionals and subsequently created ‘mentoring networks’ with advisors to visit practitioners at their
offices to monitor and evaluate their progress. The JSSP also developed a Case Management System
(CMS) to track the progress of criminal cases in the Afghan justice system.
With the commencement of the JTTP, at the start of 2013, IDLO assumed responsibility for the
professional legal training and individual capacity-development support previously provided by JSSP,
while the JSSP/PAE remained responsible for other institutional support for these ministries and the
CMS.32
In its original program proposal for the JTTP, IDLO stated that the program would ‘be
implemented using a holistic approach to ensure that all partners involved will mutually benefit from
the IDLO fully-coordinated and information-sharing approach.’33
The evaluation team is not aware of whether or not a baseline assessment was carried out before the
start of the JSSP nor whether organizational assessments had been carried out of the target institutions.
Aswill be discussed below in the section of this report on Impact,the JTTP did commission a consultant
to produce a base-line assessment of the capacity of these institutions to implement training themselves
as well as some aspects related to curriculum development.34
The study was eventually ‘valued
unsatisfactory’, but it did reveal ‘inconsistencies between information provided by the institutions and
the JSSP saturation study regarding reliable number of justice professionals actually employed in each
institution.’35
The IDLOproposal statedthat the JTTP would be based,in part,on the training component of the JSSP.
It noted that this had been ‘well-received and pre- and post-tests conducted in training programs have
shown that participants have benefited in terms of increased knowledge and skills.’36
IDLO would
‘continue the positive aspects of the JSSP training programs, in particular the Advanced Continuing
Legal Education Course for Afghanistan (ACLEA) training course and other shorter workshops which
have been previously designed and implemented.’ It would, however, ‘move beyond the previous
iterations of the JSSP, with an increased emphasis on results evaluation, as well as a clear focus on
sustainability through transition to Afghan ownership of training programs.’37
It also undertook to
conduct an ‘assessmentofthe delivery modalities with a view to enhancing, if necessary,its pedagogical
quality and impact’ and to train all existing ACLEA Afghan trainers on IDLO’s own training
methodology.38
31 See JSSP website http://jsspafghanistan.com/ accessed 18 February 2016
32 Ibid.
33 International Development Law Organization, Completing the transition in Afghanistan:Justice Training
Transition Program (JTTP), program proposal submitted to U.S. Department of State Bureau of International
Narcotics and Law Enforcement for Afghanistan and Pakistan,IDLO, 21 December 2012, [Hereinafter IDLO
Proposal, 2012] p.8.
34 Baseline Assessment Report of the Justice System of Afghanistan and Justice Sector Capacity Building, June
2013.
35 JTTP Quarterly Report Jul-Sep 2013, 31 Oct 2013.
36 Ibid., p.8.
37 Ibid.
38 Ibid., p.20.
Page 25 of 105
The JTTP was clearly designed to build upon the INL’s JSSP regional training component for Afghan
legal professionals and it inherited both many staff and facilities from the latter. This brought savings
in terms of continuity and institutional knowledge. It also enabled the JTTP to continue – almost
without interruption – the training component activities that had originally been designed under the
JSSP while new staff members were being recruited and the curriculum and teaching methodology was
being revised and developed. It appears that the number of training and coaching activities that the
JTTP carried out was also based on the numbers carried out under the JSSP and this also defined its
geographical scope. As will be discussed further later in this report, the JSSP was able to deliver – and
even exceed its original targets and so these can be said to have been realistic, at least as far as Outcome
One was concerned. The evaluation team, however, encountered two criticisms, from various sources,
relating to the way in which the JTTP was essentially passed from one implementing organization to
the other, as well as a broader concern about the way in which what had previously been a single
program was divided between two different implementing organizations.
In January 2014 the US Special Inspector General for Afghanistan (SIGAR) published an audit report
on reconstruction support for Afghanistan’s justice sector, which made a number of specific criticisms
of the JSSP under its previous private sector management,but also raised concern about the decision to
transfer the program from PAE to IDLO. It questioned, in particular, IDLO’s capacity to manage such
a large program and referred to challenges facing its headquarters in Rome at the time, which it claimed
raised ‘serious questions about the future sustainability of the organization’.39
The MTE did not encounter any evidence to justify the fears and neither did the current evaluation. As
will be discussed further below, in the section of this report on Efficiency, IDLO did succeed in scaling
up its administrative and logistical support for the JTTP, which, although large by IDLO’s previous
standards is actually quite modest when compared to similar programs in Afghanistan. Indeed, it
appears that the extra capacity that IDLO has developed to implement the JTTP could be used to take
on other larger scale programs in both Afghanistan and elsewhere. The overwhelming majority of
interviewees spoken to by the present evaluation team stated that they considered the capacity-
development support provided by the JTTP to be a considerable qualitative improvement to that
provided under its predecessor. This view was expressed in interviews with Afghan justice sector
officials, UN staff, representatives of non-government organizations (NGOs) and participants in the
FGDs who had received trainings under both programs.40
Indeed it wasvery noticeable that institutions
such as the Supreme Court and the AGO, who had previous expressed some skepticism about aspects
of the capacity-building support that they had received from the JTTP during the MTE, now declared
themselves to be far more satisfied with the program’s implementation.41
The other criticism that the evaluation team heard about the handover focused on perceived weaknesses
in the program that the JTTP inherited. In particular, it was stated – from a range of interviews and in
FGDs – that the JSSP had placed too great an emphasis on the number of trainings and participants in
each course,rather than the quality of their legal content and delivery.42
The terms of reference of both
the MTE and the current evaluation preclude a more detailed examination of the original JSSP training
program and so the evaluation team is in no position to make findings on these issues. As will be
discussed below, under the section of this report on Effectiveness, however, there is evidence that the
quality of the capacity-development support provided by the JTTP has improved during the period of
its implementation and has also improved since the MTE.
39 Special InspectorGeneral for Afghanistan, Reconstruction Support for Afghanistan’s Justice Sector:State
Department Programs Need BetterManagement and Stronger Oversight, SIGAR 14-26 Audit Report/State
Department Support of Afghan Justice Sector, January 2014.
40 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
41 Interviews conducted in Kabul on 9 and 10 February 2016. See also MTE, 2014, p.33.
42 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
Page 26 of 105
The evaluation teamalso heard from both the donor and some representativesof Afghan national justice
sector institutions that IDLO was the right strategic partner for INL to choose in implementing the
JTTP, particularly given the emphasis on sustainable transition contained in Outcome Two. IDLO is
an inter-governmental organization (IGO), with a 30-year history of delivering legal education and
capacity support in the global south. It has been working in Afghanistan since 2001, implementing a
number of programs in the field of justice sector and legal reform in partnership with the Government
of Afghanistan, which made their state a Member Party to IDLO in November 2012.43
IDLO also has political and diplomatic leverage as an IGO. IDLO’s Director General, Irene Khan,
attended the launch of the program in March 2013 in Kabul. In May 2015 she visited Afghanistan again
where she held bilateral meetings with senior government ministers including: the Minister of Justice,
the Minister of Women’s Affairs, the Acting Chief Justice, the Acting AGO, the Deputy Minister of
Interior, the Chairperson of the Afghanistan Independent Human Rights Commission and both the
President and Vice President of the country. Khan also met with the UN Deputy Secretary General’s
Special Representative for Afghanistan, the Senior US Civilian Representative in the country and senior
INL officials.44
It is clear that IDLO has some comparative advantages in implementing a program
such as the JTTP. While perhaps the policies and procedures of an IGO may be more unwieldy than
the flexibility and rapid responsiveness of a for-profit contractor,these would seem to be far more than
outweighed by the commitment to long-term sustainability that IDLO can provide to one of its own
member states.
The quite different organizational cultures of the two different implementing organizations, may,
however, have inhibited communication between them. It is clear that the claim in the IDLO program
proposal that it would establish a ‘fully-coordinated and information-sharing approach’ between the
JTTP and the JSSP was not always achieved. There were serious breakdowns of communication
between the two organizations at particular moments in time, some of which may have been related to
internal staff turn-over within the JTTP. The appointment of a new JTTP Chief of Party in October
2014 does, however,seemto have easedthese difficulties and both organizations reported a much better
and closer working relationship in the second part of the program’s implementation.
The JTTP Project had more or less kept the geographical coverage of the JSSP but gradually reduced
the international presence in the field offices. With the closing of the Provincial offices in Badakshan
and Kunduz though the reachout to beneficiaries wasnegatively affected.45
The Mobile Teamwasused
to reachout to difficult geographical areas.46
Trainings held at the provincial hubs for surrounding areas
has allowed for greater participation of women.47
The decentralised implementation and methodology
was appreciated and praised by the beneficiary institutions as one of the greatest assistance in the
transition both serving as a link between the local institutions in Kabul and the provinces and as
complementary capacity building when local budget funds are lacking.48
The project has taken into
account ethnic and linguistic differences when necessary and has paid attention to the specificities of
different regions (certain provinces more progressive than others). The M&E has kept geographically
differentiated data for each type of training and institution. Relevant both for cost-saving and
minimising security risks in the filed was the frequent co-location of training venues and
accommodation. It was though often difficult for female participants to travel and to plan overnight
stays in province hubs.49
43 IDLO Homepage, where we work, Afghanistan,
http://www.idlo.org/Archive/WhereWeWork/afghanistan/Pages/default.aspx,accessed 18 February 2016
44 JTTP Quarterly Report Apr-Jun 2015, 31 July 2015
45 JTTP Quarterly Report December 2015 p.
46 JTTP Quarterly Report December 2015 p. 9.
47 Ibid. p. 10.
48 Interview with the Supreme Court JED
49 Focus Group Discussion for women 2 February 2016
Page 27 of 105
EFFECTIVENESS
3.3 EVALUATION QUESTION 3:WAS THE JTTP SUCCESSFUL IN DELIVERING THE
EXPECTED OUTPUTS WITH A FOCUS ON INCREASED QUALITYAT THE SAME
TIME?
By December 2015, the JTTP had delivered over 382 legal training courses to more than 8,900
graduates,in total.50
This included: 69 ACLEA courses for almost 2,000 participants; 44 ACLEA Plus
for 694 participants, and 247 CLE for more than 5,990 participants. The JTTP’s capacity-building
support under Outcome One involved delivering just over 4,500 days and 27,000 hours of training.
These figures include some double-counting, because if an individual has participated in an ACLEA
course and a CLE, she or he will be counted twice, and JTTP staff estimated the individual number of
recipients of this capacity-development support was a bit over 5,000 people.51
Around 13 per cent of its
direct beneficiaries were women and 87 per cent men.
As well as delivering the standard ACLEA course, the JTTP developed a special ACLEA tailored for
members of the Supreme Court. This has been delivered to 305 participants, who are counted within
the numbers given above for its ACLEA courses. The program also provided around 23,300 hours of
follow-on coaching in just over 13,000 individual sessions to 2,355 recipients. The JTTP hasmaintained
a monthly record of the numbers of training courses that it delivers in its regular reports and these show
that it has completely fulfilled – and in some cases exceeded – all of its original numerical targets under
Outcome One. The JTTP has also conducted twenty-two Training of Trainer (ToT) courses for 294
participants and has certified 146 GIROA Trainers.52
These trainers have themselves delivered over
11,300 of training under Outcome One. The JTTP has delivered a total of 16 trainings, and 504 training
hours to 219 potential future trainers under Outcome Two and provided further personal support to
these through 619 coaching hours. It has also refurbished 16 training rooms and is in the process of
handing over curriculum material and training handbooks and manuals.
The JTTP has become more effective at developing co-ordination and outreach mechanisms with both
the national Afghan institutions and other international stakeholders. Reporting, and monitoring and
evaluation systems have been developed, which have been used for effective communication with the
donor, strategic planning and measurement of the program’s results. The JTTP has also become
increasingly effective at working with the target justice sector institutions to define the assistance that
it can provide towards establishing and supporting operational in-house training departments. This has
included helping each institution to establishing governance arrangements and helping to recruit and
train instructors for the units once these have been approved.
The MTE noted that the JTTP could have been ‘more pro-active in looking for linkages between its
work on strengthening the capacity of the justice sector and those organizations that are involved in
monitoring its performance in relation to justice and human rights.’53
It noted that while the JTTP was
a member of the Criminal Law Reform Working Group (CLRWG), and participated ‘in occasional
policy and planning meetings with INL, the JSSP, and Afghan line-ministries, it seemed much less
present in the wider international community of agencies represented by the UN Country Team,
individual UN agencies, and the World Bank.’54
These criticisms may have been unfair at the time, since the JTTP’s Chief of Party had resigned
shortly before the MTE was carried out and IDLO was in the process of recruiting a new one. The
new Chief of Party, who joined JTTP in October 2014, has significantly transformed this situation and
is widely known and respected within by both Afghan and international actors. She clearly takes her
50 JTTP Report Card dated January 2016 (Reporting period January 2013-December 2015)
51 Interview carried out in Kabul on 7 February 2016.
52 JTTP Report Card dated January 2016 (Reporting period January 2013-December 2015)
53 MTE, 2014, p.8. Executive Summary.
54 Ibid., p.50.
Page 28 of 105
‘Ambassadorial’ role extremely seriously and engages in active and ongoing work to raise awareness
about the JTTP’s transition efforts, and seek support on key transition objectives. She holds regular
cycles of meetings with the target national justice sector institutions, and also actively participates in
UNAMA justice sector and rule of law meetings. One member of the evaluation team attended one
such meeting, which included representatives of UN Agencies; the French, German, British and US
Embassies, the European Union, USAID,Italian Cooperation, the UK Department for International
Development (DfID),the US army, the North Atlantic Treaty Organization (NATO),the US Institute
for Peace (USIP),the Asia Foundation, the Afghan Analysts and various civil society groups.55
The
importance of maintaining and cultivating such contacts is now clearly understood within the JTTP.
The JTTP has also established a Cordination working group for International organizations/donors
working with the Supreme Court. This has met periodically and its objective is enable the Judicial
Education Department (JED) to manage trainings conducted by various organizations. IDLO is also a
member of the International Training Coordinatioms group, which consists of international advisors
who mentor the Ministry of Interior’s Training Group.
The MTE had also recommended setting up a ‘Transition Unit’ of international and national staff who
would be relinquished from their activities coordinating the delivery of training to work full-time on
the transition strategy.56
An Institutions Team was subsequently created in 2015, and now consists of
two international and eight national JTTP staff, plus administrative support. JTTP staff members are
now embedded in each institution ensuring effective day-to-day cooperation over the transition
arrangements. The JTTP has also developed a clear understanding of the structures and internal
dynamics of each institution and this has helped it to devise tailored approach strategies along with
back-up plans and alternatives when these have been blocked or stymied by external events. As one
senior international JTTP staff member explained:
When we found that one of the institutions that we were working with did not have any budget
in place for training the following year we decided to re-sequence our activities and began to
work in the margins to see what could be accomplished without this. We started working on
their M&E Records Database,to identify who had already received training, and cooperated
more closely with other actors to fill any training gaps and establish coordination mechanisms.
The idea was to put the elements of the training department in place for when budget approval
finally came. We learnt that you always have to have a plan B.57
In early May 2015 IDLO convened a transition workshop to engage its institutional stakeholders at
management and operational levels within the Supreme Court, the AGO and the Ministries of Justice
and Interior. This was attended by senior staff within the training human resources and planning
structures of the four institutions and was used to review transition plans, agree on key deliverables and
milestones. The meeting was also attended by the World Bank, PAE, European Union Police Mission
(EUPOL), UN Development Programme (UNDP) and Deutsche Gesellschaft für Internationale
Zusammenarbeit (GIZ). The JTTP has also facilitated both the Ministry of Justice and the Supreme
Court to convene meetings to present their own training institutional work plans and budgets, and share
scheduling information and participant data with other international organizations, to avoid repetition
and duplication of training activities.
Underpinning such efforts is an understanding within the JTTP of the need to cultivate long-term
personal relationships with key players in each target institution. This is particularly important given
Afghanistan’s political and institutional culture. The JTTP has been able to root its plans in Afghan
realities, which are often more messy and imprecise than can be adequately presented in a graph or
matrix. The evaluation team believes that such efforts, while time-consuming and often arduous, are
essential to long-term sustainability and are too often neglected by international organizations, who
over-prioritize quantitative achievements.
55 Rule of Law Donors Board Meeting, Kabul, 10 February 2016.
56 MTE, 2014, p.61.
57 Interview conducted in Kabul on 7 February 2016.
Page 29 of 105
In its original program proposal IDLO had stated that it would continue to deliver the basic ACLEA
course, and other shorter coursesand workshops, which had beenpreviously designed and implemented
under the JSSP. It would also design and deliver an additional, more in-depth, ‘ACLEA-Plus’ course,
for past ACLEA graduates and undertake an ‘assessment of the delivery modalities’ of all courses with
a view to enhancing their pedagogical quality and impact’.58
It also noted that, given the baseline from
which it was starting and the well-known challenges facing the justice sector in Afghanistan it was
simply not realistic ‘to plan for a situation whereby in less than three years all Afghan citizens have
access to a court which is able to deliver criminal justice in-line with the law and international human
rights standards.’ 59
The JTTP instead aimed ‘to provide a significant number of justice sector
professionals with the capacity to fulfill their duties as defined by the Afghan Constitution and related
laws and international commitments, so that their improved capacity builds public confidence in the
justice sector.’60
The legal capacity building component of the JTTP was extensively covered by the MTE.61
This noted
that although curriculum and lesson plans for each topic of the ACLEA and CLE existed, lectures were
initially delivered class room style, there had been virtually no hand-outs or overhead visual materials
provided and that the training delivery did not follow basic andragogical rules to allow greater
interaction between the trainer and participants. JTTP staff members had begun to make changes to the
course delivery, presentation styles, and hand-outs from the start of the program although much of this
was done in an ad hoc manner by individual training teams.62
JTTP started to revise the ACLEA curriculum in late 2013 and this was finalized in April 2014. The
new course was structured in six topics, reduced from 10. JTTP also finalized the new ACLEA-Plus
curriculum in June 2014 and started piloting courses that November.. The MTE noted that there had
been ‘good consultation about the development of the new curriculum’, and stated that ‘the content
sounds encouraging, although the fact that these courses are still in the development stage so late in the
program’s development is a cause for concern.’63
Since the field mission for the MTE took place in
August and September 2014, the evaluation team was unable to observe any of these courses.
The present evaluation team was,however, able to observe both ACLEA and ACLEA-Plus courses and
also to discuss their development in individual interviewees and FGD participants. It was also able to
review the most recent version of the JTTP’s curriculum and discuss the plans for handing this over to
the Afghan national institutions with the JTTP’s Curriculum Unit. The evaluation team was extremely
impressed by the parts of the ACLEA-Plus course that it observed. The emphasis of the training is on
the practicalapplication of skills, with groups of Afghan legal professionals – from different institutions
– working together on particular cases. In the ACLEA-Plus course that one evaluation team member
observed, one group of CID Officers was working together with Investigative Prosecutors from the
AGO to gather evidence during a crime scene investigation (CSI),while Trial Prosecutors and Defense
Lawyers discussed together how evidence collected could be presented at trial. It was explained that
Judges would be brought into the course the following week to preside over a mock trial, which all
course participants would be able to witness, and where all the previous procedures and evidence
gathered would be reviewed in courtroom conditions.64
58 IDLO Proposal, 2012, p.20.
59 Ibid.
60 Ibid.
61 For further discussion see MTE, 2014, pp.27-33.
62 Ibid., pp.31-3.
63 MTE, 2014, p.32.
64 Interviews and observations in Kabul on 9 February 2016.
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF
IDLO Final Evaluation 23 April Report Final CF

Mais conteúdo relacionado

Destaque

CarlissScaifeMarketingResume
CarlissScaifeMarketingResumeCarlissScaifeMarketingResume
CarlissScaifeMarketingResumecarliss scaife
 
Untitled Presentation
Untitled PresentationUntitled Presentation
Untitled PresentationShannon Minns
 
왓치베이비 사업계획서
왓치베이비 사업계획서왓치베이비 사업계획서
왓치베이비 사업계획서민구 강
 
Kelas 12 kimia gas mulia dan hologen kelompok 6
Kelas 12   kimia gas mulia dan hologen kelompok 6Kelas 12   kimia gas mulia dan hologen kelompok 6
Kelas 12 kimia gas mulia dan hologen kelompok 6bellakrismawati
 
Présentation Etude EMA by Come&Stay et SNCD
Présentation Etude EMA by Come&Stay et SNCDPrésentation Etude EMA by Come&Stay et SNCD
Présentation Etude EMA by Come&Stay et SNCDCome&Stay
 

Destaque (7)

CarlissScaifeMarketingResume
CarlissScaifeMarketingResumeCarlissScaifeMarketingResume
CarlissScaifeMarketingResume
 
Apuntes.Biología II
Apuntes.Biología IIApuntes.Biología II
Apuntes.Biología II
 
Untitled Presentation
Untitled PresentationUntitled Presentation
Untitled Presentation
 
Meios de comunicao
Meios de comunicaoMeios de comunicao
Meios de comunicao
 
왓치베이비 사업계획서
왓치베이비 사업계획서왓치베이비 사업계획서
왓치베이비 사업계획서
 
Kelas 12 kimia gas mulia dan hologen kelompok 6
Kelas 12   kimia gas mulia dan hologen kelompok 6Kelas 12   kimia gas mulia dan hologen kelompok 6
Kelas 12 kimia gas mulia dan hologen kelompok 6
 
Présentation Etude EMA by Come&Stay et SNCD
Présentation Etude EMA by Come&Stay et SNCDPrésentation Etude EMA by Come&Stay et SNCD
Présentation Etude EMA by Come&Stay et SNCD
 

Semelhante a IDLO Final Evaluation 23 April Report Final CF

Organization assessment guide
Organization assessment guideOrganization assessment guide
Organization assessment guiderajatoba1
 
YJC Summer Guidelines
YJC Summer GuidelinesYJC Summer Guidelines
YJC Summer GuidelinesEmily Chatten
 
Social Impact Evaluation Report
Social Impact Evaluation ReportSocial Impact Evaluation Report
Social Impact Evaluation ReportColin Richardson
 
Conducting BIAs for UASC in Cairo (RefugePoint 2012)
Conducting BIAs for UASC in Cairo (RefugePoint 2012)Conducting BIAs for UASC in Cairo (RefugePoint 2012)
Conducting BIAs for UASC in Cairo (RefugePoint 2012)Devon Cone
 
Challenges and Changes The Political Economy of National Development Banks in...
Challenges and Changes The Political Economy of National Development Banks in...Challenges and Changes The Political Economy of National Development Banks in...
Challenges and Changes The Political Economy of National Development Banks in...Mondher Khanfir
 
Report iatc-final
Report iatc-finalReport iatc-final
Report iatc-finalagjinopulli
 
La calidad de la AOD
La calidad de la AODLa calidad de la AOD
La calidad de la AODManfredNolte
 
Advancing regional recommendations on the post 2015 development agenda(1)
Advancing regional recommendations on the post 2015 development agenda(1)Advancing regional recommendations on the post 2015 development agenda(1)
Advancing regional recommendations on the post 2015 development agenda(1)Dr Lendy Spires
 
Advancing regional recommendations on the post 2015 development agenda
Advancing regional recommendations on the post 2015 development agendaAdvancing regional recommendations on the post 2015 development agenda
Advancing regional recommendations on the post 2015 development agendaDr Lendy Spires
 
9059_GPF Learning Series_Zambia v4 (1)
9059_GPF Learning Series_Zambia v4 (1)9059_GPF Learning Series_Zambia v4 (1)
9059_GPF Learning Series_Zambia v4 (1)Victoria Cabral
 
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...Mainstreaming Governance in Country Programs: Insights from the Governance Pa...
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...Victoria Cabral
 
NGFP_Annual_Report_2014-2015
NGFP_Annual_Report_2014-2015NGFP_Annual_Report_2014-2015
NGFP_Annual_Report_2014-2015ward33
 
Pacebook vol 1- Getting to grants
Pacebook vol 1- Getting to grantsPacebook vol 1- Getting to grants
Pacebook vol 1- Getting to grantsPACE LEBANON
 
Summary of-results-and-findings-of-the-2011-international-comparison-program
Summary of-results-and-findings-of-the-2011-international-comparison-programSummary of-results-and-findings-of-the-2011-international-comparison-program
Summary of-results-and-findings-of-the-2011-international-comparison-programBFSICM
 
Aid partnerships a vehicle to strengthen ng os in somalia
Aid partnerships  a vehicle to strengthen ng os in somaliaAid partnerships  a vehicle to strengthen ng os in somalia
Aid partnerships a vehicle to strengthen ng os in somaliaAlnaaze Nathoo
 
SUCCESSMODELFINALFEB2010
SUCCESSMODELFINALFEB2010SUCCESSMODELFINALFEB2010
SUCCESSMODELFINALFEB2010Leslie Traub
 

Semelhante a IDLO Final Evaluation 23 April Report Final CF (20)

Organization assessment guide
Organization assessment guideOrganization assessment guide
Organization assessment guide
 
YJC Summer Guidelines
YJC Summer GuidelinesYJC Summer Guidelines
YJC Summer Guidelines
 
Social Impact Evaluation Report
Social Impact Evaluation ReportSocial Impact Evaluation Report
Social Impact Evaluation Report
 
Conducting BIAs for UASC in Cairo (RefugePoint 2012)
Conducting BIAs for UASC in Cairo (RefugePoint 2012)Conducting BIAs for UASC in Cairo (RefugePoint 2012)
Conducting BIAs for UASC in Cairo (RefugePoint 2012)
 
UNGC_FinalReport-postcomments_04June16
UNGC_FinalReport-postcomments_04June16UNGC_FinalReport-postcomments_04June16
UNGC_FinalReport-postcomments_04June16
 
Challenges and Changes The Political Economy of National Development Banks in...
Challenges and Changes The Political Economy of National Development Banks in...Challenges and Changes The Political Economy of National Development Banks in...
Challenges and Changes The Political Economy of National Development Banks in...
 
Report iatc-final
Report iatc-finalReport iatc-final
Report iatc-final
 
La calidad de la AOD
La calidad de la AODLa calidad de la AOD
La calidad de la AOD
 
Advancing regional recommendations on the post 2015 development agenda(1)
Advancing regional recommendations on the post 2015 development agenda(1)Advancing regional recommendations on the post 2015 development agenda(1)
Advancing regional recommendations on the post 2015 development agenda(1)
 
Advancing regional recommendations on the post 2015 development agenda
Advancing regional recommendations on the post 2015 development agendaAdvancing regional recommendations on the post 2015 development agenda
Advancing regional recommendations on the post 2015 development agenda
 
9059_GPF Learning Series_Zambia v4 (1)
9059_GPF Learning Series_Zambia v4 (1)9059_GPF Learning Series_Zambia v4 (1)
9059_GPF Learning Series_Zambia v4 (1)
 
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...Mainstreaming Governance in Country Programs: Insights from the Governance Pa...
Mainstreaming Governance in Country Programs: Insights from the Governance Pa...
 
NGFP_Annual_Report_2014-2015
NGFP_Annual_Report_2014-2015NGFP_Annual_Report_2014-2015
NGFP_Annual_Report_2014-2015
 
Pacebook vol 1- Getting to grants
Pacebook vol 1- Getting to grantsPacebook vol 1- Getting to grants
Pacebook vol 1- Getting to grants
 
Summary of-results-and-findings-of-the-2011-international-comparison-program
Summary of-results-and-findings-of-the-2011-international-comparison-programSummary of-results-and-findings-of-the-2011-international-comparison-program
Summary of-results-and-findings-of-the-2011-international-comparison-program
 
2015 IABC Annual Report
2015 IABC Annual Report2015 IABC Annual Report
2015 IABC Annual Report
 
2015 Point-in-Time HUD Methodology & Guide
2015 Point-in-Time HUD Methodology & Guide2015 Point-in-Time HUD Methodology & Guide
2015 Point-in-Time HUD Methodology & Guide
 
Aid partnerships a vehicle to strengthen ng os in somalia
Aid partnerships  a vehicle to strengthen ng os in somaliaAid partnerships  a vehicle to strengthen ng os in somalia
Aid partnerships a vehicle to strengthen ng os in somalia
 
SUCCESSMODELFINALFEB2010
SUCCESSMODELFINALFEB2010SUCCESSMODELFINALFEB2010
SUCCESSMODELFINALFEB2010
 
KosovoEvaluation
KosovoEvaluationKosovoEvaluation
KosovoEvaluation
 

IDLO Final Evaluation 23 April Report Final CF

  • 1. Page 1 of 105 Final Evaluation of the Afghanistan Justice Training Transition Program Prepared by Evaluation Team: Conor Foley and Orsolya Székely March 2016
  • 2. Page 2 of 105 ACKNOWLEDGEMENTS AND DISCLAIMER This is an independent, external evaluation report carried out by the above authors into the Justice Training Transition Program(JTTP),implemented by the International Development LawOrganization (IDLO). The contents and conclusions of this evaluation report are based solely and strictly on the opinions of the authors, and should in no way be seen as reflecting those of IDLO, its program staff or donors. The evaluation team would nevertheless like to thank the staff of the JTTP,IDLO in Headquarters and in Afghanistan, particularly in the JTTP M&E unit, who not only put up with our intrusive presence and questioning, but actively and generously assistedour work. They provided us with invaluable feed- back and additional guidance on the production of our initial Inception Report, promptly furnished us with all necessary documents and facilitated us in making contacts and arranging interviews in Afghanistan and Rome. They also maintained and updated a daily interview schedule for us, which maximized the number of interviews that we were able to conduct. IDLOorganized our accommodation and board during the field mission, provided us with transport, interpreters and security clearance for each external interview, and took care of our personal security. IDLO staff also made themselves available for interview outside office hours and at weekends, which maximized our time available to speak to Afghan government and justice sector officials during the official working week. The JTTP translation team in Kabul also worked extremely hard to translate over 200 questionnaires distributed during the focus group process. The JTTP finance team in Rome prepared a useful summary of the program’s budget and expenditure up to December 2015, which helped structure our discussion of this issue. The evaluation team owe an enormous debt of gratitude not only for the support provided, but for the unfailingly polite, constructive and friendly manner in which it was given. Afghanistan has a much deserved reputation for the hospitality of its people. It is a difficult, dangerous and stressful place for the staff of international organizations to work in at the moment, and is rightly considered a hardship post for those who are based there. But we were made to feel welcome and supported by IDLO’s national and international staff every moment that we were in the country. Needless to say, any errors of fact or interpretation are solely the responsibility of the evaluation team.
  • 3. Page 3 of 105 TABLE OF CONTENTS Acknowledgements and Disclaimer ............................................................................................... 2 Table of Contents ......................................................................................................................... 3 Program Geographical Scope in Afghanistan.................................................................................. 5 Acronyms .................................................................................................................................... 6 Executive Summary...................................................................................................................... 8 1 Evaluation Purpose, Scope, Methodology and Limitations ......................................................12 1.1 Purpose and Scope....................................................................................................... 12 1.2 Evaluation Questions to be addressed............................................................................ 13 1.3 Evaluation Questions ................................................................................................... 14 1.4 Data Sources and collection methods ............................................................................ 15 1.5 Management and logistical support ............................................................................... 17 1.6 Limitations of the evaluation ........................................................................................ 17 1.7 Conflicts of Interest...................................................................................................... 18 2 History and Overview of the Program....................................................................................19 3 Main Evaluation Findings .....................................................................................................21 Relevance............................................................................................................................... 21 3.1 Evaluation Question 1: To what extent was the program theory of change able to respond to the changing needs and context of the Afghan justice sector,including changes to the political and security context?..................................................................................................................... 21 3.2 Evaluation Question 2: How able was the implementing organization to respond to the identified needs/priorities?....................................................................................................... 24 Effectiveness.......................................................................................................................... 27 3.3 Evaluation Question 3: Was the JTTP successfulin delivering the expected outputs with a focus on increased quality at the same time?............................................................................. 27 3.4 Evaluation Question 4: Was JTTP able to identify innovative approaches and implementing mechanisms to facilitate the delivering of the expected outcomes and outputs despite the challenges of Afghanistan? ...................................................................................................................... 33 3.5 Evaluation Question 5: What were the key internal and external factors that contributed to the program successes / hindered its effectiveness? ................................................................... 36 3.6 Evaluation Question 6: To what extent did funding, staff, time and other resources contribute to achieving the results?........................................................................................... 40 3.7 Evaluation Question 7: Did the Program deliver its outputs in a cost-efficient and timely manner? ................................................................................................................................. 44 Impact.................................................................................................................................... 48 3.8 Evaluation Question 8: What results did JTTP produce according to the planned logical framework, and did it go beyond the originally planned results?................................................. 48 3.9 Evaluation Question 9: Have there been any lessons learnt in terms of result delivery which could be transferred as a knowledge pool to other programs in the future? .................................. 55
  • 4. Page 4 of 105 3.10 Evaluation Question 10: What communication strategies and other tools were put in place to enhance program visibility? ..................................................................................................... 57 3.11 Evaluation Question 11: Will the results produced by capacity development of justice professionals likely to sustain in the future and what are the necessary conditions for the benefits achieved to be consolidated?.................................................................................................... 59 3.12 Evaluation Question 12: To what extent are the systems, capacities, and facilities transferred to in-house training departments going to be retained within the institutions without external support (organizational sustainability)? .................................................................................... 62 3.13 Evaluation Question 13: Has gender been mainstreamed and aligned into the JTTP activities especially with regard to the recommendations of the mid-term review?..................................... 64 4 Conclusions .........................................................................................................................67 5 Recommendations ................................................................................................................68 6 Appendices..........................................................................................................................69 Annex 1: List of Documents Reviewed..................................................................................... 69 Annex 2: List of Interviews ..................................................................................................... 76 Annex 3: Focus Group Discussions .......................................................................................... 79 Annex 4: Phases of Work....................................................................................................... 85 Annex 5: Conduct of Focus Group Discussions Guide.............................................................. 86 Annex 6: Focus Group Questionnaire ...................................................................................... 88 Annex 7: Notes for conducting an interview with provincial officials ........................................ 89 Annex 8 IDLO Terms of Reference For the Evaluation, 15 December 2015................................ 91 Annex 9 Evaluation Matrix...................................................................................................... 97
  • 5. Page 5 of 105 PROGRAM GEOGRAPHICAL SCOPE IN AFGHANISTAN
  • 6. Page 6 of 105 ACRONYMS ACLEA Advanced Continuing Legal Education for Afghanistan AIBA Afghanistan Independent Bar Association AIHRC Afghanistan Independent Human Rights Commission ANA Afghanistan National Army ANP Afghanistan National Police AGO Attorney General’s Office CID Criminal Investigation Division CLE Continuing Legal Education CLRWG Criminal Law Reform Working Group CMS Case Management System CPC Criminal Procedure Code CSI Crime Scene Investigation ETC Education Training Committee DAC OECD Development Assistance Committee of the Organization for Economic Co-operation and Development DOWA Department of Women’s Affairs EVAW law Elimination of Violence Against Women law EUPOL European Union Police Mission FGD Focus Group Discussion FRU Family Response Unit GDET General Directorate of Education and Training GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GTC General Training Command HOOAC High Office of Oversight and Anti-Corruption IANDS Interim Afghanistan National Development Strategy IDLO International Development Law Organization IGO Inter-Governmental Organization IIPLD Institute International de Paris le Defense ILFA International Legal Foundation for Afghanistan ILF International Legal Foundation INL U.S. State Department Bureau for International Narcotics and Law Enforcement Affairs ISISC International Institute of Higher Studies in Criminal Science ITA International Training Advisor GiROA Government of the Islamic Republic of Afghanistan JSSP Justice Sector Support Program JTTP Afghanistan's Justice Training Transition Program LAD Legal Aid Department LAOA Legal Aid Organization of Afghanistan M&E Monitoring and Evaluation MOWA Ministry of Women’s Affairs MTT Mobile Training Team MOA Memoranda of Agreement MOI Ministry of Interior MOJ Ministry of Justice MOU Memorandum of Understanding NAD Needs Assessment Department NGOs Non-Governmental Organizations NJP National Justice Program NJSS National Justice Sector Strategy NLA National Legal Adviser
  • 7. Page 7 of 105 NRC Norwegian Refugee Council OCAT Organizational Capacity Assessment Tool OVI Objective Verifiable Indicator PHQ Police Head Quarters PPP Procurement Policies and Procedures R&R Rest and Recreation SIGAR Special Inspector General for Afghanistan TCUB Training Capacity Building Unit TIMS Training Information Management System ToC Theory of Change ToR Terms of Reference ToT Training of Trainers UNAMA UN Assisted Mission to Afghanistan UNDP UN Development Programme UNODC UN Office on Drugs and Crime UNDSS UN Department of Security Sevices US United States USAID US Agency for International Development USIP US Institute for Peace VAWU Violence Against Women Unit
  • 8. Page 8 of 105 EXECUTIVE SUMMARY Afghanistan faces huge ongoing challenges as an increasingly violent insurgency threatens government control over ever larger parts of the country. There is now a very real danger that the precipitate withdrawal of international support could reverse the gains of recent years. Nevertheless, there are some grounds for optimism and some areas where international investment eventually paid dividends: the capacity-building of Afghan criminal justice professionals is one of these. This report presents the results of a final evaluation of the Justice Training Transition Program (JTTP), funded by the US State Department Bureau for International Narcotics and Law Enforcement Affairs (INL) and implemented by the International Development Law Organization (IDLO) in partnership with the Government of Afghanistan. This evaluation covers almost the whole period of the JTTP’s existence from its establishment in January 2013 to the final completion of the extended program in March 2016. The purpose of the evaluation is to:  Assess the extent to which the JTTP was able to achieve its intended objectives;  Provide IDLO and INL useful information to inform/validate a follow-on program design;  Identify best practices and lessons learned in the JTTP’s theory of change and implementation that can be valuable for future capacity development and transition programs. The evaluation team used the following interconnected data collection methods: (i) document review; (ii) field mission; (iii) interviews; (iv) focus groups and a questionnaire form distributed at these; and (v) observations. Whereverpossible, the evaluation teamtriangulated this data using a range of sources. Its findings have been structured around the five internationally-accepted evaluation criteria set out by the Development Assistance Committee of the Organization for Economic Co-operation and Development (DACOECD):Relevance,Effectiveness,Efficiency, Impactand Sustainability. The main findings of this evaluation are as follows: Relevance The JTTP’soverall goal is to increase the trust of the citizens of Afghanistan in the formal state criminal justice sector. It provides direct capacity-building support activities – through training and coaching – to individual Afghan legal professionals within four targetinstitutions: the Supreme Court, the Attorney General’s Office, the Ministry of Justice and Ministry of Interior, while working to build the capacity of these institutions to continue these activities themselves. This corresponds with the Afghan government’s own goals, the objectives of its main donors and IDLO’s global strategic priorities. The evaluation team believe that the capacity-building support provided to individual Afghan legal professionals was relevant and appropriate to their needs. While its transition strategy was somewhat delayed it was appropriate to the challenges that Afghanistan faces. Indeed, these challenges are precisely why programs such as the JTTP are so necessary and relevant. The transition strategy was designed and implemented in full consultation with the target institutions and took account of their needs and capacities. Memoranda of Understanding (MoUs) have now been signed with all four national justice institutions, which clarify the transition related co-operation for the period of the JTTP and potentially beyond, should another project continue to be implemented with the same target institutions. IDLO proved capable of implementing the JTTP and brought added value when compared to the previous implementing organization. The decision in the JTTP’s initial design to focus exclusively on the official state criminal justice system, however, significantly reduced the program’s scope in addressing how justice is actually administered and perceived in Afghanistan. Effectiveness The JTTP became increasingly effective at improving the quality of its individual capacity development support and at developing co-ordination and outreach mechanisms with both the national Afghan
  • 9. Page 9 of 105 institutions and other international stakeholders, aimed at a sustainable transition. Reporting, and monitoring and evaluation systems have been developed, which have been used for effective communication with the donor, strategic planning and measurementof the program’s results. The JTTP has also become increasingly effective at working with the target justice sector institutions to define the assistance that it can provide towards establishing and supporting operational in-house training departments. This has included helping each institution to establishing governance arrangements and helping to recruit and train instructors for the units once these have been approved. The JTTP is one of the largest justice sector capacity-building project in Afghanistan and this scale, combined with the strong emphasis on sustainability, has led IDLO to develop some highly innovative approaches during the course of the program. The evaluation team believes that the JTTP has put considerable effort into improving the material, content and delivery of its courses. Its Curriculum Unit has continued the development of this material and produced individualized training manuals to hand over a well-developed and designed package of training materials to the national justice sector institutions. The recipient institutions, for their part, also appear to have increasingly recognized the value of the program in addressing their own capacity gaps. The JTTP has transformed itself from a relatively passive deliverer of legal training services, against set quantitative targets, to a far more engaged and pro-active program that is actively attempting to transform the justice sector in Afghanistan. It exceeded its achievements in both the numbers of people that it trained and coached and by increasing the number of training opportunities offered. It also developed a pro-active strategy of outreach and engagement with both national and international institutions. Efficiency The JTTP budget makes it by far the largest program that IDLO has ever implemented, although it is relatively small in the overall context of the investment by the international community in rule of law, justice sector reform and governance in Afghanistan. The JTTP obviously has a low number of direct beneficiaries, if these are measured solely in terms of individual graduates of its capacity-building support activities. If its theory of change is accepted, however, its number of indirect beneficiaries could be multiplied to include everyone who has an interest in seeing a prosperous, fair and stable Afghanistan. The JTTP has benefited from a close relationship with its donor, INL, which has a good institutional memory of the program and has provided strong oversight of it. The JTTP has had a significant under-spend, which enabled a no-cost program extension of nine months and still left over money for a follow-on program. The under-spends seemto have beencausedby a combination of some delays in fulfilling some activities on time – principally due to problems associated with recruiting and retaining international staff – some cost-efficiencies implemented by the program, and some evidence that the original budget may have been too generous in some areas. The evaluation team believe that IDLOcould usefully explore how to maximize the organizational and thematic synergies betweenthese programs as part of an overall effort to establish a more unified profile of its work in Afghanistan. The evaluation team is satisfied that IDLO was able to scale up its organizational capacity to implement the JTTP and the extra capacity that it has developed in the process means that it is now capable of credibly taking on further large programs, both in Afghanistan and elsewhere. Impact The JTTP has pioneered the development of methodology to evaluate the effectiveness of its capacity development support activities on Afghanistan’s justice sector institutions and the quality of justice that these deliver. This appears to enable the JTTP to evaluate the impact of its activities at an institutional level. Preliminary data shows significant improvements in the quality of justice delivered by the graduates of its training and coaching programs and also to identify where ‘capacity gaps’ still exist. During all interviews and FGDs it was repeatedly stated that the criminal justice system is improving in Afghanistan and that its legal professionals are using the knowledge that they have acquired from the JTTP to improve the quality of their work at both an individual and institutional level. Although its transition strategy started late there has been significant progress with all the target institutions. A Lessons Learned Brief (LLB) was in the process of being drafted during the present evaluation, which should enable IDLO to develop a detailed and evidence-based theory of change that can be used both
  • 10. Page 10 of 105 for program development and evaluation purposes in the future. The JTTP’s transition strategy contained clear benchmarks to indicate progress in each institution. This shows that progress in the Supreme Court and Ministry of Interior hasbeen considerable and the progressin the Ministry of Justice could be regarded as satisfactory. Implementation within the AGO is behind schedule, however, although this was mainly for externalfactors,including delays to the appointment of a permanent AGO that were beyond the capability of the JTTP to control. The four institutions also had quite different levels of capacity at the start of the JTTP and this clearly influenced progress under Outcome Two. There are many other lessons that can be learned from the program’s implementation, in particular how a sustainable transitional training program canbe developed if it is not clearly linked to the management structures, human resources and performance appraisals systems of the institutions to whom it is intended to subsequently transfer the training. The JTTP succeeded in scaling up to implement its largest ever program in the most dangerous and insecure environment in which it has ever operated. Sustainability The JTTP has developed clear criteria for selecting candidates for participation in its training courses and coaching, good pre and post course testing of the knowledge gained and follow-up methodology to measure changes in work behavior and the impact this has had on the relevant institutions. While there is no guarantee that the capacity development achievements of the JTTP will be fully sustainable over the long-term, it does seem reasonable to conclude that the program has met the objective set out in its original proposal. There remains a clear question mark over the sustainability of its transition arrangements given both the newness of the institutions and the parlous state of the government of Afghanistan’s finances. Afghanistan is almost entirely reliant on international donors for continuing financial support. Given its other domestic priorities, it is unrealistic to expect the government to fund the establishment of the training units envisaged under Outcome Two without external donor support, but it is vital the government takes ‘ownership’ of this process through a serious budgeting process. Although the target institutions can be given budgets to conduct training operations they still need to develop the capacity to manage, design, conduct, and evaluate institutional trainings. The evaluation team, therefore,believes that a successor program to the JTTP is needed to continue to provide support to the new training units. IDLO was in the process of discussing a follow-up program to the JTTP, while this evaluation was taking place. The evaluation team hopes that this may include some of the JTTPs most obvious strengths: an impressive curriculum, a developed and sophisticated M&E system, a team dedicated to maintaining links with the target institutions and some capacity to deliver particular training to target groups of Afghan legal professionals – particularly at an inter-institutional level. Conclusions The evaluation’s overall conclusion is that the JTTP was relevant to the context in which it was implemented and that its activities were generally conducted in an effective and efficient manner. Its individual capacity-building activities aimed at supporting Afghan legal professionals have performed a valuable service that was appreciated both by the Afghan national justice institutions and by the participants in the training themselves. The JTTP became increasingly effective at working with the target justice sector institutions to define the assistance that it can provide towards establishing and supporting operational in-house training departments. It also became more proactive in looking for linkages between its work on strengthening the capacity of the justice sector and those organizations that are involved in monitoring its performance in relation to justice and human rights. The evaluation team believes that the JTTP’s activities have had a positive impact, which could be sustainable in at least the medium term given some continuing external support. Recommendations The key overall recommendations arising from the above findings are as follows: 1. IDLO should build on the strengths of the JTTP in its follow-up program, which should focus on helping the Afghan justice sector institutions to deliver their own training
  • 11. Page 11 of 105 programs, ensuring quality control of this training and promoting greater transparency and legal clarity within the Afghan justice system. 2. IDLO should actively consider how to increase synergies between this program and other programs that it is implementing in Afghanistan. 3. IDLO should also actively work to increase its linkages and synergies with other similar programs working in the area of justice sector reform and capacity-building. 4. The new program should maintain the capacity to deliver inter-institutional specialized and tailored courses on particular legal topics as requested by Afghan institutions or other stakeholders. 5. The new program should have the capacity to enable IDLO to participate in debates about legal reform, including developments in parliament and at the Supreme Court. 6. The new program should have the capacity to continue monitoring and evaluating the impact of capacity-building development on partner institutions and to provide some support to address ‘capacity gaps’ where necessary. 7. The new program should conduct proactive outreach with organizations that are currently monitoring the performance of the Afghan justice sector. 8. IDLO should draw on its Lessons Learned Brief (LLB) in Afghanistan to develop a detailed and evidence-based theory of change that can be used both for program development and evaluation purposes in the field of developing justice sector capacity to organize continuing professional development. 9. IDLO should draw on the methodology used to conduct its Levels 3 and 4 Assessments in other programs. 10. IDLO should also consider what other lessons can be learned from its experiences implementing the JTTP in the particular circumstances of dangerous and insecure situations such as Afghanistan and continue to develop organizational policies and procedures that take account of such situations while providing field offices with tools tailored to their particular situations. 11. IDLO should ensure that it develops a communications strategy for all future program activities, carefully tailored to the program and country in which it is to be implemented. 12. IDLO should develop a comprehensive risk-management strategy into its programming methodology. 13. IDLO should continue to ensure that gender equality and gender mainstreaming are fully integrated into all its existing program activities and reflected in all relevant program documents, in line with its gender strategy.
  • 12. Page 12 of 105 1 EVALUATION PURPOSE, SCOPE, METHODOLOGY AND LIMITATIONS 1.1 PURPOSE AND SCOPE This evaluation covers almost the whole period of the Justice Training Transition Program (JTTP)’s existence from its establishment in January 2013 to January 2016 – two months before the final completion of the extended program in March 2016. The purpose of the evaluation is to:  Assess the extent to which the JTTP was able to achieve its intended objectives;  Provide IDLO and INL useful information to inform/validate a follow-on program design;  Identify best practices and lessons learned in the JTTP’s theory of change and implementation that can be valuable for future capacity development and transition programs. A mid-term evaluation (MTE) was carried out between July and October 2014, which noted the JTTP was broadly ‘on track’ towards achieving its targets under the program’s first objective, capacity development and training, but that it had been very late in drafting a transition strategy for handing over the training activities to national justice sector institutions in a sustainable manner. 1 The MTE concluded that the JTTP’s Transition Plan was ‘basically sound as a working document’ but required ‘buy in and national ownership’. The MTE did not believe that this could be accomplished by July 2015, the original date for the completion of the program. It recommended the strategy be revised with new timelines and increased staff resources devoted towards this transition. It further recommended that ‘IDLO should, as a matter of priority seek to negotiate a no-cost contract extension for the JTTP with INL to early 2016.’ It also recommended that: ‘IDLO and INL should begin discussions about a follow up project starting in 2016. This should focus on helping the Afghan justice sector institutions deliver their own training programs and ensuring quality control of this training. The project should also conduct pro-active outreach with organizations that are currently monitoring the performance of the Afghan justice sector.’2 These proposals were broadly endorsed by IDLO’s senior management team.3 IDLO submitted a no- cost contract extension request until 31 March 2016, which foresaw ‘completing all training activities and close provincial offices by January 2016; and continue to work on the Transition in early 2016 to support the target institutions towards the next Afghan government budget cycle.’ The JTTP also created a new unit with dedicated focal points specifically dedicated to supporting the transition process.4 While the legal capacity building component of the JTTP (Outcome One) was extensively covered by the MTE, the effortsof transferring this training and coaching capacity to the Afghan justice institutions (Outcome Two), synthesized in the Transition Plan, is assessed here for the first time. This final evaluation also addresses the JTTP’s impact, which was omitted from the terms of reference of the MTE. It seeks to highlight innovative measures,good practices and lessons learned that may be useful to the JTTP and INL in discussing the design of a follow up program as well as providing IDLO with information that may help the design of other similar programs elsewhere in the world. As the MTE noted, evaluating any justice sector capacity-building program is inherently difficult, because the results and impact of these efforts are not immediately visible and may take years or even decades to become apparent. These difficulties are particularly pronounced in Afghanistan, given the 1 Conor Foley and Katerina Stolyarenko, External mid-term evaluation ofAfghanistan Justice Training Transition Program (JTTP) (2013-2014),November 2014, p.8. Executive Summary [Hereinafter MTE, 2014]. 2 Ibid. 3 IDLO, Implementation Tracking Matrix of the Management response to the mid-term evaluation ofthe “Completing the transition in Afghanistan:Justice Training Transition Program (JTTP)”, DRAFT, 25 May 2015 [Hereinafter IDLO Management Team Response,2015]. 4 Ibid.
  • 13. Page 13 of 105 weaknesses of the rule or law, widespread corruption and a growing anti-government insurgency. Nevertheless, this final evaluation sought evidence of changes in the work-related behavior of the JTTP’s graduates and also analyzed what positive and negative factors contributed to changes in performance in the different institutions that are being supported. Some aspects of the individual capacity development support provided by the JTTP were criticized during the MTE, by both representatives of the target institutions and graduates of its coaching and training programs. This evaluation re-visits some of these criticisms and the extent to which they were subsequently addressed by the JTTP, but its main focus is on the sustainable transition of the support provided to the Afghan justice sector institutions themselves. 1.2 EVALUATION QUESTIONS TO BE ADDRESSED The findings of this evaluation have been structured around the five internationally-accepted evaluation criteria set out by the Development Assistance Committee of the Organization for Economic Co- operation and Development (DAC OECD): Relevance, Effectiveness, Efficiency, Impact and Sustainability.5 After detailed discussion with IDLO during the Inception Phase of this evaluation, these were sub-divided further into a total of 13 evaluation questions that are set out below. A detailed Matrix of the Evaluation Methodology, including questions, indicators, data sourcesand collection tools is contained in Appendix Nine. 5 International Development Law Organization, A guide for course/project managers, assistants, training focal points,data entry staff on Implementing the IDLO approach to evaluating training,April 2010.
  • 14. Page 14 of 105 1.3 EVALUATION QUESTIONS Relevance Evaluation Question 1: To what extent was the program theory of change able to respond to the changing needsand context of the Afghan justice sector,including changesto the political and security context? Evaluation Question 2: How able was the implementing organization to respond to the identified needs/priorities? Effectiveness Evaluation Question 3: Was the JTTP successful in delivering the expected outputs with a focus on increased quality at the same time? Evaluation Question 4: Was JTTP able to identify innovative approaches and implementing mechanisms to facilitate the delivering of the expected outcomes and outputs despite the challenges of Afghanistan? Evaluation Question 5: What were the key internal and external factors that contributed to the program successes / hindered its effectiveness? 
 Efficiency Evaluation Question 6: To what extent did funding, staff, time and other resources contribute to achieving the results? Evaluation Question 7: Did the Program deliver its outputs in a cost-efficient and timely manner? Impact Evaluation Question 8: What results did JTTP produce according to the planned logical framework, and did it go beyond the originally planned results? Evaluation Question 9: Have there been any lessons learnt in terms of result delivery which could be transferred as a knowledge pool to other programs in the future? Evaluation Question 10: What communication strategies and other tools were put in place to enhance program visibility? Sustainability Evaluation Question 11: Will the results produced by capacity development of justice professionals be sustainable in the future and what conditions are necessary for the benefits achieved to be consolidated? Evaluation Question 12: To what extent are the systems, capacities, and facilities transferred to in- house training departments going to be retained within the institutions without external support (organizational sustainability)? Cross-cutting Evaluation Question 13: Has gender been mainstreamed and aligned into the JTTP activities especially with regard to the recommendations of the mid-term review?
  • 15. Page 15 of 105 1.4 DATA SOURCES AND COLLECTIONMETHODS The evaluation team consisted of two international contractors who were assisted during the Field Mission by four national contractors: Although all four national contractors had previously worked for the JTTP, care was taken to ensure that they remained rigorously independent of the program for the purposes of the evaluation. All four were only used in provinces where they had not previously worked for IDLO and all received prior training and instruction on how to conduct interviews and focus group discussions (FGDs) in an impartial manner. The evaluation team used the following interconnected data collection methods: (i) document review; (ii) field mission; (iii) interviews; (iv) focus groups and a questionnaire form to be distributed at these; and (v) observations. Wherever possible, the evaluation team triangulated this data using a range of sources. Document Review A full list of all documents reviewed is contained in Appendix One. The evaluation team also drew on other published reports on the state of the justice system in Afghanistan and its adherence to international human rights standards, as well as critical evaluations of efforts to strengthen its capacity, such as the current program. The evaluation team also reviewed gender related documents relevant to the JTTP’s progress since the mid-term evaluation. Field Mission The evaluation team conducted a field mission to Afghanistan in January and February 2016, which covered five out of the seven provinces where the JTTP had been implemented through its provincial offices: Kabul, Nangarhar,Balkh, Heratand Badakhshan. Asdiscussed above, four national contractors were hired by the team in order to maximize both the number of interviews and FGDs and the geographical scope of the mission. The security situation precluded a visit to Kunduz, by either national or international contractors,and it was also decided not to visit Bamyan as the winter snow can lead to flights in out of the province being cancelled, which could have left a team member marooned there. Initial interviews and the first FGD were conducted in Kabul at the start of the field mission and then the international contractors visited Herat and Balkh respectively, accompanied by two national contractors. They conducted more interviews and FGDs in these provinces and then returned to Kabul where they conducted more interviews, observed training course and attended other meetings. The two national contractors proceeded onto Nangarhar and Badakhshan where they conducted interviews and FGDs, using a standard questionnaire that the evaluation team had previously designed and tested. Two other national contractors assisted in the administrative organization of two FGDs in Herat and then travelled to Kabul where they conducted more FGDs using the same format. The two international contractors participated in and observed some of the FGDs conducted in Kabul to ensure consistency of their delivery and conduct. Six of the JTTP provincial offices were closed between November 2015 and January 2016, before the evaluation team could visit them, but the team were able to visit regional training centers,established by the Ministry of Interior (MoI), interview relevant officials from the four partner institutions, conduct FGDs with direct program beneficiaries and also try to seek some independent assessment of the program’s results, relevance, effectiveness and sustainability. Meetings were conducted with the regional offices of the Afghan Independent Bar Association (AIBA) and UN Assistance Mission to Afghanistan (UNAMA) Judicial Affairs Officers who provided valuable insights into how the justice system was functioning at the provincial level. The evaluation team also attended a Board of Donors meeting convened by UNAMA and observed some training classes as they were being conducted. The schedule of the evaluation team’s visits is set out below:
  • 16. Page 16 of 105 Dates January – February 201 Province 28 - 30 January. All Kabul 31 January – 4 February. One national and one international Balkh 31 January – 4 February. One national and one international Herat 5 - 10 February. One national Badakhshan 5 - 10 February. One national Nangarhar 4 – 14 February. Two international and two national Kabul 10 – 14 February. All Kabul Interviews The evaluation team conducted one-to-one interviews with almost 100 people. A full list of interviewees is contained in Appendix Two. A standardized interview question sheet is contained in Annex Seven. Interviews were conductedface-to-face whereverpossible and those thatwere conducted by Skype are identified accordingly. The interviewees included representatives of the following organizations:  IDLO personnel in Headquarters and Afghanistan, including the JTTP and IDLO’s other two programs in Afghanistan  Representatives from the Ministry of Interior, Attorney General’s Office, Supreme Court, Ministry of Justice;  Ministry of Women Affairs;  the Afghan Independent Bar Association;  Training staff from the target institutions;  US State Department (INL), USAID and the US Institute of Peace;  UNAMA, UNDP,and UN Women;  Other international organizations working in the field of access to justice, gender justice and justice sector capacity building in Afghanistan, including JSSP and other relevant organisations;  NGOs working directly on justice related issues or analyzing the Afghan justice sector. Focus Group Discussions The evaluation team also went to considerable lengths to organize as many FGDs with JTTP graduates aspossible. Participants in the FGDs included: graduates of the AdvancedContinuing Legal Education for Afghanistan (ACLEA) and ACLEA-Plus Courses, graduates of Continuing Legal Education (CLE) Course, and the staff of the training units and seconded trainers. Simple questionnaire forms were also distributed at the start of each FGD and collected at the end. A sample questionnaire and the Guidance for the FGDs are contained as Appendices Five and Six of this report. The aim of the FGDs was to explore whether and how the training, and coaching support provided to individual legal professionals (graduates) also benefited the performance of their department and institutions as a whole and what other capacity outcomes can be identified at the individual level as a result of the JTTP intervention and engagement strategy. FGDs were also organized during the MTE and provided valuable information about the quality and conduct of the training and coaching provided by the JTTP. For this evaluation, the emphasis of the guiding questions was less on the participants’ satisfaction with the training that they received and far more on questions such as changes in work- related behavior of the graduates, and what positive and negative factors they believe contribute to improving the performance of their respective institutions and the sustainability of the capacity-building support once it is taken over by the Afghan national institutions themselves. A total of 14 FGDs were organized involving just over 200 participants. As well as providing useful critical feed-back on the perceived benefits of the training and capacity development by those who
  • 17. Page 17 of 105 received it, under Outcome One, these FGDs provided a statistically significant sample of both the impact of the Programin terms of improving the functioning of the Afghan justice sector and reasonably well-informed observers about the sustainability of this support once responsibility for this passes to the national institutions themselves as envisaged in the JTTP’s transition strategy under Outcome Two. Information from the FGDs was therefore used to triangulate some findings. 1.5 MANAGEMENT ANDLOGISTICAL SUPPORT The international contractors were contracted for a total of 42 days each. This broke down as seven days to draft and finalize the Inception Report, twenty days to conduct the field mission, ten days to produce the first draft of the Final Report and five days to finalize the report after receiving comments from IDLO. The phases of work of the evaluation is set out in Appendix Four. The evaluation process was managed by IDLO’s JTTP M&E Unit, in close coordination with the Program Team, ensuring that all information was provided and requests for feedback to the evaluation were responded to in a timely manner. The JTTP M&E Unit reviewed the Evaluation Questions, Evaluation Matrix and data collections tools, which were contained in anInception Report and provided comments on these. The evaluation team submitted a first draft copy of their Final Report on 7 March 2016. IDLO provided a single set of consolidated comments on the draft on 17 March. Upon receipt of these written comments, the evaluation team agreed to correct all documented factual errors and inaccuracies and make changes related to the report’s structure, consistency, analytical rigor, validity of evidence, and requirements in the TOR. After making the necessary changes, the evaluation team undertook to submit the Final Evaluation Report for IDLO approval. The JTTP Team in Afghanistan assisted with arrangements for the evaluation including travel and appropriate logistic support and security. This included liaising with local stakeholders to schedule meetings and providing interpretation services. After an initial discussion with key program personnel in-country the evaluators submitted a detailed plan for the FGDs, interviews, and visits at the start of the mission. IDLO undertook to respect the operational independence of the evaluation team to organize their own methods of working without external interference. The evaluation team accepted that its members were fully bound by IDLO’s security procedures and accepted that decisions of the JTTP Chief of Party were final on all issues relating to security and logistics. IDLO kept INL informed and involved at key times during the process, seeking input to draft deliverables as appropriate. The evaluation team conducted one interview with two INL staff in Kabul and one telephone interview with an INL staff member in Washington. 1.6 LIMITATIONS OF THE EVALUATION The biggest limitation of the evaluation was the compressed time within which it was carried out, due to delays in contracting the international contractors and the fact that both of these had taken on other work and personal commitments by the time they were hired. In consultation with IDLO,the evaluation team mitigated these limitations by hiring four national contractors to help with data collection and organizing their time to maximize the number of interviews and FGDs that could be carried out in the field. This problem exacerbated a number of more general limitations under which the evaluation was conducted. The size and diversity of Afghanistan and the challenges faced by its justice sector,meant that even the original time-scale within which the field mission was proposed to take place was quite limited. The continually worsening security situation and the possibility of winter snow could have made the logistics of organizing the field mission and – in particular – visiting the provinces very difficult. The evaluation was also drawing on interviews from a range of sources, which are by their nature subjective. Cultural factors could also have made it difficult to rely on tools such as self- assessment questionnaires to provide quantitative and qualitative evidence that training of Afghan legal professionals has actually improved their effectiveness.
  • 18. Page 18 of 105 The evaluation team sought to mitigate these limitations and minimize possible biases through triangulation of methods and data. This was achieved by interviewing a range of stakeholders at different levels from a variety of institutions and reviewing a wide range of documents. The evaluation team also reviewed independent reports on the state of the justice system in Afghanistan as well as critical evaluations of efforts to strengthen its capacity, such as the current program. The team also actively sought to interview people who could provide an independent perspective on these issues. In the event, the weatherwascomparatively mild during the field mission and although there were some problems with internal flights, this did not excessively disrupt the data collection. While there were a number of security incidents and terrorist attacks during the time in which the field mission was being conducted, the evaluation team was able to carry out its full schedule of external meetings and there were no lock-downs or major disruptions of their work. Triangulation of data was relatively straight forward because the evaluation team received very consistent information from different sources and the evaluators were impressed at the open and honest accounts that they received from a wide range of interviewees, including in the justice sector institutions and the JTTP staff themselves. These were frank about the difficulties and challenges that they faced and prepared to be self-critical about failures, while maintaining an underlying optimism about the overall progress and achievements accomplished. 1.7 CONFLICTS OF INTEREST The two international contractors are independent and have no prior involvement with the programming-setting, design, or overall management of the JTTP. Neither of the international contractors on the evaluation team has ever worked for IDLO,although one has evaluated two previous programs implemented by it. As discussed above the four national contractors do have prior connections to both IDLO and the JTTP,but care was taken to ensure that they maintained an impartial and objective approach to the data collection tasks assigned. This report was written solely by the international contractors and its findings are their own responsibility alone.
  • 19. Page 19 of 105 2 HISTORY AND OVERVIEW OF THE PROGRAM The ‘Justice Training Transition Program’ (JTTP) is funded by the US State Department Bureau for International Narcotics and Law Enforcement Affairs (INL) and implemented by the International Development Law Organization (IDLO) in partnership with the Afghan government. The program aims to increase the confidence of Afghan citizens in the justice sector. It focuses on achieving two outcomes: (a) Beneficiary Afghan justice sector professionals achieve increased capacity and competencies in delivering justice according to Afghan law; and (b) Afghan justice institutions are capable of managing the sustainable implementation of training programs. It provides five types of capacity-building support to Afghan legal professionals:  Advanced Continuing Legal Education for Afghanistan (ACLEA) – classroom teaching of 10 subjects (now reduced to six) designed to increase legal and technical knowledge among Afghan judges, prosecutors, defense attorneys, and investigative police officers from the Criminal Investigation Division (CID) of the Afghanistan National Police (ANP). The course requires 240 hours of intensive classroom sessions;  Continuing Legal Education (CLE) outreach seminars, which address practicalchallenges that justice professionals face during their practice;  Needs–based mentorship sessions in which JTTP legal experts provide on–the–job technical support to justice professionals; and  One hundred and twentyhours of follow up advancedtraining for ACLEAgraduates(ACLEA– Plus).  A set of paper, electronic and multimedia training materials developed and distributed to participants. The program was established in January 2013 and was originally due to close in July 2015, but this was extended by an agreement between IDLO and INL and it will now close at the end of March 2016. The MTE provided a brief overview of previous attempts to strengthen the Afghan justice system, dating back to the United Nations (UN) conference in Bonn, Germany, after the fall of the Taliban in October 2001.6 The Bonn Agreement specified that a Judicial Commission be established to rebuild the domestic justice system ‘in accordance with Islamic principles, international standards, the rule of law and Afghan legal traditions.’7 Hamid Karzai,who had beenappointed head of aninterim administration at Bonn, was elected President of Afghanistan in 2004 and re-elected in 2009, serving two full terms in office. Although customary (non-state) justice mechanisms still account for a high proportion of dispute resolutions in Afghanistan, the capacity of the major justice institutions have considerably strengthened in the last decade,and there is some evidence that public confidence in official institutions has increased in recent years.8 Annual surveys by the Asia Foundation, for example, show a rising willingness of people to report on crimes and violence, and seekredressthrough the state system.9 Anew Constitution was adopted in 2004, which sets out the hierarchy of domestic law, provides for a three-tiered judiciary and sets out the functions and powers of the other main justice institutions.10 Since that date, a number of significant new laws have been adopted or reviewed and drafted, including: a Criminal Procedure Code; the Laws of Organization of the Courts, of the Office of the Attorney General and of the Ministry 6 MTE, 2014. See also Sheila Reed (Team Leader), Conor Foley and Hamayoun Hamed, Evaluation Report on IDLO Italian-funded projects (2005 – 2008) for legal and judicial reform in Afghanistan,2008. 7 The Agreement on Provisional Arrangements in Afghanistan Pending the Re-establishment of Permanent Government Institutions, II Legal framework and judicial system, [Hereinafter Bonn Agreement, 2001], Article 2. 8 For an overview see: Lenny Linke, Dispute Resolution in Afghanistan,Presentation to the Rule of Law Board of Donors Meeting,undated 9 Afghanistan in 2015:a survey of the Afghan people,Asia Foundation,2015 10 2004. Constitution of the Islamic Republic of Afghanistan,Article one hundred and sixteen.
  • 20. Page 20 of 105 of Justice; the Police Law; the Elimination of Violence Against Women Law; a Defense Attorney Law; and the ‘Advocates Law’ of 2007, which established the Afghanistan Independent Bar Association (AIBA). A number of different international organizations have developed justice sector training and capacity- building programs over the last decade. Since 2008, considerable financial support has been given to justice sector reform and capacity-building. Training by different actors initially developed in an ad hoc mannerand sufferedfrom lack of coordination.11 Various attemptshave been made to improve this but it still remains fragmented. Tensions also arose due to questionable judicial appointments by the Karzai government, which made some international organizations reluctant to work with some Afghan justice institutions.12 Presidential elections in 2014 eventually resulted in an agreement to form a power-sharing administration betweenthe two leading candidates, AshrafGhani, who became President,and Abdullah Abdullah, who was made a Chief Executive Officer, with powers similar to that of a prime minister.13 The first round of the election took place in April 2014 and the second in June. The full details of the deal were not agreed until September 2014 and this led to an effective paralysis of some parts of the government for more than six months. The advent of the newadministration also led to senior personnel changes in Afghan ministries and justice institutions which, as will be discussed below, has clearly impacted on the JTTP’s transition strategy. The MTE was positive about the quality of the training and coaching delivered by the JTTP. It noted that while the delivery of some courses had been delayed and there were some areas where the training could have been improved, it had generally been effectively implemented in terms of quantity and quality. The MTE also found that the JTTP had been delivered in a cost-efficient manner, by reducing the costs of international staff and associated security costs, while at the same time ‘nationalizing’ its training team to its Afghan staff. It noted that the JTTP had developed useful internal monitoring and evaluation mechanisms, but it did not at that time have the capacity to assess the impact of the training in the targeted institutions. It stated that without such monitoring it was very difficult to identify the direct and indirect, medium and long-term results of such projects, nor to analyze how they can positively contribute to the goal of strengthening the Afghan justice system and restoring public confidence in it. The MTE made a number of recommendations about how the JTTP could become more proactive in looking for linkages between its work on strengthening the capacity of the justice sector and those organizations that are involved in monitoring its performance in relation to justice and human rights. The MTE was considerably more critical about the progress achieved under Outcome Two of the JTTP. It acknowledged that the extent of engagement in the target institutions varied from institution to institution and progress had been made in some areas. It also noted that many of the delays were due to a variety of factors, some of which were clearly beyond the JTTP’s control to influence, such as six months of impasse in the government after the disputed elections in 2014. The MTE found that despite these difficulties the JTTP had made considerable progress and a sustainable transition could still take place ‘if the project is granted a contract extension and supported by a follow-up considerably scaled down transition project.’ The MTE made a number of recommendations, which it was hoped would positively assist the JTTP strengthen its activities in the second part of the program and to address some of the systemic weaknessesofthe Afghan justice system. This evaluation will review the progress since the MTE focusing, in particular, on the issues described in the scope and purpose section of this report. 11 J. Alexander Their, Re-establishing the Judicial System in Afghanistan, Center on Democracy, Development and the Rule of Law, Stanford Institute for International Studies, September 2004. 12 International Crisis Group, Reforming Afghanistan’s broken judiciary,ICG, Asia Report N°195 – 17 November 2010. 13 Guardian, ‘Afghanistan presidential candidates strike power-sharing deal’, 20 September 2014.
  • 21. Page 21 of 105 3 MAIN EVALUATION FINDINGS RELEVANCE 3.1 EVALUATION QUESTION 1:TO WHAT EXTENT WAS THE PROGRAM THEORY OF CHANGE ABLE TO RESPOND TO THE CHANGING NEEDS ANDCONTEXT OF THE AFGHAN JUSTICE SECTOR, INCLUDING CHANGES TO THE POLITICAL AND SECURITYCONTEXT? The JTTP contains two objectives. The first of these is to provide direct capacity-building support activities – through training and coaching – to individual Afghan legal professionals within four target institutions: the Supreme Court, the Attorney General’s Office (AGO), the Ministry of Justice and Ministry of Interior. The second is to assist Afghan justice institutions so that they will be capable of taking on responsibility for building the capacity of their own personnel as part of a comprehensive and sustainable strategy of Afghan institution-led continuing legal education. The underlying assumption of the program is that this individual capacity development support can lead to demonstrable and sustainable improvements in the administration of the formal justice sector in Afghanistan, which, in turn, can ultimately lead to greater public confidence in this system. All Judges in the country come under the Supreme Court for the purposes of training. The AGO contains both Investigative Prosecutors and Trial Prosecutors – both of whom are supported by the JTTP. The JTTP’s support in the Ministry of Justice is for Defence Lawyers in the Legal Aid Department (LAD) and its support for the Ministry of Interior is for its Criminal Investigation Officers (CID)within the police. The JTTP’sfirst objective (Outcome One)is to provide direct capacitybuilding support for individual Afghan legal professionals. Its second objective (Outcome Two) is to develop a transition strategy to assist the above institutions so that they will be capable of taking on responsibility for building the capacity of their own personnel as part of a comprehensive and sustainable strategy of Afghan institution-led continuing legal education. As the MTE noted, the JTTP corresponds with both the Afghan government’s own goals and those of the US Mission in Afghanistan, its largest donor. It is also in line with IDLO’s Global Strategy Plan14 and the findings of the IDLO survey and publication: Women in Afghanistan’s Justice Sector.15 In interviews with representatives of the target institutions this evaluation team was repeatedly told that the capacity development support that they had received from the JTTP was relevant and appropriate to their needs.16 This view was also expressed in the FGDs, with many participants giving examples of how they support that they had received had helped them to be more effective in their work. During all interviews and FGDs during the current evaluation it was repeatedly stated that the criminal justice system is improving in Afghanistan and that Afghan legal professionals are using the knowledge that they have acquired from the JTTP to improve the quality of their work at both an individual and institutional level. The evaluation team spoke to over 100 individual interviewees and over 200 participants in the FGDs. This can be seen as a statistically significant cross-section of Afghan legal opinion and the view was expressed consistently by almost everyone. The courses that the JTTP provides – which will be discussed further below – are based on detailed needs-assessments and have been developed and modified to meet the specific needs of the target beneficiaries. Since 2014, all courses have been reviewed and commented on by the institutions 14 International Development Law Organisation: Creating a Culture of Justice Strategic Plan 2013-2016 http://www.idlo.int/sites/default/files/Strategic%20Plan%202013-2016.pdf, accessed 20 February 2016. 15IDLO: Out of the shadows,onto the bench: Women in Afghanistan’s Justice Sector, p.6 http://www.idlo.int/publications/womens-professional-participation-afghanistans-justice-sector-challenges-and, accessed 20 February 2016. 16 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
  • 22. Page 22 of 105 themselves, to ensure relevance to the work of their professionals. This also serves to ensure these institutions have the necessary curricular tools to deliver training on their own. The JTTP theory of change assumed that the target institutions were willing to take over full responsibility for the capacity development activities themselves and would allocate sufficient resources in terms of personnel, funding and equipment within the original planned duration of the program.17 This, however, proved over-optimistic for reasons that will be discussed further later in this report. The JTTP subsequently modified this strategy and its logical framework was updated according to the specifications of the transition strategy developed under Outcome Two. The transition strategy was based on the assessed needs of the target institutions, was drawn up in full consultation with them and was relevant to their needs and capacity. This was confirmed to the evaluation team in meetings with representatives of the targetinstitutions and is also borne out by its program documents. The JTTP now has ten outputs in comparison with the previous eight (five under Outcome One and three under Outcome Two, respectively). A list of 30 indicators was developed as a part of the logical framework reflecting both quantitative and qualitative results. 18 The defined indicators continue to reflect adequately what the program is doing and they also measure progress made towards the achievements of expected results. The original logical framework was quite specific regarding the activities under Outcome One, but rather imprecise under Outcome Two. The overall objective of the latter was clearly defined but there was no roadmap, the indicators were very broad and no timelines were given. This should not necessarily be seen as a criticism, however, as a case can be made that the more detailed strategy for the transition neededto be developed during the JTTP’simplementation. Memoranda of Understanding (MoUs) have now been signed with all four national justice institutions, which clarify the transition related co-operation for the period of the JTTP and potentially beyond, should another project continue to be implemented with the same target institutions.19 The first Monitoring and Evaluation (M&E) Plan was developed in June 2013 and an Evaluation Strategy was finalized September 2013.20 This has helped the program to develop its overall strategy in furtherance of the above objectives, taking into account the changing needs of the Afghan justice sector, although it seems that it would have been helpful to have had these systems in place earlier in the JTTP’sexistence. The M&E systemis connected to the US Government’s Afghanistan Performance Monitoring Plan (PMP)21 and provides a system of regular reporting to INL. The system also provides periodic and regular information to inform the JTTP’s own program management on progress made towards intended objectives, as well as to identify deviations from these targets and support decision making processes to strengthen the program’s effectiveness. The M&E system also fulfils donor reporting requirements, which include biweekly, monthly and quarterly reports.22 The program’s M&E system also evaluates the effectiveness of the JTTP’s individual capacity support activities, basedon the IDLO’sstandard evaluation approach to trainings basedon the Kirkpatrick four- level model. As will be discussed further below, the JTTP evaluates not just the perceived quality of the training and acquired knowledge, but also how trainees have changed their behavior, and applied the knowledge they have received in their work. It has also begun to measure the impact of these improvements on the quality of justice provided by the institutions themselves. 17 JTTP Logic Model July 2013 and reconstructed theory of change (undated) 18 JTTP Monitoring Evaluation Plan and Logframe of 21 April 2015. 19 Ministry of Interior January 2015, Ministry of Justice April 2014, Supreme Court undated,Attorney General’s Office January 2015. 20 JTTP M&E Strategy, version 3, 12 September 2013. See also: JTTP M&E Framework, 24 February 2014; and JTTP M&E System and Approaches,22 July 2014. 21 https://www.usaid.gov/afghanistan/performance-monitoring-plan, accessed 20 February 2016. 22 JTTP, Monitoring and Evaluation Concept Note,IDLO, undated,p. 1.
  • 23. Page 23 of 105 The MTE noted that some of the JTTP assumptions on the outcome level did not ‘reflect all necessary conditions which are essential for achievement of expected results.’ It cited as an example, ‘a political commitment to improve the justice sectorsystem and create favorable conditions for fair administration of justice, political will to fight corruption and allocate sufficient resources (personnel, funds and equipment) for ongoing professional development’.23 These assumptions were clearly over-optimistic. Afghanistan’s governance structures remain weak and became totally paralyzed during the 2014 election. The country’s Budget is parlous and its government is facing a fiscal crisis, while the security situation is visibly deteriorating every year. However, it is precisely because of the challenges that Afghanistan faces in relation to weak rule of law, corruption, intimidation and violence that makes programs such as the JTTP so necessary and relevant. A number of senior justice sector officials, stated that they felt the training and coaching provided by the JTTP had tangibly improved not just the skills of individual professionals but the culture of the institutions as well. These officials are also noticeably taking ownership of the transition process and some candidly acknowledged that the delays had been the responsibility of their own institutions. 24 Afghanistan’s Deputy Minister for Justice,for example, stated that ‘the JTTP has done wonderful work in helping us to change ourselves. Of course we really needan externalevaluation of our own institution to assess whether we are improving our own services to the Afghan people!’ 25 The Head of Afghanistan’s Supreme Court reportedly told IDLO staff in Rome that ‘you did not just give us a raft, you helped us to build a bridge.’26 There are,however, two areas where this evaluation team believe that IDLO may wish to reflect on the relevance of the JTTP’s activities when devising future programming. The first is the program’s exclusive focus on the official state criminal justice system. Although the reasons for this are understood in terms of program design and donor priorities, this evaluation team believes that ignoring both customary law and civil law significantly reduced the JTTP’s overall relevance in seeking to strengthen the capacity of the Afghan justice sector. It is widely accepted that around 80 per cent of legal disputes in Afghanistan are settled through customary mechanisms: Shuras and Jirgas.27 The role and importance of customary law increased in Afghanistan due to the break-down of the official institutions of law and order during the conflict.28 While the problems with the customary system are well-known,29 the evaluation team believes that the decision to ignore it completely when designing the JTTP,significantly reduced the program’s scope in addressing how justice is actually administered and perceived in Afghanistan. Linked to this is the exclusive focus on criminal law, which, by ignoring civil law entirely, meant that the JTTP was unable to address issues such as land and property rights, commercial law, and divorce, custody and inheritance rights. Given that the most contact that most people have with legal systems throughout the world is on such issues and they will generally judge the effectivenessof a justice system by its ability to handle them, it seems strange to omit this from a program whose overall goal is to strengthen public confidence in it. The omission is particularly regrettable in Afghanistan where the division between criminal and civil law is more blurred and where some disputes, particularly related to land and property, can encompass both civil and criminal issues.30 23 MTE, 2014, p.23. 24 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016. 25 Interview conducted in Kabul on 9 February 2016. 26 Interview conducted by Skype in Rome on 4 March 2016. 27 Lenny Linke, Informality and formal land administration, Working Paper, Policy advisory group on land – Sub-working group on formal/informal land issues,UNAMA Rule of Law Working Group, no date. 28 Ali Wardak, ‘The Tribal and Ethnic Composition of Afghan Society’ in Afghanistan:Essential Field Guides to Humanitarian and Conflict Zones (Second Edition), edited by Edward Girardet & Jonathan Walter, Geneva: Crosslines Ltd, 2003. 29 Dr. Martin Lau, Afghanistan’s Legal System and its Compatibility with InternationalHuman Rights Standards, The International Commission of Jurists, November 2002. 30 Conor Foley, A Guide to Property Law in Afghanistan,UNHCR/Norwegian Refugee Council, 2005.
  • 24. Page 24 of 105 3.2 EVALUATION QUESTION 2:HOWABLEWAS THE IMPLEMENTING ORGANIZATION TO RESPOND TO THEIDENTIFIEDNEEDS/PRIORITIES? As discussed above, a number of different international organizations have provided considerable capacity-building support to the Afghan justice sector. In 2005, INL established the Afghanistan Justice Sector Support Program (JSSP), which is implemented by a private sector contractor PAE.31 This has trained and mentored justice sector officials, helped to restructure criminal justice institutions and improved court administration and facilities. The JSSP established six provincial training teams located in Herat,Balkh, Nangarhar, Bamiyan, Kabul, Kunduz and Paktia, as well as a Kabul-based mobile team. These ran courses that brought together different groups of Afghan legal professionals and subsequently created ‘mentoring networks’ with advisors to visit practitioners at their offices to monitor and evaluate their progress. The JSSP also developed a Case Management System (CMS) to track the progress of criminal cases in the Afghan justice system. With the commencement of the JTTP, at the start of 2013, IDLO assumed responsibility for the professional legal training and individual capacity-development support previously provided by JSSP, while the JSSP/PAE remained responsible for other institutional support for these ministries and the CMS.32 In its original program proposal for the JTTP, IDLO stated that the program would ‘be implemented using a holistic approach to ensure that all partners involved will mutually benefit from the IDLO fully-coordinated and information-sharing approach.’33 The evaluation team is not aware of whether or not a baseline assessment was carried out before the start of the JSSP nor whether organizational assessments had been carried out of the target institutions. Aswill be discussed below in the section of this report on Impact,the JTTP did commission a consultant to produce a base-line assessment of the capacity of these institutions to implement training themselves as well as some aspects related to curriculum development.34 The study was eventually ‘valued unsatisfactory’, but it did reveal ‘inconsistencies between information provided by the institutions and the JSSP saturation study regarding reliable number of justice professionals actually employed in each institution.’35 The IDLOproposal statedthat the JTTP would be based,in part,on the training component of the JSSP. It noted that this had been ‘well-received and pre- and post-tests conducted in training programs have shown that participants have benefited in terms of increased knowledge and skills.’36 IDLO would ‘continue the positive aspects of the JSSP training programs, in particular the Advanced Continuing Legal Education Course for Afghanistan (ACLEA) training course and other shorter workshops which have been previously designed and implemented.’ It would, however, ‘move beyond the previous iterations of the JSSP, with an increased emphasis on results evaluation, as well as a clear focus on sustainability through transition to Afghan ownership of training programs.’37 It also undertook to conduct an ‘assessmentofthe delivery modalities with a view to enhancing, if necessary,its pedagogical quality and impact’ and to train all existing ACLEA Afghan trainers on IDLO’s own training methodology.38 31 See JSSP website http://jsspafghanistan.com/ accessed 18 February 2016 32 Ibid. 33 International Development Law Organization, Completing the transition in Afghanistan:Justice Training Transition Program (JTTP), program proposal submitted to U.S. Department of State Bureau of International Narcotics and Law Enforcement for Afghanistan and Pakistan,IDLO, 21 December 2012, [Hereinafter IDLO Proposal, 2012] p.8. 34 Baseline Assessment Report of the Justice System of Afghanistan and Justice Sector Capacity Building, June 2013. 35 JTTP Quarterly Report Jul-Sep 2013, 31 Oct 2013. 36 Ibid., p.8. 37 Ibid. 38 Ibid., p.20.
  • 25. Page 25 of 105 The JTTP was clearly designed to build upon the INL’s JSSP regional training component for Afghan legal professionals and it inherited both many staff and facilities from the latter. This brought savings in terms of continuity and institutional knowledge. It also enabled the JTTP to continue – almost without interruption – the training component activities that had originally been designed under the JSSP while new staff members were being recruited and the curriculum and teaching methodology was being revised and developed. It appears that the number of training and coaching activities that the JTTP carried out was also based on the numbers carried out under the JSSP and this also defined its geographical scope. As will be discussed further later in this report, the JSSP was able to deliver – and even exceed its original targets and so these can be said to have been realistic, at least as far as Outcome One was concerned. The evaluation team, however, encountered two criticisms, from various sources, relating to the way in which the JTTP was essentially passed from one implementing organization to the other, as well as a broader concern about the way in which what had previously been a single program was divided between two different implementing organizations. In January 2014 the US Special Inspector General for Afghanistan (SIGAR) published an audit report on reconstruction support for Afghanistan’s justice sector, which made a number of specific criticisms of the JSSP under its previous private sector management,but also raised concern about the decision to transfer the program from PAE to IDLO. It questioned, in particular, IDLO’s capacity to manage such a large program and referred to challenges facing its headquarters in Rome at the time, which it claimed raised ‘serious questions about the future sustainability of the organization’.39 The MTE did not encounter any evidence to justify the fears and neither did the current evaluation. As will be discussed further below, in the section of this report on Efficiency, IDLO did succeed in scaling up its administrative and logistical support for the JTTP, which, although large by IDLO’s previous standards is actually quite modest when compared to similar programs in Afghanistan. Indeed, it appears that the extra capacity that IDLO has developed to implement the JTTP could be used to take on other larger scale programs in both Afghanistan and elsewhere. The overwhelming majority of interviewees spoken to by the present evaluation team stated that they considered the capacity- development support provided by the JTTP to be a considerable qualitative improvement to that provided under its predecessor. This view was expressed in interviews with Afghan justice sector officials, UN staff, representatives of non-government organizations (NGOs) and participants in the FGDs who had received trainings under both programs.40 Indeed it wasvery noticeable that institutions such as the Supreme Court and the AGO, who had previous expressed some skepticism about aspects of the capacity-building support that they had received from the JTTP during the MTE, now declared themselves to be far more satisfied with the program’s implementation.41 The other criticism that the evaluation team heard about the handover focused on perceived weaknesses in the program that the JTTP inherited. In particular, it was stated – from a range of interviews and in FGDs – that the JSSP had placed too great an emphasis on the number of trainings and participants in each course,rather than the quality of their legal content and delivery.42 The terms of reference of both the MTE and the current evaluation preclude a more detailed examination of the original JSSP training program and so the evaluation team is in no position to make findings on these issues. As will be discussed below, under the section of this report on Effectiveness, however, there is evidence that the quality of the capacity-development support provided by the JTTP has improved during the period of its implementation and has also improved since the MTE. 39 Special InspectorGeneral for Afghanistan, Reconstruction Support for Afghanistan’s Justice Sector:State Department Programs Need BetterManagement and Stronger Oversight, SIGAR 14-26 Audit Report/State Department Support of Afghan Justice Sector, January 2014. 40 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016. 41 Interviews conducted in Kabul on 9 and 10 February 2016. See also MTE, 2014, p.33. 42 Interviews conducted throughout the Field Mission, 28 January – 15 February 2016.
  • 26. Page 26 of 105 The evaluation teamalso heard from both the donor and some representativesof Afghan national justice sector institutions that IDLO was the right strategic partner for INL to choose in implementing the JTTP, particularly given the emphasis on sustainable transition contained in Outcome Two. IDLO is an inter-governmental organization (IGO), with a 30-year history of delivering legal education and capacity support in the global south. It has been working in Afghanistan since 2001, implementing a number of programs in the field of justice sector and legal reform in partnership with the Government of Afghanistan, which made their state a Member Party to IDLO in November 2012.43 IDLO also has political and diplomatic leverage as an IGO. IDLO’s Director General, Irene Khan, attended the launch of the program in March 2013 in Kabul. In May 2015 she visited Afghanistan again where she held bilateral meetings with senior government ministers including: the Minister of Justice, the Minister of Women’s Affairs, the Acting Chief Justice, the Acting AGO, the Deputy Minister of Interior, the Chairperson of the Afghanistan Independent Human Rights Commission and both the President and Vice President of the country. Khan also met with the UN Deputy Secretary General’s Special Representative for Afghanistan, the Senior US Civilian Representative in the country and senior INL officials.44 It is clear that IDLO has some comparative advantages in implementing a program such as the JTTP. While perhaps the policies and procedures of an IGO may be more unwieldy than the flexibility and rapid responsiveness of a for-profit contractor,these would seem to be far more than outweighed by the commitment to long-term sustainability that IDLO can provide to one of its own member states. The quite different organizational cultures of the two different implementing organizations, may, however, have inhibited communication between them. It is clear that the claim in the IDLO program proposal that it would establish a ‘fully-coordinated and information-sharing approach’ between the JTTP and the JSSP was not always achieved. There were serious breakdowns of communication between the two organizations at particular moments in time, some of which may have been related to internal staff turn-over within the JTTP. The appointment of a new JTTP Chief of Party in October 2014 does, however,seemto have easedthese difficulties and both organizations reported a much better and closer working relationship in the second part of the program’s implementation. The JTTP Project had more or less kept the geographical coverage of the JSSP but gradually reduced the international presence in the field offices. With the closing of the Provincial offices in Badakshan and Kunduz though the reachout to beneficiaries wasnegatively affected.45 The Mobile Teamwasused to reachout to difficult geographical areas.46 Trainings held at the provincial hubs for surrounding areas has allowed for greater participation of women.47 The decentralised implementation and methodology was appreciated and praised by the beneficiary institutions as one of the greatest assistance in the transition both serving as a link between the local institutions in Kabul and the provinces and as complementary capacity building when local budget funds are lacking.48 The project has taken into account ethnic and linguistic differences when necessary and has paid attention to the specificities of different regions (certain provinces more progressive than others). The M&E has kept geographically differentiated data for each type of training and institution. Relevant both for cost-saving and minimising security risks in the filed was the frequent co-location of training venues and accommodation. It was though often difficult for female participants to travel and to plan overnight stays in province hubs.49 43 IDLO Homepage, where we work, Afghanistan, http://www.idlo.org/Archive/WhereWeWork/afghanistan/Pages/default.aspx,accessed 18 February 2016 44 JTTP Quarterly Report Apr-Jun 2015, 31 July 2015 45 JTTP Quarterly Report December 2015 p. 46 JTTP Quarterly Report December 2015 p. 9. 47 Ibid. p. 10. 48 Interview with the Supreme Court JED 49 Focus Group Discussion for women 2 February 2016
  • 27. Page 27 of 105 EFFECTIVENESS 3.3 EVALUATION QUESTION 3:WAS THE JTTP SUCCESSFUL IN DELIVERING THE EXPECTED OUTPUTS WITH A FOCUS ON INCREASED QUALITYAT THE SAME TIME? By December 2015, the JTTP had delivered over 382 legal training courses to more than 8,900 graduates,in total.50 This included: 69 ACLEA courses for almost 2,000 participants; 44 ACLEA Plus for 694 participants, and 247 CLE for more than 5,990 participants. The JTTP’s capacity-building support under Outcome One involved delivering just over 4,500 days and 27,000 hours of training. These figures include some double-counting, because if an individual has participated in an ACLEA course and a CLE, she or he will be counted twice, and JTTP staff estimated the individual number of recipients of this capacity-development support was a bit over 5,000 people.51 Around 13 per cent of its direct beneficiaries were women and 87 per cent men. As well as delivering the standard ACLEA course, the JTTP developed a special ACLEA tailored for members of the Supreme Court. This has been delivered to 305 participants, who are counted within the numbers given above for its ACLEA courses. The program also provided around 23,300 hours of follow-on coaching in just over 13,000 individual sessions to 2,355 recipients. The JTTP hasmaintained a monthly record of the numbers of training courses that it delivers in its regular reports and these show that it has completely fulfilled – and in some cases exceeded – all of its original numerical targets under Outcome One. The JTTP has also conducted twenty-two Training of Trainer (ToT) courses for 294 participants and has certified 146 GIROA Trainers.52 These trainers have themselves delivered over 11,300 of training under Outcome One. The JTTP has delivered a total of 16 trainings, and 504 training hours to 219 potential future trainers under Outcome Two and provided further personal support to these through 619 coaching hours. It has also refurbished 16 training rooms and is in the process of handing over curriculum material and training handbooks and manuals. The JTTP has become more effective at developing co-ordination and outreach mechanisms with both the national Afghan institutions and other international stakeholders. Reporting, and monitoring and evaluation systems have been developed, which have been used for effective communication with the donor, strategic planning and measurement of the program’s results. The JTTP has also become increasingly effective at working with the target justice sector institutions to define the assistance that it can provide towards establishing and supporting operational in-house training departments. This has included helping each institution to establishing governance arrangements and helping to recruit and train instructors for the units once these have been approved. The MTE noted that the JTTP could have been ‘more pro-active in looking for linkages between its work on strengthening the capacity of the justice sector and those organizations that are involved in monitoring its performance in relation to justice and human rights.’53 It noted that while the JTTP was a member of the Criminal Law Reform Working Group (CLRWG), and participated ‘in occasional policy and planning meetings with INL, the JSSP, and Afghan line-ministries, it seemed much less present in the wider international community of agencies represented by the UN Country Team, individual UN agencies, and the World Bank.’54 These criticisms may have been unfair at the time, since the JTTP’s Chief of Party had resigned shortly before the MTE was carried out and IDLO was in the process of recruiting a new one. The new Chief of Party, who joined JTTP in October 2014, has significantly transformed this situation and is widely known and respected within by both Afghan and international actors. She clearly takes her 50 JTTP Report Card dated January 2016 (Reporting period January 2013-December 2015) 51 Interview carried out in Kabul on 7 February 2016. 52 JTTP Report Card dated January 2016 (Reporting period January 2013-December 2015) 53 MTE, 2014, p.8. Executive Summary. 54 Ibid., p.50.
  • 28. Page 28 of 105 ‘Ambassadorial’ role extremely seriously and engages in active and ongoing work to raise awareness about the JTTP’s transition efforts, and seek support on key transition objectives. She holds regular cycles of meetings with the target national justice sector institutions, and also actively participates in UNAMA justice sector and rule of law meetings. One member of the evaluation team attended one such meeting, which included representatives of UN Agencies; the French, German, British and US Embassies, the European Union, USAID,Italian Cooperation, the UK Department for International Development (DfID),the US army, the North Atlantic Treaty Organization (NATO),the US Institute for Peace (USIP),the Asia Foundation, the Afghan Analysts and various civil society groups.55 The importance of maintaining and cultivating such contacts is now clearly understood within the JTTP. The JTTP has also established a Cordination working group for International organizations/donors working with the Supreme Court. This has met periodically and its objective is enable the Judicial Education Department (JED) to manage trainings conducted by various organizations. IDLO is also a member of the International Training Coordinatioms group, which consists of international advisors who mentor the Ministry of Interior’s Training Group. The MTE had also recommended setting up a ‘Transition Unit’ of international and national staff who would be relinquished from their activities coordinating the delivery of training to work full-time on the transition strategy.56 An Institutions Team was subsequently created in 2015, and now consists of two international and eight national JTTP staff, plus administrative support. JTTP staff members are now embedded in each institution ensuring effective day-to-day cooperation over the transition arrangements. The JTTP has also developed a clear understanding of the structures and internal dynamics of each institution and this has helped it to devise tailored approach strategies along with back-up plans and alternatives when these have been blocked or stymied by external events. As one senior international JTTP staff member explained: When we found that one of the institutions that we were working with did not have any budget in place for training the following year we decided to re-sequence our activities and began to work in the margins to see what could be accomplished without this. We started working on their M&E Records Database,to identify who had already received training, and cooperated more closely with other actors to fill any training gaps and establish coordination mechanisms. The idea was to put the elements of the training department in place for when budget approval finally came. We learnt that you always have to have a plan B.57 In early May 2015 IDLO convened a transition workshop to engage its institutional stakeholders at management and operational levels within the Supreme Court, the AGO and the Ministries of Justice and Interior. This was attended by senior staff within the training human resources and planning structures of the four institutions and was used to review transition plans, agree on key deliverables and milestones. The meeting was also attended by the World Bank, PAE, European Union Police Mission (EUPOL), UN Development Programme (UNDP) and Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ). The JTTP has also facilitated both the Ministry of Justice and the Supreme Court to convene meetings to present their own training institutional work plans and budgets, and share scheduling information and participant data with other international organizations, to avoid repetition and duplication of training activities. Underpinning such efforts is an understanding within the JTTP of the need to cultivate long-term personal relationships with key players in each target institution. This is particularly important given Afghanistan’s political and institutional culture. The JTTP has been able to root its plans in Afghan realities, which are often more messy and imprecise than can be adequately presented in a graph or matrix. The evaluation team believes that such efforts, while time-consuming and often arduous, are essential to long-term sustainability and are too often neglected by international organizations, who over-prioritize quantitative achievements. 55 Rule of Law Donors Board Meeting, Kabul, 10 February 2016. 56 MTE, 2014, p.61. 57 Interview conducted in Kabul on 7 February 2016.
  • 29. Page 29 of 105 In its original program proposal IDLO had stated that it would continue to deliver the basic ACLEA course, and other shorter coursesand workshops, which had beenpreviously designed and implemented under the JSSP. It would also design and deliver an additional, more in-depth, ‘ACLEA-Plus’ course, for past ACLEA graduates and undertake an ‘assessment of the delivery modalities’ of all courses with a view to enhancing their pedagogical quality and impact’.58 It also noted that, given the baseline from which it was starting and the well-known challenges facing the justice sector in Afghanistan it was simply not realistic ‘to plan for a situation whereby in less than three years all Afghan citizens have access to a court which is able to deliver criminal justice in-line with the law and international human rights standards.’ 59 The JTTP instead aimed ‘to provide a significant number of justice sector professionals with the capacity to fulfill their duties as defined by the Afghan Constitution and related laws and international commitments, so that their improved capacity builds public confidence in the justice sector.’60 The legal capacity building component of the JTTP was extensively covered by the MTE.61 This noted that although curriculum and lesson plans for each topic of the ACLEA and CLE existed, lectures were initially delivered class room style, there had been virtually no hand-outs or overhead visual materials provided and that the training delivery did not follow basic andragogical rules to allow greater interaction between the trainer and participants. JTTP staff members had begun to make changes to the course delivery, presentation styles, and hand-outs from the start of the program although much of this was done in an ad hoc manner by individual training teams.62 JTTP started to revise the ACLEA curriculum in late 2013 and this was finalized in April 2014. The new course was structured in six topics, reduced from 10. JTTP also finalized the new ACLEA-Plus curriculum in June 2014 and started piloting courses that November.. The MTE noted that there had been ‘good consultation about the development of the new curriculum’, and stated that ‘the content sounds encouraging, although the fact that these courses are still in the development stage so late in the program’s development is a cause for concern.’63 Since the field mission for the MTE took place in August and September 2014, the evaluation team was unable to observe any of these courses. The present evaluation team was,however, able to observe both ACLEA and ACLEA-Plus courses and also to discuss their development in individual interviewees and FGD participants. It was also able to review the most recent version of the JTTP’s curriculum and discuss the plans for handing this over to the Afghan national institutions with the JTTP’s Curriculum Unit. The evaluation team was extremely impressed by the parts of the ACLEA-Plus course that it observed. The emphasis of the training is on the practicalapplication of skills, with groups of Afghan legal professionals – from different institutions – working together on particular cases. In the ACLEA-Plus course that one evaluation team member observed, one group of CID Officers was working together with Investigative Prosecutors from the AGO to gather evidence during a crime scene investigation (CSI),while Trial Prosecutors and Defense Lawyers discussed together how evidence collected could be presented at trial. It was explained that Judges would be brought into the course the following week to preside over a mock trial, which all course participants would be able to witness, and where all the previous procedures and evidence gathered would be reviewed in courtroom conditions.64 58 IDLO Proposal, 2012, p.20. 59 Ibid. 60 Ibid. 61 For further discussion see MTE, 2014, pp.27-33. 62 Ibid., pp.31-3. 63 MTE, 2014, p.32. 64 Interviews and observations in Kabul on 9 February 2016.