Primer on Performance Audit to measure Institional effectiveness

1. Performance Audit

2. What is Performance Audit - I?
INTOSAI Auditing Standard 1.0.40
Performance audit concerned with the audit of economy,
efficiency and effectiveness and embraces:
Audit of the economy of administrative activities in
accordance with sound administrative principles and
practices and management policies;
Audit of the efficiency of utilization of human, financial
and other resources, including examination of
information systems, performance measures and
monitoring arrangements and procedures followed by
audited entities for remedying identified deficiencies; and
Audit of the effectiveness of performance in relation to
the achievement of the objectives of the audited entity
and audit of the actual impact of activities compared with
the intended impact

3. What is Performance Audit - II?
“Performance auditing is an
assessment of the activities of an
organization to see if the
resources are being managed
with due regard for
economy, efficiency and
effectiveness and accountability
requirements are being met
reasonably.”
Muhammad Akram Khan, D-

4. Our Legal Mandate for Performance Audit
Comptroller & Auditor General’s
(Duties, Powers & Conditions of Service)
Act, 1971
16 17
15 19
14 20
13 23

5. Objectives of a Performance Audit
Quality of information and advice available to government
for the formulation of policy
Existence and effectiveness of administrative machinery to
inform the government whether program objectives and
targets have been determined with a view to fulfilling policy
objectives
Whether and to what extent, stated program objectives
have been met
Economy, efficiency, effectiveness, equity and ethics of the
means used to implement a program/activity
Intended and unintended, direct and indirect other impacts
of programmes/activities, e.g. environmental impact of
government activity
Compliance with applicable laws and regulations in the
context of the performance audit objectives

6. Performance vs. Regularity Audit - I
Features Regularity Performance
Generally covers a Covers the program over a
financial period period of time
Coverage is for the Focused only on a part of the
Scope whole of the entity for entity’s
the period programs
Audit to be
completed by a Coverage is selective
stipulated time

7. Performance vs. Regularity Audit - II
Features Regularity Performance
Attestation of Covers the subject/program over a
financial accountability period of time;
Audit of financial systems Focused only on a part of the entity’s
activities/program
Audit of internal control for Coverage is selective
safeguarding assets and
completeness and accuracy of
Accounts
Evaluation of financial records Audit of economy, efficiency and
effectiveness
Objectives Audit of propriety of Audit of internal controls – 3Es,
administrative decisions adherence to management policies,
timely and reliable financial and
management information
Limited to financial matters Extends to non-financial /governance
issues too
Test for assuring compliance to Assessment of compliance to
laws, regulations and rules applicable laws and regulations
required in the context of audit
objectives
Opinions generally Conclusions related to audit
with reference to standardized objectives set by auditors
requirements

8. Performance vs. Regularity Audit - III
Features Regularity Performance
Financial statements Variety of forms of evidence
per se, accounting
documents, etc.;
Budgetary
assumptions and Quite often qualitative in
appropriation nature
authorizations, etc.;
Evidence
Transaction
documents
Conclusive nature of Persuasive rather than
evidence conclusive - evidence related
to pre-determined
audit objectives
Materiality by amount Materiality guided more by
the nature or by context
rather than amount alone

9. Performance vs. Regularity Audit - IV
Features Regularity Performance
Academic Always knowledge-based: of
base Generally laws, social sciences,
accounting economics, development
knowledge studies, public affairs, science
and technology, etc
More or less Varies widely from subject –
Approach & standardized: manual generally serves as
methodology manual generally basic framework
forms the
entire basis
Widely varying and subjective
with ample scope for
Assessment Standardized interpretations & assessment
Criteria with little scope criteria support the audit
for subjectivity objectives, which are unique
to the subject of performance
audit

10. Performance vs. Regularity Audit - V
Features Regularity Performance
Opinion on the Conclusions on 3Es
financial
statements;
Opinion on Report presented in
compliance to variety of forms
laws, regulations and
rules
Report
Generally opinion on Wide-ranging in nature,
financial statements open to interpretations
in standardized and subjective judgment
format
Related to specific Generally separate
financial periods- publications on each
there is a periodicity subject of performance
of reporting audits
as and when conducted

11. Typology of Program Evaluation
Type Objective
Process Assesses program activities’ conformance to
statutory and regulatory requirements, program
design and professional standards or
expectations
Outcome Focuses on processes, outputs and outcomes
(including unintended effects) to judge program
effectiveness
Impact Compares program outcomes with an estimate of
what would have happened in its absence -
employed when external factors are known to
influence the program’s outcomes
Cost Assesses the least costly alternative to meet to
Effectiveness achieve a single goal or objective

12. Changing Jargon & Avatars of Performance Audit

13. New Dimensions of PA’s Operating Environ - I
New models of governance
Greater role for NGOs in program/service
delivery
Public Private Partnerships
Direct transfer of development funds to
provinces
Enhanced delegation of spending powers to
relatively junior state officials
Commercial exploitation of nation’s natural
resources by private sector & their regulation
Mass privatization of public utilities & their
regulation by the state

14. New Dimensions of PA’s Operating Environ - II
Mass privatization of public utilities & their
regulation by the state
Rising education and standard of living begets
greater demand for accountability
Better education also provides better skill sets to
SAI personnel
Declining standards of probity in public life
Young generation no longer used to government
inefficiency, waste and corruption
Right to Information Act, Lok Pal Bill, etc.
Increasing judicial intervention for accountability
Corruption affects foreign investment and
retards domestic saving and investment

15. Why is Performance Audit good for SAIs?
Concentrates the audit effort in a public
program/project
Utilizes slack in diverse skill sets and
synchronizes multi-disciplinary evaluation by SAIs
Administrative-cum-financial external appraisal
synched with program is greater help to policy
planners, makers and implementers
Synergize policy delivery mechanisms with
concurrent accountability
Well-suited for developing countries where
government still delivers most public programs
Greater public visibility, debate and
accountability of public resources

16. The Performance Audit Schema
Economy Efficiency
Effectiveness
Performance Audit

17. The Performance Audit Schema

18. Key Terms of Performance Audit
Accountability Efficiency
Activities Management
Assessment Organization
Economy Reasonableness
Effectiveness Resources

19. Accountability & Reporting
Legislative control over public finances
established by:

20. Assessment & Reasonableness
• Auditor formulates a judgment on the
basis of relevant and reliable evidence,
discusses with management and validates
I procedures
• Considers constraints of the management
and operating environment
II
• Does not base its judgment on hindsight
wisdom
III

21. Organization, Management & Resources
Performance auditing takes an overview of the
activities and functions of an organization as a
whole. In exceptional cases, it may review an
isolated project or a programme as well
Resources of an organization consist of
money, men, materials, and machines.
Performance auditing reviews all these
resources
Management covers such functions as planning,
organization, resourcing, directing and
controlling. Performance auditing reviews all
these phases of management cycle. It is not
concerned merely with the 'use' of resources.

22. The 3E Schema

23. What are the 3 Es?
Economy
Acquiring resources at the lowest cost keeping in view the
objectives of the organization
Efficiency
Relationship of inputs and outputs - utilization of
resources
Effectiveness
Extent to which an organization achieves its objectives

24. How is PA different from Internal
Appraisal?
Audit Finding
Criteria Cause
Conditions Effect

25. Criteria Characteristics
Reliable
Comparable Acceptable
Comprehensible Complete
Objective

26. What is an Audit Criteria?

27. What is Audit Criteria in Real Life?

28. A Program Logic Model
Socio-economic Problems
Needs Objectives
Action
Inputs Processes
Outcome
Output Results
Impact on Socio-economic Problems

29. Where do the 3Es fit in?
Allocation of Inputs
Use of Inputs
Processes
Outputs
Results > Objectives
Impact > Objectives

30. What is Economy?
Spending only that much which is
barely essential to achieve
organizational goals
For example, if lowest bidder leads
to interruption in operations, it may
not be the most economical manner
of buying these supplies
Therefore, while reporting on
economy the, auditor should keep
in view effectiveness of the
organization in achieving its goals

31. How do I judge economy?
Right Right Right
Time Place Cost
Capital
Demand
Idle HR
Operating
Procurement Maintenance
Automated Down Time
assembly
Availability
Salvage Value

32. Yardsticks for Judging Costs
• Costs estimated in the form of streams over the
life-time of the asset
I
• Cost streams then discounted to the present
using an appropriate discount rate
II
• Not a simple comparison of one-time capital
cost, usually done in traditional auditing
III
• Calculation of minimum cost presumes
estimates of cost would be prepared for all
IV available alternatives

33. What is Efficiency?
An economic state is one in
which every resource is
optimally allocated to serve each
person in the best way while
minimizing waste and
inefficiency

34. Can Government Always be Efficient?
• Unlike profit-making entities, government is a
I developmental entity in which there are few choices
• Government is slave to people’s will represented by
II the Legislature
• Government focuses on entire nation and its
III problems
• Availability & Flexibility of use of resources limited
IV
• Public policy is therefore a political animal
V
• Yet Governments remain accountable to the people
VI

35. How do I measure efficiency?
• Are all levels of the organization aware of their
I goals?
• Is there a reliable system in place to monitor the
II output of each level?
• Does management receive regular reports on
III current levels of efficiency or services rendered?
• What are the measures taken by management to
IV improve its efficiency?
• What efficiency measures are feasible in the
V given situation?

36. How does efficiency relate to productivity?
Productivity = Amount of goods or services
produced / amount of resources used i.e. ratio
of input and output or capital output ratio
Efficiency is the relationship of actual
input/output (productivity) to a prescribed
performance standard
Time taken for producing 80 bags of cement is
one machine-hour is productivity of the plant
This level of productivity may however, be 80%
of the standard which is 100 bags per machine-
hour
Efficiency is, therefore, measured in terms of
rate of return of production, work done in a time
span or unit cost

37. Prerequisites for Measuring Efficiency
Outputs should be uniform and repetitive, e.g. filling in
prescription by the pharmacist in a hospital
Non-uniform output like police programme, writing
press briefs by Information Department etc., cannot be
measured
There must be an agreed standard of performance
against which actual output has to be measured, e.g. in
case of a school the number of class-contact hours for a
teacher in a week should be laid down as a standard
If standards do not exist then output may be measured
with previous years
Sometimes measurements are compared between two
geographical locations, or, comparison with private
sector - data of past years should be available

38. Measuring Efficiency in Government - I
Assess whether the objectives of and the
means provided (legal, financial, etc.,) for
a new – or ongoing – public sector
program are proper, consistent, suitable
or relevant to the policy
Assess the effectiveness of public sector
programs and/or individual components,
i.e. assess whether objectives are met
Assess whether the observed direct or
indirect social and economic impacts of a
policy are due to the policy or to other
causes, but only if it can be established
with evidence

39. Measuring Efficiency in Government - II
Identify factors inhibiting satisfactory
performance or goal-fulfilment
Assess whether the programme
complements, duplicates, overlaps or
counteracts other related programmes
Assess the adequacy of the
management control system for
measuring, monitoring and reporting a
program’s effectiveness
Identify ways of making programmes
work more effectively

40. Advantages of Efficiency Measures
I
• Control quality and quantity
II
• Demonstrate achievements
III
• Prepare Budgets and Plans
IV
• Rationalize pricing of products
V
• HR performance appraisal

41. Efficiency Measurement Tools– I
Inter-authority Comparison
Comparison with similar
authorities, delivering similar programs and
operating in comparable environments with
following illustrative parameters:
Operating expenses
Usage of service or the client population
Unit cost of service provision
Manpower levels
Performance indicators

42. Efficiency Measurement Tools – II
Internal Comparison
Comparison of costs between different services
or facilities provided by the same authority at
different locations or in different departments
For example, repair costs by class of vehicle, or
energy consumption by schools or residential
units
Large range of performance indicators can be
developed by the auditor from the internal data
such as:
ratio of administrative cost to operational costs
ratio of over time payments to total payroll
number of issues per hour in a library

43. Efficiency Measurement Tools – III
Past Performance & Targets
Comparison with past years may
show potential areas for saving
Auditor should analyze cost and
output trends
Targets set and achieved by the
management for itself should be
used as comparison measures

44. Judging Effectiveness - I
• Are objectives precisely defined ?
I
• Are objectives based on reliable facts and
II forecasts?
• Are policy instruments to operationalize
objectives based on detailed analysis of
III alternatives?
• Can progress be reasonably attributed to the
IV activities and policies of t he auditee?
• Does the auditee adequately monitor the
V effectiveness of a policy instrument?

45. Judging Effectiveness - II
• In measuring effectiveness a
distinction needs to be made
I between output and outcome
• Output relates to results of
certain inputs within the
II organization
• Outcome relates to the results
external to the organization
III

46. Performance Measures for Effectiveness
Three main problems:
Jointness: Where a number of different policies may
contribute to satisfying unmet needs
Ex: Educational standards may be affected by the size
of classes, the quality of teachers and the supply of
equipment
External factors: May affect the outcome of a project or
programme
Ex: Income and social status of consumers
Cost: Sometimes programmes cannot be carried out in
the most effective manner due to prohibitive cost.
Ex: It may be more effective to have more teachers than
to provide more books. But the cost of the former may be
prohibitively high.

47. What should the Auditor do?
Decide indicators in consultation with
management
Also keep in view any national or
international standards already available
Objectives laid down in the plan may
be taken as a bench mark for some of
the outcomes
In no case should the auditor
restrict his audit criteria to the
original plan only

48. Some PA Tools We Use
I • Survey
II • Statistical sampling
III • Benchmarking
IV • Focus groups
V • Interviews
VI • Case studies
VII • Qualitative data analysis
VIII • Program Logic Model
IX • Flow charts

49. Potential Impact Areas of PA
Economy Reduction in costs through better
contracting, bulk buying, etc
Reduction in costs through economies on usage
of personnel or other Resources
Greater outputs from same inputs
Efficiency Remedying duplication of effort or lack of
coordination
Better sub-objectives and targets
Effectiveness Better achievement of objectives by changing
the nature of outputs or improved targeting
Shorter waiting lists
Service Improved equity in access to programmes
Planning & Better control and management of human
Control resources, assets, projects and resources
Improved forms of account, including
commercial formats
Accountability Better and/or more accurate performance
indicators

50. PA of I-T Systems - I
Strategic and operational management of IT within
the entity, including assurance that IT is included in
the overall corporate governance of the entity
IT project management within the entity, including
the entity’s record in meeting legislative and other
deadlines;
The risk management practices of the entity in
relation to IT
IT system design, development and maintenance
controls
Compliance with standards, including external
standards
Application controls
Processing controls, including audit trails

51. PA of I-T Systems - II
Business continuity arrangements
Data integrity, including sampling of data
(possibly using CAATs)
Access controls and the physical and logical
security of networks and computers, including
Internet firewalls
Controls to safeguard against illegal software
Performance management and measurement
Other issues that arise during the audit

52. Software used in I-T PA Audit
Teamrisk – risk assessment tool for
preparing risk based audit plan
TeamMate Teamschedule – tool for scheduling staff
(PWC) resources to audit
Teammate (EWP) – electronic work paper
system
Teamcentral – web-based issue tracking
system
Crystal Ball Uses Monte Carlo simulation in
(US GAO) conjunction with a cost estimation
spreadsheet on the costs to give a range
of values and the probability or the
certainty of the values - useful for
performing cost and uncertainty analysis
Earned Value Analysis Measure progress of project
Provide schedule and budget variances
on a particular date
Forecast completion date and cost of
completion of project

53. Sources of Performance Measures
Citizen surveys: Especially in areas of social
services, police, garbage collection etc. - extreme
care be exercised in determining sample size and
preparing questionnaires
Trained observer ratings: Used to rate conditions
in a particular area, using a pre-designated rating
criteria such as street cleaning
Industry standards: Comparison of the results
with prevalent standards in that area or with similar
programs
Internal records: Occupancy rate for residential
homes, user complaints, etc.

54. Some Words of Caution - I
Auditor not expected to carry out the
above exercise himself
Primary duty is to see if the auditee has
a reliable system to evaluate its own
effectiveness
In that case, he would focus his attention
on the adequacy of these procedures
In case the auditee does not have a
reliable mechanism to measure its own
effectiveness the auditor may have to do it
himself

55. Some Words of Caution - II
Simple explanation of possible reasons for
shortfall in achievement of goals not enough
Ex: Not enough to say that the agency could not
achieve its goals because of cost over run or time
over-run
Instead auditor should analyze the reasons in
following two categories:
Factors that could have been influenced by the
agency
Factors that could not be influenced by the
agency such as Government policy, applicable
rules, socio- political environment, appropriation
of funds and availability of trained manpower, etc.

56. Format of PA Report - I
If the management has a system of effectiveness measurement
the PA auditor:
Documents the management's system of effectiveness
measurements
Assesses the appropriateness of performance indicators
Verifies the results and compares them with the appropriate standard
Formulates an opinion on the effectiveness of management's system
Effectiveness measures differ from efficiency measures
Ex: A government department has the mandate to develop and
economically backward region
Efficiency measures would be the number of industries which
moved to the backward region
Effectiveness measures would relate to the figures of unemployment
or income level

57. Format of PA Report - II
If there does not exist (or is weak) a system of
effectiveness measurement then the auditor
has to generate his or her own data
Auditor should be careful about:
Externalities of the programme
Cost of data collection
Method should be agreeable with
management

58. Some Sample Report Layouts - I
Report No. 9 of 2009-10, National Rural Health Mission
Executive Summary
Introduction
Planning & Monitoring
Convergence
Fund Flow Management
Infrastructure Development & Capacity Building
Procurement & Supply of Medicine & Equipment
Information, Education & Communication
Achievements in healthcare
Conclusions & Summary of Recommendations

59. Some Sample Report Layouts - II
Report No. 35 of 2010-11
Management of Defense Estates
Preface
Summary of Recommendations
Introduction
Land norms, records & ownership
Land use
Management of Leases
Management of old grant bungalows
Conclusion

60. Some Sample Report Layouts - III
Report No. 18 of 2010-11
Taxation of payments to non-residents
Preface
Executive Summary
Introduction
Oversight on forex flows
Controls on Remittances
Sectorwise Analysis
Mistakes in Assessment

61. Concluding Concerns
• PA reporting an administrative appraisal
I than financial/certification audit
• Fine line of distinction between audit of
II policy and audit of policy implementation
• Clash of perspectives and interpretations
III between audit and management
• Charges of data manipulation and
IV interpretation by auditors
• Charges of auditors turning blind eye to
V management replies

62. The Performance Auditor’s Profile
Who you ARE:
√
Independent evaluator
Economist/Statistician
Public policy analyst
Accountant/Auditor
Aid to administration Who you are NOT:
Legislative policymaker
X
Implementing executive
Judicial officer
RTI activist/Journalist
Anti-corruption crusader