Ofqual's approach to validity audits - December 2014

1. Preparing for validity audits

2. OUR APPROACH
Validity audits - the bigger picture

3. Validity at the core of our approach
Our General Conditions of
Recognition already make validity a
key requirement:
(D1) Qualifications must be fit for
purpose and valid
(E1) Qualifications must have a
clear objective and user
support
(D3) Awarding organisations must
review and enhance their
approach on the basis of user
feedback

4. Validity audits
Our validity audits will be scheduled both by random selection and by
the level of risk associated with:
• the awarding organisation
• the qualification type
• specific points in the lifecycle of the qualification
Our priorities will continue to be those qualifications:
• which are publicly funded
• taken by large numbers of students in full time education in schools
and colleges
• commonly used in apprenticeships
However whilst risk is a driver – every awarding organisation should
expect to be visited tomorrow, and be able to demonstrate how standards
are being set and maintained – leading to valid qualifications

5. Validity audits
Do you have a viable delivery model, for each stage in the lifecycle?
adequate processes, systems and resources in place
Do you deliver successfully, each assessment session?
procedural adherence
reliability / accuracy of outcomes
Our initial focus
has been on
Design &
Development
stages –
although we are
extending to
other parts of the
lifecycle

6. BEFORE AND DURING…
Validity audits - what we’ll ask for

7. How we do the validity audits
• You will be provided with a specification approximately one
month before the audit that sets out the focus of the audits, our
‘key lines of enquiry’ and the evidence we will be looking for
• Validity audits consist of a series of semi-structured interviews
with groups of AO personnel guided by the lines of enquiry and a
review of relevant documentation and evidence, including:
• qualification development files, plans, business cases, risk logs,
meeting minutes, emails, technical reports, draft and final
assessment materials, draft and final specifications and test
specifications
• In most cases audits take 1-2 days although some may take
longer

8. Key lines of enquiry
• Typically the validity audits will follow a series of ‘key lines of
enquiry’
• The ‘design & development’ audits used the following ‘key lines
of enquiry’:
1. Appropriateness of processes for setting the objective and establishing
support
2. Appropriateness of processes for defining the knowledge, skills and
understanding required to achieve the qualification
3. Effectiveness of processes for developing the assessment framework to
ensure that it is fit for purpose and can be delivered
4. Effectiveness of processes to ensure that future individual assessments
are fit for purpose
5. Appropriateness of the processes used to set and maintain standards
6. Effectiveness of planning for qualification delivery
7. Effectiveness of processes to monitor whether the qualification is
achieving its objectives

9. AFTERWARDS…
Validity audits

10. Key lines of enquiry
• We will gather evidence but we will not provide feedback on the
day
• We will review evidence and compare against other information
that we have – standardisation and moderation
• We will use the evidence to inform our overall approach to risk
• We may scrutinise qualifications to test our views on the audit
outcomes
• Where we see potential issues – we will explain what we believe
our evidence tells us and allow you to make representations
• We may follow up this work:
• Agree an approach to address areas of concern
• Further evidence gathering
• No action – outputs fed to our risk model
• Action

11. TO CONSIDER NOW…
Validity audits

12. Questions to ask…
• Are all your qualifications that are currently on RITS valid, ‘fit for
purpose’, and able to stand up to scrutiny?
• Is the purpose of each of your qualifications clear?
• Are lifecycle risks being identified and managed properly?
• Is your engagement with Users strong enough?
• Could you strengthen your feedback loops?
• Could your organisation’s knowledge of validity be improved?
• Have you thought through how you document your approach to
validity – e.g. checking that other requirements are not
undermining validity?