Presentation at EdTech 2023 Conference, Irish Learning Technology Association. 2nd June 2023.

1. Photo by Rae Wallis on Unsplash
Mark Brown
Eamon Costello & Prajakta (Lily) Girme
2nd June 2023
Quality is like a Box of Chocolates:
Developing National QA Guidelines for Digital Education

2. “Quality is an elusive term
for which there is a wide
variety of interpretations
depending upon the views of
different stakeholders.”
(Schindler, et al., 2015, p. 4)
Photo by Rae Wallis on Unsplash
Quality is like a Box of Chocolates:
Developing National QA Guidelines for Digital Education

3. Insights from OECD study
Development of National QA Guidelines
Lessons for European Standards & Guidelines
Outline

4. 2004
2005
Before we start…
2011 2016

5. “The overarching paradox is
that online and distance
education systems with their
digital content and the persistent
record of online transactions
provide a rich source of
evidence to enable quality
assurance and audit processes.
https://www.pedocs.de/volltexte/2015/10879/pdf/Ossiannilsson_et_al_2015_Qualitymodels.pdf
Before we start…

6. 1. Insights from OECD study
2022
https://www.oecd-ilibrary.org/education/digital-higher-education_f622f257-en

7. QA / QE
Gap
Digital
Education
- Macro-Level
- Meso-Level
- Micro-Level
- Nano-Level
 Multifaced and
multifunctional
 Highly contextualised
 Institutions are central
 Owned & distributed
across institutions
 Conversational,
shares experiences
and involves feedback
loops
 Dynamic as part of a
living and thriving
quality culture
1. Insights from OECD study

8. 1
In 23 OECD jurisdictions, we found no national framework, standards or guidelines for digital higher education,
and no evidence of a decision taken on how to approach the quality assurance of digital higher education.
2
In 8 OECD jurisdictions, we found common standards for digital and traditional study modes, with evidence of a
decision to apply the standards for the quality assurance of on-campus instruction to digital higher education.
3
In 12 OECD jurisdictions, we identified specific standards or guidelines for digital higher education. These either
cover all types of digital education or a specific type (or types) of digital education (e.g., hybrid education).
No approach for the quality assurance of digital higher education
Common standards and guidance for the quality assurance of digital higher education
Specific standards for the quality assurance of digital higher education
1. Insights from OECD study

9. 2018
https://www.qqi.ie/sites/default/files/media/file-uploads/Statutory%20QA%20Guidelines%20for%20Blended%20Learning%20Programmes.pdf
1. Insights from OECD study

10. 1. Insights from OECD study

11. Most
Common
Least
Common
Aggregated summary…
1. Insights from OECD study

12. • An abundance of QA frameworks
• Many common dimensions shared across
the different QA frameworks
• Absence of output measures of QA
• Emergence of new QA considerations
• More research needed on how institutions
implement QA/QE frameworks
Findings…
https://www.tandfonline.com/doi/abs/10.1080/08923647.2018.1417658
1. Insights from OECD study

13. What else is missing
from current QA frameworks?
(Brown, 2022)
1. Insights from OECD study

14. 2. Development of National QA Guidelines
2018 2023

15. What in a single word is the most
important quality consideration
that you would like to see in the
new QQI guidelines?
2. Development of National QA Guidelines

16. Adapted
from
OECD
2. Development of National QA Guidelines

17. 2. Development of National QA Guidelines

18. 2. Development of National QA Guidelines

19. 2. Development of National QA Guidelines

20. 2. Development of National QA Guidelines

21. 2. Development of National QA Guidelines

22. 2. Development of National QA Guidelines

23. 2. Development of National QA Guidelines

24. https://www.qqi.ie/what-we-do/engagement-insights-and-knowledge-sharing/current-consultations
2. Development of National QA Guidelines

25. 2. Development of National QA Guidelines

26. 3. Lessons for European Standards and Guidelines
2018

27. Policy for Quality
Assurance
• Policies for quality
assurance (ESG
1.1)
Teaching, Learning
and Assessment
Learning Resources
and Infrastructure
• Design and approval
of programmes
(ESG 1.2)
• Student-centred
learning, teaching
and assessment
(ESG 1.3)
• Learning resources
and student support
(ESG 1.6)
• Student-centred
learning, teaching
and assessment
(ESG 1.3)
• Student admission,
progression,
recognition and
certification (ESG
1.4)
• Learning resources
and student
support (ESG 1.6)
Support for
Instructors and
Students
Feedback, Review
and Performance
Monitoring
• Programme design,
approval,
monitoring and
review (ESG 1.2
and 1.9)
• Information
management (ESG
1.7)
• Public information
(ESG 1.8)
• Cyclical external
quality assurance
(ESG 1.10)
• Student-centred
learning, teaching
and assessment
(ESG 1.3)
• Student admission,
progression,
recognition and
certification (ESG
1.4)
• Teaching staff
(ESG 1.5)
• Learning resources
and student
support (ESG 1.6)
Do we need to update ESGs?
3. Lessons for European Standards & Guidelines

28. Adapted from
EADTU 2020
3. Lessons for European Standards & Guidelines

29. > What models or approaches to
learning design inform the
development, delivery and
evaluation of programmes?
> To what extent are students
engaged in active learning through
their use of digital tools and
resources?
> How does digital innovation support
assessment for learning and student
feedback?
> How is teaching, learning and
assessment informed by best
practice in digital higher education?
INDICATOR EVIDENCE
> An explicit model of learning design
is adopted for programme
development.
> Student evaluation data and results
from national surveys of learner
engagement.
> Mapping of programme assessment
and range and variety of digital tools
> Programme development plans
implement key lessons from
contemporary theory, research and
practice
TYPE
Ways of Measuring and Reporting
> Input | Resource
> Qualitative
> Output
> Quantitative
> Process
> Qualitative
> Process
> Qualitative
3. Lessons for European Standards & Guidelines

30. Adapted from
ENQA 2018
Horizon
Indicator
3. Lessons for European Standards and Guidelines

31. > The institution’s Virtual Learning
Environment (VLE) is regularly
updated and supports a variety of
digital tools and learning resources.
> Students can access electronic
library resources and digital
textbooks from wherever they
choose to study.
> Digital media and Open Educational
Resources (OER) are embedded in
the curriculum to enhance the
student learning experience.
INDICATOR EVIDENCE
> Explicit VLE development plan and
learning analytics data on
instructors’ and students’ use of
digital tools.
> Student evaluation data and results
from national surveys of learner
engagement.
> Mapping the integration of digital
media and resources across the
curriculum and learning data
analytics on instructors’ and
students’ level of use.
TYPE
Ways of Measuring and Reporting
> Process | Output
> Qualitative | Quantitative
> Process |Output
> Qualitative | Quantitative
> Process |Output
> Qualitative | Quantitative
3. Lessons for European Standards and Guidelines

32. Crucial
Importance of
training and
professional
development
Adapted from
ENQA 2018
Adapted from
EADTU 2020
3. Lessons for European Standards and Guidelines

33. > Do staff engaged in teaching have
the appropriate qualifications,
knowledge and skills required to
promote digital innovation and
study flexibility?
> What training and professional
development activities are available
to new instructors and existing staff
to harness the potential of digital
innovation and the provision of
study flexibility?
> What expert professional support
staff and internal service units are
available for digitally enhanced
course design, pedagogy and
assessment?
INDICATOR EVIDENCE
> Staff profile, skills and competence
assessment for digital innovation
and the provision of study flexibility.
> Inventory of staff training
opportunities and participation data
on number of instructors engaging in
training and continuing professional
development
> Number of support staff and
description of both the structure and
type service units available to
promote digital innovation and the
design of study flexibility
TYPE
Ways of Measuring and Reporting
> Input | Process
> Qualitative | Quantitative
> Input | Output
> Quantitative
> Input | Resource
> Qualitative | Quantitative
3. Lessons for European Standards and Guidelines

34. Final comment…
Chocolate is always
the answer no matter what
the question!
Quality remains elusive but…

35. Staring, F., Brown, M., Bacsich, P., & Ifenthaler, D. (2022). Digital higher
education: Emerging quality standards, practices and supports, OECD Education
Working Papers, No. 281, OECD Publishing, Paris,
https://doi.org/10.1787/f622f257-en.
Volungevičienė, A., Brown, M., Greenspon, R., Gaebel, M., & Morrisroe, A.
(2021). Developing a High-Performance Digital Education System: Institutional
Self-Assessment Instruments. European University Association, Brussels.
Ossiannilsson, E., K. Williams, A. Camilleri and M. Brown (2015), Quality models
in online and open education around the globe: State of the art and
recommendations. International Council for Open and Distance Education, Oslo,
Norway. https://files.eric.ed.gov/fulltext/ED557055.pdf
Esfijani, A. (2018), “Measuring quality in online education: A metasynthesis”,
American Journal of Distance Education, 32/1, pp. 57-73, DOI:
10.1080/08923647.2018.1417658
Brown, M. (2022). The quest for quality in digital higher education: A critical
analysis of QA frameworks. Full paper at EDEN Digital Learning Europe Annual
Conference, Tallinn, Estonia, 20th June.
Key references
2022
https://link.springer.com/referenceworkentry/10.1007/978-981-19-0351-9_41-1

36. https://eden-europe.eu/event/eden-2023-annual-conference-in-dublin/
#EDEN2023

37. Go raibh maith
agaibh!
Thank you