Webinar Week | José Aznarte

1. AVOIDING INVASIVE SURVEILLANCE, ENSURING TRUST: UNED’S
VIRTUAL EXAMINATION HALL (AVEX)
José L. Aznarte
14/06/2022

2. OUTLINE
About me
About my institution: UNED
Ethical framework for the use of data-based technologies
Assessment under COVID19
1/23

3. ABOUT ME

4. TEACHING AND RESEARCH
• I’m ”profesor titular” at the Artificial Intelligence Department of UNED.
• I teach Data Mining and Machine Learning in master’s level.
• I also supervise PhD students and master’s theses.
• My main research line:
• Time series forecasting with deep learning & statistics
• Applied to common/shared problems: air quality, traffic intensity, epidemic
propagation…
• I coordinate some R&D projects:
• SOCAIRE (Municipality of Madrid)
• PreCoV2.org (Ministry of Health)
• Chair EMT/UNED for Air Quality and Sustainable Mobility
2/23

5. SERVICE AT RECTORATE
• Appointed in 2019 as deputy vicerector in charge of digitalisation and
innovation.
• Co-leading the ED3 institutional project (”Digital, Distant and Data-powered
Education”).
• Interface with IT department.
3/23

6. SERVICE AT RECTORATE
• Appointed in 2019 as deputy vicerector in charge of digitalisation and
innovation.
• Co-leading the ED3 institutional project (”Digital, Distant and Data-powered
Education”).
• Interface with IT department.
Actually…
3/23

7. SERVICE AT RECTORATE
• Appointed in 2019 as deputy vicerector in charge of digitalisation and
innovation.
• Co-leading the ED3 institutional project (”Digital, Distant and Data-powered
Education”).
• Interface with IT department.
Actually…
Many doubts about the use of AI tools for dealing with people’s data.
3/23

8. ABOUT MY INSTITUTION: UNED

9. GENERAL INFORMATION
”National Distance Learning University” – UNED
• One of the biggest universities in Europe (>150.000 students)
• Founded 50 years ago (no digitalization)
• Mixed remote and face-to-face model:
• Network of regional centers (>50)
• Remote teaching during the course
• F2F exams
• Wide range of different student situations. Mostly not ”freshers” coming from
high-school.
4/23

10. ED3: DISTANT, DIGITAL AND DATA-POWERED EDUCATION (2019)
5/23

11. ED3: DISTANT, DIGITAL AND DATA-POWERED EDUCATION (2019)
«Develop a framework for evidence-based interventions to improve
teaching/learning processes through response-able exploitation of data.
• Policies which guarantee that use of data takes into account potential social
and ethical consequences.
• Identify, gather, curate and make accesible all the data sources related with
teaching/learning processes, with different access rights for each profile.
• Analize data and prepare them for knowledge creation through exploratory
analysis and operational models.
• Promote interventions over teaching/learning processes based on evidences
resulting from data and the knowledge of the different stakeholders».
5/23

12. ETHICAL FRAMEWORK FOR THE USE OF
DATA-BASED TECHNOLOGIES

13. PARTICIPATORY PROCESS
• First things first: we needed a shared ethical common sense, which goes
beyond regulation, about what happens with data and how can we extract
value from it.
• We carried out a participatory process, open to the entire UNED community,
to agree upon a series of cautions that should be taken into account when
using data-based technologies.
6/23

14. PARTICIPATORY PROCESS
• First things first: we needed a shared ethical common sense, which goes
beyond regulation, about what happens with data and how can we extract
value from it.
• We carried out a participatory process, open to the entire UNED community,
to agree upon a series of cautions that should be taken into account when
using data-based technologies.
• Open during 2 months in 2020, >2.500 people participated.
• Used the ”Decidim” participatory software.
6/23

15. PARTICIPATORY PROCESS
• First things first: we needed a shared ethical common sense, which goes
beyond regulation, about what happens with data and how can we extract
value from it.
• We carried out a participatory process, open to the entire UNED community,
to agree upon a series of cautions that should be taken into account when
using data-based technologies.
• Open during 2 months in 2020, >2.500 people participated.
• Used the ”Decidim” participatory software.
• 9 basic cautions were proposed by the rectorate.
• During the process, these were rearranged in terms of the community’s
perception of their importance and new 4 with high support were added to
the document. 6/23

16. OUTCOME
An official document encoding a set of 13 cautions for the use of data-based
technologies, related with:
• care
• response-ability
• transparency
• consent
• property & control
• validity and trust
• participation
• privacy
• preventing potential adverse
impacts
• effective communication
• adaptability
• right to explanations
7/23

17. Then…
8/23

18. Then… COVID19! :(
8/23

19. ASSESSMENT UNDER COVID19

20. PRE-COVID19 EXAMINATION PROCEDURE
9/23

21. PRE-COVID19 EXAMINATION PROCEDURE
• Students choose one of two alternative dates and go to a regional center to
undertake an exam.
• Exams take place synchronously in halls and are invigilated by ad-hoc teams
of faculty staff.
• Students might be required to leave bagpacks, purses and other material
before entering the hall.
• Exams are scanned upon hand-over, gathered in a centralized computer
system and distributed to each course’s team.
9/23

22. AVEX: VIRTUAL EXAMINATION HALL
We needed a tool to reproduce the conditions of usual examination under
lockdown:
• Synchronous questionnaire distribution and response management.
• Variety of exam types (test, essay, math…).
• Proper identification of students.
• Anti-fraud measures.
10/23

23. AVEX: VIRTUAL EXAMINATION HALL
We needed a tool to reproduce the conditions of usual examination under
lockdown:
• Synchronous questionnaire distribution and response management.
• Variety of exam types (test, essay, math…).
• Proper identification of students.
• Anti-fraud measures.
These measures must be non-invasive (at least no more than exam halls).
10/23

24. AVEX: VIRTUAL EXAMINATION HALL
We needed a tool to reproduce the conditions of usual examination under
lockdown:
• Synchronous questionnaire distribution and response management.
• Variety of exam types (test, essay, math…).
• Proper identification of students.
• Anti-fraud measures.
These measures must be non-invasive (at least no more than exam halls).
Q: Can we delegate exam invigilation to an AI?
10/23

25. FACIAL RECOGNITION TECHNOLOGIES (FRT) IN EDUCATION
• First step: study the risks of facial recognition technologies for exam
invigilation.
• Facial recognition systems applied to surveillance are expanding.
• There are mounting evidences that these technologies can be problematic:
technical, legal and ethical difficulties.
11/23

26. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
12/23

27. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
12/23

28. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
1. FRT can violate privacity rights.
12/23

29. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
1. FRT can violate privacity rights.
1. FRT are naturally imprecise, and its software is fallible.
12/23

30. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
1. FRT can violate privacity rights.
1. FRT are naturally imprecise, and its software is fallible.
1. FRT can produce automatization bias.
12/23

31. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
1. FRT can violate privacity rights.
1. FRT are naturally imprecise, and its software is fallible.
1. FRT can produce automatization bias.
1. FRT can produce discriminations and violations of the equality principle.
12/23

32. RISKS OF FACIAL RECOGNITION TECHNOLOGIES
1. There is no clear legal framework for invasive surveillance technologies.
1. The use of FRT might imply a violation of the legal principles of necessity and
proportionality.
1. FRT can violate privacity rights.
1. FRT are naturally imprecise, and its software is fallible.
1. FRT can produce automatization bias.
1. FRT can produce discriminations and violations of the equality principle.
1. FRT can generate discriminations based on different functional abilities.
12/23

33. FACIAL RECOGNITION TECHNOLOGIES IN EDUCATION
13/23

34. AVEX: VIRTUAL EXAMINATION HALL
14/23

35. AVEX: VIRTUAL EXAMINATION HALL
• We underwent express development of our own technological solution: AvEx.
• In less than 6 weeks, first version operational.
• In first exam call, over 400.000 exams with few issues (~1%).
14/23

36. SECURITY MEASURES
Layered security scheme:
• Wide question banks
• Reasoning-based questions
• Open-book exams
• Tight timing per question
15/23

37. SECURITY MEASURES
Layered security scheme:
• Wide question banks
• Reasoning-based questions
• Open-book exams
• Tight timing per question
• Randomization of q&a
• Sequential access
• Copy/paste disabled
• Random photographs
15/23

38. SECURITY MEASURES
Layered security scheme:
• Wide question banks
• Reasoning-based questions
• Open-book exams
• Tight timing per question
• Randomization of q&a
• Sequential access
• Copy/paste disabled
• Random photographs
• Sworn affidavit
• Plagiarism: text similarities
• Phone interview after the exam
15/23

39. SECURITY MEASURES
Layered security scheme:
• Wide question banks
• Reasoning-based questions
• Open-book exams
• Tight timing per question
• Randomization of q&a
• Sequential access
• Copy/paste disabled
• Random photographs
• Sworn affidavit
• Plagiarism: text similarities
• Phone interview after the exam
15/23

40. RESULTS (I)
First analysis (after 2 exam calls):
16/23

41. RESULTS (I)
First analysis (after 2 exam calls):
• Amount of exams: ↑~20% (of courses)
• Pass marks: ↑~40%
• Average marks: ↑~50%
• Remote sessions attendance: ↑~300%
• Disciplinary issues: ↑
16/23

42. RESULTS (II)
17/23

43. RESULTS (III)
18/23

44. RESULTS (IV)
19/23

45. DISCUSSION
What do these results mean? Is the system less secure?
20/23

46. DISCUSSION
What do these results mean? Is the system less secure?
Hypotheses:
• More time to study in lockdown
• Changes in the assessment design and criteria
• More continuous evaluation
• Easier to attend exams
• Fraud?
20/23

47. STUDENT SATISFACTION
• 86% of students declares negative impacts of lockdown in study.
• 40% link it to mood status
• Students declare more difficulties to study:
• 54% due to work-related issues
• 35% due to illness
• 20% less time
21/23

48. STUDENT SATISFACTION
• 86% of students declares negative impacts of lockdown in study.
• 40% link it to mood status
• Students declare more difficulties to study:
• 54% due to work-related issues
• 35% due to illness
• 20% less time
• Elements hitting on performance:
21/23

49. STUDENT SATISFACTION
• Academic results:
• 50% similar, 25% better or much better
• 70% identify some positive element (more time, more motivation…)
22/23

50. STUDENT SATISFACTION
• Academic results:
• 50% similar, 25% better or much better
• 70% identify some positive element (more time, more motivation…)
• 67% is happy or very happy with AvEx
• Most students appreciate the efforts of UNED
• Young people have worse oppinion
22/23

51. STUDENT SATISFACTION
Privacy protection is enabled under AvEx:
23/23

52. THANKS!
jlaznarte@dia.uned.es 23/23