Artificial Intelligence webinar week, day 1: Human computer learning interaction in a virtual laboratory by Vasilis Zafeiropoulos (HOU)

1. Human-Computer
Learning Interaction in
a Virtual Laboratory
Vasilis Zafeiropoulos
PhD Candidate
HELLENIC OPEN UNIVERSITY
School of Science and Technology

2. Learning Practices Pyramid2

3. Οn-site laboratory education
- no plain sailing
 Large number of trainees
 Short time for training
 Lab equipment not really completely available
(sensitive, wear and tear, consumables)
 Risks of accidents (health and maintenance)
3

4. About Onlabs
 Simulation of a real Biology Lab
 Main characteristics:
 3D
 Realism
 Interactivity
 Genre: Modern Adventure Games
 Development under:
 Hive3D by Eyelead Software (2012-2015)
 Unity (2016-now)
4

5. Onlabs Screenshot5

6. Purposes
Decongestion of real labs
Familiarizing oneself with equipment - safely
Protection of lab equipment from damages
Capability of infinite trial & error testing
(more) Pleasant training process
(more) Effective learning
Virtual lab education is to complement on-
site lab education, not substitute it!
6

7. Experiments simulated in Onlabs (1)
 Α. Use of Optical Microscope
1. Configuration of the microscope (e.g. adjusting light, iris, lenses)
2. Creation of a test specimen (a slide with a piece of paper and water
on it)
3. Microscoping with all objectives lenses
7

8. Experiments simulated in Onlabs (2)
 Electrophoresis
1. Use of the electric scale for scaling the various powders
2. Use of electronic pipette for drawing and pouring precise quantities of fluids
3. Use of magnetic stirrer for fully dissolving and stirring the components of a dilution
4. Use of automatic pipette for extracting DNA from PRC tubes
5. Use of microwave oven for dissolving agarose powder in electrophoresis buffer
6. Use electrophoresis device for separating nucleic acid samples by size
7. Use UV viewer for visualizing nucleic acid bands
8

9. Operation Modes
 Experimentation Mode
The human user makes free use of all the lab equipment
 Instruction Mode
The computer guides the human user to complete an experiment
 Evaluation Mode
The computer evaluates the performance of a human user on conducting
an experiment
 Computer Training Mode
A human expert (expert ≠ user) teaches the computer with Machine
Learning to:
 rate properly (Rater Training Sub-Mode), i.e. provide an accurate score for
the user’s performance)
 play properly (Bot Training Sub-Mode), i.e. conduct an experiment

10. Instruction Mode

11. Evaluation Mode

12. Scoring Mechanism (Evaluation Mode)
 Consists of two parts:
 Success Rate [0-100%] (how “close to” or “far from” the user is from the
experiment’s final state)
 For the i-th action performed, a score xi ∈ [0,1] is assigned, e.g.:
― if the microscope is connected to the socket (1st action), score x1 = 1; otherwise x1 = 0.
― if the microscope switch is turned on (2nd action), score x2 = 1; otherwise x2 = 0.
― if the microscope light intensity knob is set to 18 (3rd action), score x3 = 1; for any other
case x3 lies within (0,1)
 Each action has different significance, so we intuitively define a particular
weight wi for each one of them
 Success rate is the weighted average of the various scores xi:
𝑠𝑢𝑐𝑐𝑒𝑠𝑠 𝑟𝑎𝑡𝑒 ←
σ 𝑖=1
𝑛
𝑤 𝑖∙𝑥 𝑖
σ 𝑖=1
𝑛 𝑤 𝑖
∙ 100
 From weights wi, we create a weight vector 𝑤 = 𝑤1, 𝑤2, … , 𝑤 𝑛
𝑇
(useful for
Machine Learning later…)
 Penalty points (they are received whenever the user performs actions in
wrong order)
 The Aggregate Score [0-100%] is calculated combining the Success
Rate and the Penalty Points

13. Computer Training Mode (1)
 Rater Training Sub-Mode
The student plays various sessions
The human expert evaluates each session
The computer scoring mechanism is adjusted
according to human expert’s feedback
Machine learning techniques used:
 Genetic Algorithm
 Artificial Neural Network

14. Rater Training Sub-Mode

15. Computer Training Mode (2)
 Bot Training Sub-Mode
The computer plays a session by itself
The human expert evaluates each session
The computer learns how to play correctly
Machine Learning technique used:
 Reinforcement Learning
(Under development…)

16. Bot Training Sub-Mode

17. Genetic Algorithm
(Rater Training Mode)
 A GA simulates biological evolution
 Our GA is interactive (a human supervisor contributes to the learning process)
 In our GA, weight vectors are the chromosomes
 Our first generation consists of 30 randomly produced weight vectors.
 Weight vectors, like chromosomes, compete against each other with respect to each
one’s fitness (expressed by a fitness function)
 For each play session, the score produced by a weighted vector and the score
provided by the human expert are compared by our fitness function
 According to weight vectors’ fitness:
 A fixed percentage of them are selected (directly copied) to the new generation
 A fixed percentage of them are chosen for crossover (reproduction) with each other
and their offspring are put into the new generation
 A fixed percentage of the chromosomes in the produced generation are mutated
 The GA stops after a termination condition is satisfied; our GA’s termination condition is
50 generations
 The fittest weight vector of the final generation is the training result with our GA

18. Artificial Neural Network
(Rater Training Mode)
 An ANN simulates neural networks in the brain
 Our ANN consists of 3 layers of neurons
 To each neuron of an ANN come weights from the neurons of the
previous layer
 Our ANN has different weights and provides a different scoring
mechanism from the ones in Evaluation Mode!
 Our ANN’s initial weights wj→i are randomly produced
 For each play session, our ANN produces a single value (score) as output
through its rightmost neuron
 For each play session, the human expert provides their own score, too
 The error between the ANN’s score and the human expert’s score is
back-propagated through the ANN and its weights are re-configured
 An ANN is re-trained several times (epochs); our ANN is being trained for
1000 epochs
 The training result is the ANN’s final weights

19. Training and Testing
 Train on set A, test on set B
 Train on all sets, test on all sets (biased)
 Train on Experts E1, E2, E3, test on E4 (cross-validation among experts)
 Train on Expert E1, test on E1 (cross-validation within the same expert)
 Train on Classifications C1, C2, test on C3 (cross-validation among
performance classifications*)
 Train on Groups G1, G2, G3, G4, test on G5 (cross-validation among
various groups of training sets**)
 Calculation of Mean Squared Error (MSE) on each testing set
 Comparison of GA’ fittest weight vector to intuitive weight vector
of evaluation mode
* Classifications (of user’s performance): Low, Medium, High
** Groups of 12 training sets (one classification from each expert in each
group)

20. Genetic Algorithm
Training Results

21. Artificial Neural Network
Training Results (1)

22. Artificial Neural Network
Training Results (2)
 Mean Squared Error to Epochs graph (train on all,
test on all):

23. Testing & Evaluation by Students
 Focus group: biology-oriented students with minimum or zero previous
knowledge in science topics
 Written conceptual pre-tests and post-tests
 Questionnaires to express opinion of satisfaction, motivation,
engagement, etc.
 Practice examinations in the wet lab
 The Virtual Lab Group (educated with Onlabs) and the Control Group
(educated without Onlabs) conducted a 22 steps microscopy experiment
 Results:
▪ a. I completed this step easily
▪ b. I completed this step on
difficulty
▪ c. I couldn’t complete this step
by myself – I asked for help

24. Future Work
 Simulation of the rest of lab instruments and biology experiments
 Simulation of instruments and experiments at:
 Chemistry lab
 Physics lab
 Any other place with instruments and procedures
 Insertion of more than one bots in the virtual lab
 Let them interact, co-operate and compete with the human user and
each other
 Evaluate learning outcomes in terms of:
 speed
 accuracy
 use of resources
etc.
24

25. The grand goal
 Reduce the cost of laboratory education
 Elevate lab education at-a-distance to at least the
same level of learning effectiveness with on-site lab
education
25

26. Development Team
 Vasilis Zafeiropoulos
 Dimitris Kalles
 Argyro Sgourou
 Achilles Kameas
 Evgenia Paxinou
 Kostas Mitropoulos
http://onlabs.eap.gr/
Thank you!