This document provides an introduction to building self-adaptive software through an affective-driven software factory approach. It discusses (1) how affective states can be sensed using various modalities like facial expressions and physiological sensors, (2) how software can be made self-adaptive by incorporating machine learning to change behavior based on sensed affective states, and (3) how a software factory framework that utilizes design patterns, tools, and reusable components can be used to develop such affective-driven, self-adaptive software.

1. An Introduction to Building
Affective-Driven Self-Adaptive Software
through a Software Factory Approach
Javier Gonzalez-Sanchez
javiergs@asu.edu
www.javiergs.com

2. Context
Human Computer Interaction
Affective Computing
Software Architecture
System Engineering

3. 1
Affective-Driven

4. Affective-Driven
(B) Emotions are generally understood as representing a
synthesis of a subjective experience, an expressive
behavior, and a neurochemical activity.
(A) Feeling
Emotion
Mood
Affective State
Affect
(C) facilitation of social communication

5. Affective-Driven
Empathy
“to put one's self in another's shoes”

6. Affective-Driven
1
Health Care
Education
Entertainment

7. Affective-Driven
Empathy
SENSING PERCEPTION BELIEFS

8. Affective-Driven

9. Face-Based
Face-based emotion recognition systems
These systems infer affective states by capturing images of the users’ facial
expressions and head movements.
We are going to show the capabilities of face-based emotion recognition systems
using a simple 30 fps USB webcam and software from MIT Media Lab [8].
[8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on
Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.

10. Brain-Based

11. Text-Based

12. Eye Tracking

13. Eye Tracking

14. Other Sensors…
Galvanic Skin Conductance
Posture Sensor
Nike+, Bodymedia, etc.

15. Other Sensors…
S. Mota, and R. W. Picard, "Automated Posture Analysis for Detecting Learners Interest Level," Proc. Computer Vision
and Pattern Recognition Workshop (CVPRW 03), IEEE Press, June 2003, vol. 5, pp. 49, doi:10.1109/CVPRW.2003.10047.
Y. Qi, and R. W. Picard, "Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure Pattern
Classification," Proc. International Conference on Pattern Recognition (ICPR 02), Aug. 2002, vol 3, pp. 30448, doi:
10.1109/ICPR.2002.1047973.
M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance
Sensor,” Proc. First International Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-
Verlang, Oct. 2005, pp. 699-706, doi:10.1007/11573548_90.

16. Machine Learning
I. Arroyo, D. G. Cooper, W. Burleson, F. P. Woolf, K. Muldner,
and R. Christopherson, “Emotion Sensors Go to School,” Proc.
Artificial Intelligence in Education; IOS Press, July 2009, vol.
Frontiers in Artificial Intelligence and Applications 200, pp.
17-24.

17. 2
Self-Adaptive

18. Self-Adaptive
Adaptive
“change behavior“

19. Self-Adaptive
LEARNING
HAVING OPTIONS
(internal software structure)
Adaptive
DYNAMIC - REAL-TIME
DECISION THEORY

20. Close-Loop
1

21. Close-Loop
Framework (i.e Components, Tools)

22. Close-Loop

23. 3
Software Factories

24. Software
1

25. Software
1

26. Software
software ARCHITECTURE
ENGINEERING
“reducing complexity through abstraction
and separation of concerns”

27. Not One… But Many

28. Not One… But Many
Process, Rules, and
Regulations
software FACTORY
Manufacturing using TOOLS…
industrial production… COMPONENTS
are transformed or assembled…

29. Not One… But Many
IMPLEMENTATION:
VALIDATION:
Frameworks
ATAM Tools
Empirical Software Methods
software ARCHITECTURE
MODELING:
Software Design Patterns
Pattern Languages
Model-Driven Design
Component-Based Engineering

30. ABE Framework

31. A Pattern Language
Design
Patterns

32. A Pattern Language
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R. and Burleson, W. (2011) Affective Computing Meets Design
Patterns: A Pattern-Based Model of A Multimodal Emotion Recognition Framework. Proceedings of the16th European
Conference on Pattern Languages of Programs. Irsee, Germany. July 2011.
Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Atkinson, R. and Burleson, W. (2012) Towards a Pattern Language for
Affective Systems. Proceedings of the19th Conference on Pattern Languages of Programs. Tucson, Arizona, USA. October
2012. In press.

33. Close-Loop
Framework (i.e Components, Tools)

34. Results
MODEL (GUIDELINES), FRAMEWORK (COMPONENTS), TOOLS

35. 4
Study Cases

36. Videogames
Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., Chavez-Echeagaray, M.E., Christopherson, C., Atkinson, R., (2012), "Lost in the Dark:
Emotion Adaption", ACM Symposium on User Interface Software and Technology 2012. Cambridge, MA USA. October 2012.

37. Visualization

38. Visualization

39. Visualization

40. Visualization

41. Questions

42. Contact
Javier Gonzalez-Sanchez
javiergs @ asu.edu
www . javiergs . com

43. Acknowledgements
This research was supported by
Office of Naval Research under Grant N00014-10-1-0143
awarded to Dr. Robert Atkinson,
and by
National Science Foundation, Award 0705554, IIS/HCC
Affective Learning Companions: Modeling and Supporting Emotion During
Teaching awarded to Dr. Beverly Woolf and Dr. Winslow Burleson.