OPENSESAME: Unlocking Smartphone Through Handwaving Biometrics
David Weaver - UCI Internship Powerpoint
1. UCI Internship: Creating a
Positive Feedback Device for the
Rehabilitation of Stroke Patients
Student: David Weaver
Professor: David Reinkensmeyer
2. Background Information
• Professor Reinkensmeyer is a world class leader
in the field of Robotic Rehabilitation and is
currently working on several interesting and
important projects
• In the past and currently, rehabilitation methods
have involved a therapist giving a patient step
by step instructions on how to perform a certain
action.
• However, Professor Reinkensmeyer believes that
methods of rehabilitation that allow patients to
explore during rehabilitation instead of just
being told what to do would be highly
beneficial.
• This is territory he is currently exploring.
3. So What I Was Tasked With
I was told to create a device that would use sound as a positive
feedback mechanism to help stroke patients relearn a movement.
• Good sounds would correlate with a correct movement
• Bad sounds would correlate with an incorrect movement
4. The Device
Based on these guidelines, I came up with the following device:
• It would have two buttons:
• When the therapist hit their respective button and moved the device, their
motion would be recorded. The patient would then try and copy the
motion made by the therapist while holding down their own button.
• The comparison would happen by breaking each of the recorded motions
into 4 sections (beginning, middle1, middle2, end), each of which had a
corresponding tone.
• Tones would then play according to the correctness of the patient’s motion
when their four sections were compared to the therapist’s four sections.
5. So Making This a Reality…
(when you’re an unskilled
high school student)
6. Creating the Device: The Tones
I created the tones for the
device using music
production software.
Each of the four segments
(Beginning, Middle1,
Middle2, and End) would
have a correct tone, and
then 3 other tones which
got progressively more and
more incorrect.
7. Creating the Device: 3D Printing
At first I was planning for the device to be a sphere.
Then I wanted it to be a sphere without curves
8. Creating the Device: 3D Printing
Finally I arrived at the Cylindrical Design because:
• It’s easy to print
• It can be grasped in several different ways
9. Creating the Device: Coding
• The most difficult part of the internship for me was learning to code.
• However, after some tutoring and staring at the screen long enough,
it clicked.
• By the end of the internship I had written a very long segment of code
that ran my entire program.
• However, the microprocessor I was using (Arduino) did not have
enough RAM to run my entire program and so I had to splice the code
into two different sections, each run by a separate board.
• This ultimately looked like this…
//This codeis synced to theboard that has theaccelerometer attached
//Written by David Weaver
//library that allows transmissionto theother board
#include<SoftwareSerial.h>
//defined values that correlateto sound files
#define START_SYMBOL 'Z'
#define STANDARD_TONE 'a'
#define COLON_START ':'
#define S_B_C 'b'
#define S_B_W1 'c'
#define S_B_W2 'd'
#define S_B_W3 'e'
#define S_M1_C 'f'
#define S_M1_W1 'g'
#define S_M1_W2 'h'
#define S_M1_W3 'i'
#define S_M2_C 'j'
#define S_M2_W1 'k'
#define S_M2_W2 'l'
#define S_M2_W3 'm'
#define S_E_C 'n'
#define S_E_W1 'o'
#define S_E_W2 'p'
#define S_E_W3 'q'
//describes theserial ports used to transfer informationfrom board to board
SoftwareSerial mySerial(0, 1); //tx, rx
//defining axis pins
#define ZPIN A0
#define YPIN A1
#define XPIN A2
//terms for thecomparefunction
#define DEFINE_1 1
#define DEFINE_2 1
#define DEFINE_3 3
#define DEFINE_4 3
#define DEFINE_5 6
//buttonpins
#define PATIENT 7
#define THERAPIST 8
//Array Size
#define ARRAYSIZE 100
//delay for patient and doctor
#define DELAY_TIME 100
#define MOTION_SENSE 0.1
//defining structs
struct three_axis {
float x;
float y;
float z;
};
struct four_returns{
float beginning;
float middle1;
float middle2;
float ending;
};
//functions
int button_read(intPin);
struct three_axis read_entire_accel(void);
float magnitude_of_accel_read(struct three_axis dog);
int move_check(void);
struct four_returnsarray_divider(int counter, float*array);
void compare_patient_and_doctor(structfour_returns*x, struct four_returns*y);
struct four_returnsratio_multiplier(float ratio, struct four_returnsx);
void setup()
{
mySerial.begin(9600);
// delay(1000);
mySerial.println("Starting Program");
pinMode(PATIENT, INPUT);
pinMode(THERAPIST, INPUT);
}
void loop()
{
//----------------------------------------------------------------------------------------------------------------------------------------Therapist Button Press
//status variableto check to seeif thetherapist has already pressed thebutton
static int Therapist_Button_Pressed_Previously =0;
//array thetherapist populates
float one_array[ARRAYSIZE];
//declaring struct four_returns for therapistand patient
static struct four_returnstherapist_four_parts;
static struct four_returnspatient_four_parts;
//counter for therapist
static int therapist_counter =0;
//counter for patient
static int patient_counter =0;
//checks if thetherapist button is pressed
if (button_read(THERAPIST)) {
Therapist_Button_Pressed_Previously =1;
//reset counter
therapist_counter =0;
/*
//checks if thebutton has been pushed and thedeviceis moving
while( !move_check() ) {
}
Serial.println("Therapistmovementchecked");
*/
while(button_read(THERAPIST) && therapist_counter <ARRAYSIZE) {
//this is going to betheread section
struct three_axis therapist_readout =read_entire_accel();
one_array[therapist_counter] = magnitude_of_accel_read(therapist_readout);
therapist_counter++;
delay(DELAY_TIME);
}
//functionthat divides array
therapist_four_parts=array_divider(therapist_counter, one_array);
/*
mySerial.print(therapist_four_parts.beginning);
mySerial.print(", ");
mySerial.print(therapist_four_parts.middle1);
mySerial.print(", ");
mySerial.print(therapist_four_parts.middle2);
mySerial.print(", ");
mySerial.print(therapist_four_parts.ending);
mySerial.print('n');
*/
// delay(2000);
//play standard tone
start_send(STANDARD_TONE);
}
//----------------------------------------------------------------------------------------------------------------------------------------Patient Correct Button Press
//checks if thepatient buttonis pressed and thetherapisthas already gone
if (button_read(PATIENT) && Therapist_Button_Pressed_Previously ==1) {
Serial.println("Patient Button Pressed");
//reset counter
patient_counter =0;
/*
//checks if thebutton has been pushedand thedeviceis moving
while( !move_check() ) {
}
Serial.println("Patient movementchecked");
*/
while(button_read(PATIENT) && patient_counter <ARRAYSIZE) {
//this is going to betheread section
struct three_axis patient_readout=read_entire_accel();
one_array[patient_counter] =magnitude_of_accel_read(patient_readout);
patient_counter++;
delay(DELAY_TIME);
}
//function that divides array
patient_four_parts=array_divider(patient_counter, one_array);
float ratio = therapist_counter/patient_counter;
patient_four_parts=ratio_multiplier(ratio, patient_four_parts);
/*
mySerial.print(patient_four_parts.beginning);
mySerial.print(", ");
mySerial.print(patient_four_parts.middle1);
mySerial.print(", ");
mySerial.print(patient_four_parts.middle2);
mySerial.print(", ");
mySerial.print(patient_four_parts.ending);
mySerial.print('n');
delay(2000);
*/
//compareto doctor
compare_patient_and_doctor(&therapist_four_parts, &patient_four_parts);
11. Now In Reality How Did It Work
• The device itself was able to compare simple movements (for
example a sphere, a square, a ‘Z’, etc.)
• However, complex movements were beyond it.
• Also the tones I created to differentiate the varying degrees of
incorrectness were too similar to easily distinguish.
• Therefore you could tell when the movement was right and when it
was wrong but not to what degree.
• This meant it was not good for helping patients explore to find the
correct movement.
12. What Happens Now
• The device will be passed onto another intern who will continue to
work on it.
• Maybe one day very far down the road, it will become a real medical
device.
13. Thank You
• A Big Thank You to Professor Reinkensmeyer for letting me work
under him.
• Thank you to all of the guys in the lab but a special thanks to Joan
Aguiler and Jaime Duarte who definitely helped me the most.
• Thank you to Dr. Ross Viola for setting up this internship and
Catherine Rupp for making sure everything ran smoothly.
• Finally, thank you UCI!