SlideShare uma empresa Scribd logo
1 de 25
A WEARABLE MUSICAL
INTERFACE USING A WIRELESS
SENSOR NETWORK
Francesco Corucci – – 2012
MSc in Computer Engineering – University of Pisa
Introduction
 Goal: implement a platform that allows the use of a
wireless sensor network as a wearable interface
that can be used for producing music
 Requirements:
 Modular
 Flexible
 MIDI-compliant
2
What is MIDI
 MIDI (Musical Instrument Digital Interface) is a
standard technology for the interaction between
musical devices
 It standardizes:
 A communication protocol
 An hardware interface
 Versatile technology, widely used since the 80’s and still
reliable after more than 30 years
3
Shimmer sensor nodes
 For setting up the WSN we choose Shimmer sensor
nodes :
 Suitable for wearable applications
 Integrate a 3-axis accelerometer
 Optional gyroscope shield
(not used in this project)
 Bluetooth or 802.15.4
communication
 Support TinyOS
4
Shimmer sensor nodes
5
Overview of the system
USB
802.15.4
• Two or more wireless sensors on the body
• The sensors sample accelerometric data
associated with movements
• Data are then collected from the sink and
forwarded to the computer
Sink
6
Overview of the system (cont’d)
Oscilloscope application
(extended with MIDI map module)
Virtual MIDI
device
Musical software
Data from sink
• Filtering
• Displaying
• Mapping in MIDI commands
7
Detailed architecture
Serial forwarder
Computer
USB
TCP/IP
Oscilloscope
Radio
MIDI map
MIDI forwarder
Virtual MIDI port
TCP/IP
Musical sw
(VST-host)
VST
Plugin
VST
Plugin
MIDI
MIDI
8
…
MIDI controller
This way the WSN become a standard MIDI controller,
that can interoperate with every musical software
9
How to choose the mapping
 The way to map sensed data into MIDI messages is
absolutely arbitrary, and can lead to a multitude of
expressive results
 Every data channel can be mapped in a different
 instrument
 effect
 note/chord
 etc…
 …and this can be done in a variety of ways!
10
Demonstration time
(http://youtu.be/jIrE5j2Mdtw)
11
Features
 Modular, extensible:
 Is it possible to use an arbitrary number of sensor nodes, and
choose which on-board sensors must be used as data sources
 Flexible:
 Many degrees of freedom in the mapping
 Many expressive possibilities
 Sound generation is left to specialized and mature softwares
 Standard:
 Since the WSN becomes a MIDI controller, it is able to
interoperate with a moltitude of software platforms
12
Possible applications
The platform can be used for several purposes:
 New expressive forms based on gestuality
 In combination with traditional instruments
(e.g. influence of on-stage movements on played music)
 Any application that asks for sonification of human movements:
 Sport training
 Rehabilitation
 …
13
References
 Shimmer sensor nodes
 http://www.shimmer-research.com/
 General information about MIDI
 http://en.wikipedia.org/wiki/MIDI
 VST
 http://en.wikipedia.org/wiki/Virtual_Studio_Technology
 rtMidi, a cross-platform C++ library for MIDI
 http://www.music.mcgill.ca/~gary/rtmidi/
 Sonification
 http://en.wikipedia.org/wiki/Sonification
14
Thank you for your attention :-)
15
Implementation details
16
OscilloscopeAppC - overview
 OscilloscopeAppC is the TinyOS application running
on sensor nodes
 Receives commands from the sink:
start/stop/change_freq
 Samples the on-board tri-axis accelerometer using
three ADC channels in DMA mode
 When a block of samples has been collected, the
sensor node sends it to the sink via radio
17
OscilloscopeAppC - architecture
18
OscilloscopeAppC - interactions
19
…
BasestationC - overview
 BasestationC is the TinyOS application running on
the sink node
 Acts as a bridge between the serial and the radio
link
 Implements queues in both direction to handle
traffic spikes
20
BasestationC - architecture
21
MIDI map module
 Offers a simple GUI: the user can choose how to map
the active data channels into MIDI messages from a
predefined set of mappings
22
 The module then catches
incoming data, maps it into
MIDI command according
to user-defined
preferences, and forwards
them via TCP/IP socket
toward the MIDI forwarder
MIDI forwarder
 Implements a virtual
MIDI device: the WSN is
seen as an input MIDI
peripheral plugged into
the computer
 It then acts as a bridge:
 Waits for incoming MIDI
messages from a TCP/IP
socket
 Forwards MIDI messages
to the virtual MIDI port
23
Screenshots: Oscilloscope + MidiMap
24
Screenshots
25

Mais conteúdo relacionado

Semelhante a A wearable MIDI interface using a wireless sensor network

Passenger amenities (2)
Passenger amenities (2)Passenger amenities (2)
Passenger amenities (2)Suraj Soni
 
Education set for collecting and visualizing data using sensor system based o...
Education set for collecting and visualizing data using sensor system based o...Education set for collecting and visualizing data using sensor system based o...
Education set for collecting and visualizing data using sensor system based o...IJMER
 
Experimental Analysis of an Energy-Efficient WSN
Experimental Analysis of an Energy-Efficient WSNExperimental Analysis of an Energy-Efficient WSN
Experimental Analysis of an Energy-Efficient WSNidescitation
 
StadtLärm - A Distributed Urban Noise Monitoring System
StadtLärm - A Distributed Urban Noise Monitoring System StadtLärm - A Distributed Urban Noise Monitoring System
StadtLärm - A Distributed Urban Noise Monitoring System M2M Alliance e.V.
 
Education set for collecting and visualizing data using sensor system based ...
Education set for collecting and visualizing data using sensor  system based ...Education set for collecting and visualizing data using sensor  system based ...
Education set for collecting and visualizing data using sensor system based ...IJMER
 
Decibel meter using IoT with notice board
Decibel meter using IoT with notice boardDecibel meter using IoT with notice board
Decibel meter using IoT with notice boardIRJET Journal
 
Garbage Monitoring System using Arduino
Garbage Monitoring System using ArduinoGarbage Monitoring System using Arduino
Garbage Monitoring System using Arduinoijtsrd
 
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...csandit
 
A proposed model for traffic signal preemption using global positioning syste...
A proposed model for traffic signal preemption using global positioning syste...A proposed model for traffic signal preemption using global positioning syste...
A proposed model for traffic signal preemption using global positioning syste...csandit
 
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...cscpconf
 
A ZigBee Based Wireless Sensor Network for an Agricultural Environment
A ZigBee Based Wireless Sensor Network for an Agricultural EnvironmentA ZigBee Based Wireless Sensor Network for an Agricultural Environment
A ZigBee Based Wireless Sensor Network for an Agricultural Environmentijsrd.com
 
6 article azojete vol 8 53 66
6 article azojete vol 8 53 666 article azojete vol 8 53 66
6 article azojete vol 8 53 66Oyeniyi Samuel
 
Wireless multimedia sensor networking
Wireless multimedia sensor networkingWireless multimedia sensor networking
Wireless multimedia sensor networkingKedar Kumar
 
Wireless Open Weather (WOW)
Wireless Open Weather (WOW)Wireless Open Weather (WOW)
Wireless Open Weather (WOW)Mirko Mancin
 

Semelhante a A wearable MIDI interface using a wireless sensor network (20)

Passenger amenities (2)
Passenger amenities (2)Passenger amenities (2)
Passenger amenities (2)
 
WIRELESS SENSOR NETWORK
WIRELESS SENSOR NETWORKWIRELESS SENSOR NETWORK
WIRELESS SENSOR NETWORK
 
Education set for collecting and visualizing data using sensor system based o...
Education set for collecting and visualizing data using sensor system based o...Education set for collecting and visualizing data using sensor system based o...
Education set for collecting and visualizing data using sensor system based o...
 
Experimental Analysis of an Energy-Efficient WSN
Experimental Analysis of an Energy-Efficient WSNExperimental Analysis of an Energy-Efficient WSN
Experimental Analysis of an Energy-Efficient WSN
 
StadtLärm - A Distributed Urban Noise Monitoring System
StadtLärm - A Distributed Urban Noise Monitoring System StadtLärm - A Distributed Urban Noise Monitoring System
StadtLärm - A Distributed Urban Noise Monitoring System
 
Education set for collecting and visualizing data using sensor system based ...
Education set for collecting and visualizing data using sensor  system based ...Education set for collecting and visualizing data using sensor  system based ...
Education set for collecting and visualizing data using sensor system based ...
 
Aquarius_IJARCCE
Aquarius_IJARCCEAquarius_IJARCCE
Aquarius_IJARCCE
 
Decibel meter using IoT with notice board
Decibel meter using IoT with notice boardDecibel meter using IoT with notice board
Decibel meter using IoT with notice board
 
Garbage Monitoring System using Arduino
Garbage Monitoring System using ArduinoGarbage Monitoring System using Arduino
Garbage Monitoring System using Arduino
 
published paper
published paperpublished paper
published paper
 
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
 
A proposed model for traffic signal preemption using global positioning syste...
A proposed model for traffic signal preemption using global positioning syste...A proposed model for traffic signal preemption using global positioning syste...
A proposed model for traffic signal preemption using global positioning syste...
 
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
A PROPOSED MODEL FOR TRAFFIC SIGNAL PREEMPTION USING GLOBAL POSITIONING SYSTE...
 
CDMA
CDMACDMA
CDMA
 
A ZigBee Based Wireless Sensor Network for an Agricultural Environment
A ZigBee Based Wireless Sensor Network for an Agricultural EnvironmentA ZigBee Based Wireless Sensor Network for an Agricultural Environment
A ZigBee Based Wireless Sensor Network for an Agricultural Environment
 
6 article azojete vol 8 53 66
6 article azojete vol 8 53 666 article azojete vol 8 53 66
6 article azojete vol 8 53 66
 
Contiki wsn
Contiki wsnContiki wsn
Contiki wsn
 
Wireless multimedia sensor networking
Wireless multimedia sensor networkingWireless multimedia sensor networking
Wireless multimedia sensor networking
 
Wireless Open Weather (WOW)
Wireless Open Weather (WOW)Wireless Open Weather (WOW)
Wireless Open Weather (WOW)
 
Arduino Radar System
Arduino Radar SystemArduino Radar System
Arduino Radar System
 

Mais de Francesco Corucci

Teleoperating a robotic arm through a gyroscopic helmet
Teleoperating a robotic arm through a gyroscopic helmetTeleoperating a robotic arm through a gyroscopic helmet
Teleoperating a robotic arm through a gyroscopic helmetFrancesco Corucci
 
Real-time PID control of an inverted pendulum
Real-time PID control of an inverted pendulumReal-time PID control of an inverted pendulum
Real-time PID control of an inverted pendulumFrancesco Corucci
 
Social Network Analysis Project
Social Network Analysis ProjectSocial Network Analysis Project
Social Network Analysis ProjectFrancesco Corucci
 
Implementation of a lane-tracking system for autonomous driving using Kalman ...
Implementation of a lane-tracking system for autonomous driving using Kalman ...Implementation of a lane-tracking system for autonomous driving using Kalman ...
Implementation of a lane-tracking system for autonomous driving using Kalman ...Francesco Corucci
 
An overview on Quantum Key Distribution
An overview on Quantum Key DistributionAn overview on Quantum Key Distribution
An overview on Quantum Key DistributionFrancesco Corucci
 
P-Systems for approximating NP-Complete optimization problems
P-Systems for approximating NP-Complete optimization problemsP-Systems for approximating NP-Complete optimization problems
P-Systems for approximating NP-Complete optimization problemsFrancesco Corucci
 

Mais de Francesco Corucci (6)

Teleoperating a robotic arm through a gyroscopic helmet
Teleoperating a robotic arm through a gyroscopic helmetTeleoperating a robotic arm through a gyroscopic helmet
Teleoperating a robotic arm through a gyroscopic helmet
 
Real-time PID control of an inverted pendulum
Real-time PID control of an inverted pendulumReal-time PID control of an inverted pendulum
Real-time PID control of an inverted pendulum
 
Social Network Analysis Project
Social Network Analysis ProjectSocial Network Analysis Project
Social Network Analysis Project
 
Implementation of a lane-tracking system for autonomous driving using Kalman ...
Implementation of a lane-tracking system for autonomous driving using Kalman ...Implementation of a lane-tracking system for autonomous driving using Kalman ...
Implementation of a lane-tracking system for autonomous driving using Kalman ...
 
An overview on Quantum Key Distribution
An overview on Quantum Key DistributionAn overview on Quantum Key Distribution
An overview on Quantum Key Distribution
 
P-Systems for approximating NP-Complete optimization problems
P-Systems for approximating NP-Complete optimization problemsP-Systems for approximating NP-Complete optimization problems
P-Systems for approximating NP-Complete optimization problems
 

Último

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 

Último (20)

Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 

A wearable MIDI interface using a wireless sensor network

  • 1. A WEARABLE MUSICAL INTERFACE USING A WIRELESS SENSOR NETWORK Francesco Corucci – – 2012 MSc in Computer Engineering – University of Pisa
  • 2. Introduction  Goal: implement a platform that allows the use of a wireless sensor network as a wearable interface that can be used for producing music  Requirements:  Modular  Flexible  MIDI-compliant 2
  • 3. What is MIDI  MIDI (Musical Instrument Digital Interface) is a standard technology for the interaction between musical devices  It standardizes:  A communication protocol  An hardware interface  Versatile technology, widely used since the 80’s and still reliable after more than 30 years 3
  • 4. Shimmer sensor nodes  For setting up the WSN we choose Shimmer sensor nodes :  Suitable for wearable applications  Integrate a 3-axis accelerometer  Optional gyroscope shield (not used in this project)  Bluetooth or 802.15.4 communication  Support TinyOS 4
  • 6. Overview of the system USB 802.15.4 • Two or more wireless sensors on the body • The sensors sample accelerometric data associated with movements • Data are then collected from the sink and forwarded to the computer Sink 6
  • 7. Overview of the system (cont’d) Oscilloscope application (extended with MIDI map module) Virtual MIDI device Musical software Data from sink • Filtering • Displaying • Mapping in MIDI commands 7
  • 8. Detailed architecture Serial forwarder Computer USB TCP/IP Oscilloscope Radio MIDI map MIDI forwarder Virtual MIDI port TCP/IP Musical sw (VST-host) VST Plugin VST Plugin MIDI MIDI 8 …
  • 9. MIDI controller This way the WSN become a standard MIDI controller, that can interoperate with every musical software 9
  • 10. How to choose the mapping  The way to map sensed data into MIDI messages is absolutely arbitrary, and can lead to a multitude of expressive results  Every data channel can be mapped in a different  instrument  effect  note/chord  etc…  …and this can be done in a variety of ways! 10
  • 12. Features  Modular, extensible:  Is it possible to use an arbitrary number of sensor nodes, and choose which on-board sensors must be used as data sources  Flexible:  Many degrees of freedom in the mapping  Many expressive possibilities  Sound generation is left to specialized and mature softwares  Standard:  Since the WSN becomes a MIDI controller, it is able to interoperate with a moltitude of software platforms 12
  • 13. Possible applications The platform can be used for several purposes:  New expressive forms based on gestuality  In combination with traditional instruments (e.g. influence of on-stage movements on played music)  Any application that asks for sonification of human movements:  Sport training  Rehabilitation  … 13
  • 14. References  Shimmer sensor nodes  http://www.shimmer-research.com/  General information about MIDI  http://en.wikipedia.org/wiki/MIDI  VST  http://en.wikipedia.org/wiki/Virtual_Studio_Technology  rtMidi, a cross-platform C++ library for MIDI  http://www.music.mcgill.ca/~gary/rtmidi/  Sonification  http://en.wikipedia.org/wiki/Sonification 14
  • 15. Thank you for your attention :-) 15
  • 17. OscilloscopeAppC - overview  OscilloscopeAppC is the TinyOS application running on sensor nodes  Receives commands from the sink: start/stop/change_freq  Samples the on-board tri-axis accelerometer using three ADC channels in DMA mode  When a block of samples has been collected, the sensor node sends it to the sink via radio 17
  • 20. BasestationC - overview  BasestationC is the TinyOS application running on the sink node  Acts as a bridge between the serial and the radio link  Implements queues in both direction to handle traffic spikes 20
  • 22. MIDI map module  Offers a simple GUI: the user can choose how to map the active data channels into MIDI messages from a predefined set of mappings 22  The module then catches incoming data, maps it into MIDI command according to user-defined preferences, and forwards them via TCP/IP socket toward the MIDI forwarder
  • 23. MIDI forwarder  Implements a virtual MIDI device: the WSN is seen as an input MIDI peripheral plugged into the computer  It then acts as a bridge:  Waits for incoming MIDI messages from a TCP/IP socket  Forwards MIDI messages to the virtual MIDI port 23