This document proposes a project to gather large amounts of semantic data from music producers during the music creation process. It involves developing plugins that extract low-level audio features and allow producers to annotate them with semantic descriptors. The goals are to (1) collect semantic data, (2) identify correlations in the data, and (3) use these correlations to aid music production tasks. As a proof of concept, the document outlines a mini-project to collect semantic data from musicians and evaluate suitable systems for future research.

1. Large-Scale Capture of Producer-Defined Musical
Semantics
Ryan Stables
School of Digital Media Technology
Birmingham City University

2. Problem...

3. Problem Definition
Producer:
Audio effects parameters
usually refer to low-level
attributes.
Professionally produced audio
often requires extensive
training.
Researcher:
Lack of semantically annotated
music production datasets.
How can we map low-level
descriptors to perceived
muscial timbre?

4. Problem Definition
Descriptors need to represent
the views of music producers.
These may change with genre,
musical instruments, etc...
Various terms may be used to
define similar things (colour,
texture etc...)

5. Project Aims
1. Gather large amounts of semantics data during the music
creation/production process.
Develop a series of DAW plug-ins.
Extract information and anonymously upload it to a server.
2. Identify correlation and patterns in the semantics data.
3. Use the data to improve/aid music production tasks.

6. Model...

7. Project overview
Server
Descriptor name...
Save...Load...
Save...
Semantic Descriptor
Parameter Space
Feature Set
Pre/Post Gain
Analysis...
Natural Language
Processing
Dimensionality
Reduction
Etc...
(1)
(2)(3)
Figure : Schematic Overview of the Semantic Audio Feature Extraction Project.

8. (1) Plug-in interface
Parameters can be set
experimentally.
Semantic descriptors to be
stored in text field.
Descriptors can be loaded
through same interface.
Parameters are stored and/or
set.
Figure : Semantic Audio plug-in: Multi-band distortion

9. (2) Feature Extraction
Features are extracted from the
selected region.
The parameter space is stored.
Semantic descriptors are sent
as targets.
Additional metadata is sent, if
available.
Server
Descriptor name...
Save...Load...
Save...
Semantic Descriptor
Parameter Space
Feature Set
Pre/Post Gain
Analysis...
Natural Language
Processing
Dimensionality
Reduction
Etc...
Figure : Stored attributes.

10. (3) Mapping
NLP Algorithms to identify
semantic correlation.
Dimensionality reduction to
find correlation in
features/parameters.
Additional data partitions
based on metadata (Genre,
instrument, etc...)
Results sent back to user
plug-in.
Server
Descriptor name...
Save...Load...
Sav
Semantic
Paramete
Feature Se
Pre/
Analysis...
Natural Language
Processing
Dimensionality
Reduction
Etc...
Figure : Results processing

11. Design Constraints...

12. Architecture
Requirements:
Maximisation of user-base.
Transparency: Access to the processing chain.
Design decisions:
Stand-alone plug-ins.
MultiFX.
Plug-ins within a plug-in.
Analysis-only.
Other:
Free field vs. fixed word.
Before and after.
Metadata pane.

13. Analysis framework
LibXtract.
Hard-coded, C library.
Around 400 combined
audio features*.
[Bullock, 2007]
Vamp.
Plug-in within a plug-in.
Hosts LibXtract features,
amongst others.
[Cannam et al., 2006]

14. Mini-Project...

15. Mini-Project: Aims
Analyse the production requirements of musicians.
Birmingham Conservatoire
The Music Producers Guild
The Birmingham Music Network
Build a series of prototype systems for the collection of
musical semantics data.
Use these systems to collect data from a small group of
musicians during the production process.
Evaluate the results in order to identify a suitable system for
future research.
Demonstrate the feasibility of a wider research project in this
area.

16. Mini Project: Schematic
Plug-in
development
Interface design
Algorithm
Development
Server, network,
data distribution
User Testing Data
Aquisition
Results
Analysis
Figure : Schematic Overview of the Mini-Project.

17. Positions and Timescale
2 x PhD Students: 1 x C4DM (QMUL) & 1 x DMT (BCU).
3 x Advisory roles.
Timescale: 6-months from September 2013.
Future: collaborative grant application.
Thanks!
ryan.stables@bcu.ac.uk

18. References
Bullock, J. (2007).
Libxtract: A lightweight library for audio feature extraction.
In Proceedings of the International Computer Music
Conference, volume 43.
Cannam, C., Landone, C., Sandler, M. B., and Bello, J. P.
(2006).
The sonic visualiser: A visualisation platform for semantic
descriptors from musical signals.
In ISMIR, pages 324–327.