10 Years of Tunepal: Reflections & Future Directions
•Transferir como PPTX, PDF•
1 gostou•2,432 visualizações
Tunepal is a query-by-playing search engine for traditional music that handles around 60K music searches each month.
Recomendados
Mais conteúdo relacionado
Semelhante a 10 Years of Tunepal: Reflections & Future Directions
Semelhante a 10 Years of Tunepal: Reflections & Future Directions (20)
Mais de Bryan Duggan
Último
Último (20)
10 Years of Tunepal: Reflections & Future Directions
- 1. 10 Years of Tunepal: Reflections & Future Directions Dr Bryan Duggan School of Computer Science Technological University of Dublin Ireland bryan.duggan@dit.ie http://bryanduggan.org http://github.com/skooter500 @skooter500
- 2. Tunepal • A search-by-playing app & website for Android and IOS • 20K App users • 23K music scores • Around 1K music searches per day • 2,322,682 tap-to-record searches • 2,042,877 title searches • 3,619,592 tunes downloaded • Links to Comhaltas archive • MIDI Playback & music score • Links to youtube, thesession
- 3. • Show a demo!
- 4. Way back in 1993…
- 7. Duggan, Bryan: Learning traditional Irish music using a PDA. IADIS Mobile Learning Conference, Trinity College Dublin, Dublin, Ireland, July 26, 2006.
- 12. How does it work?
- 14. 2010 • Grant from the Dept of Culture Sport and Tourism • Had to complete all the work in 10 weeks – App available in Irish and English – New professionally designed UX – Music score display – Add additional tunebooks – iPad version Tunepal HD – Tune editing & composition
- 17. In the media • Irish Times • Irish Daily mail • Sunday Times - Top twenty cultural apps • John Creedon’s Fleadh program • Foinse • Top 25 grossing apps on iTunes (2 days)
- 19. Sunday Times - February 2012
- 22. • Duggan, B., Gainza, M. & Cunningham, P. Machine annotation of sets of traditional Irish dance tunes. Paper presented at the Ninth International Conference on Music Information Retrieval (ISMIR), Drexel University, Philadelphia, USA, 2008. http://www.audioresearchgroup.com/ • Bryan Duggan, Brendan O'Shea, (2011) "Tunepal: searching a digital library of traditional music scores", OCLC Systems & Services, Vol. 27 Iss: 4, pp.284 - 297
- 23. TuneTracker • Dynamically track and publish records of folk music playing • Deployed for five months in Dublin, Ireland at The Cobblestone pub • Proceedings of the 2014 conference on Designing interactive systems • Norman Makato Su, Indiana University Bloomington
- 25. 2015 Europeanna Collaboration • Replace the Java applet with record and transcription functionality implemented in HTML5 • Return recordings of music, not just music scores. • Make all the functionality of the Tunepal, including query-by-playing work similarly across all devices including smartphones. • Open-source Tunepal and make an API server available to other projects. • Magical to explore!
- 26. Technology • MySql • Jersey for Java API server • HTML5 Audio • Node, npm • ABC2MIDI • ABCJS • Emscripten • Materialize with Angular.JS • Europeana APIs • Works on Chrome, Firefox, Edge • Show another demo!!
- 29. Transcription Accuracy Study • Tony software 30 pieces of audio • https://github.com/pierrebeauguitte/tuneset • Variety of instruments, solo and ensemble • Midi format • Used as ground truth to evaluate 4 different algorithms – pYIN, Melodia, MATT, Silvet • Frame-Level Evaluation: Melody Extraction Task • Note-Level Evaluation: Note Tracking Task • Beauguitte, P., Duggan, B., Kelleher, J. (2016). A Corpus of Annotated Irish Traditional Dance Music Recordings: Design and Benchmark Evaluations. 17th ISMIR 2016.
- 31. Key Inference Beauguitte, P., Duggan, B. and Kelleher, J. (2017) Key inference from Irish traditional music scores and recordings.14th Sound and Music Computing Conference, July 5-8, 2017, Espoo, Finland.
- 32. Errors
- 33. Rhythm Inference • Categorised into – simple duple: 4 4 (reel, hornpipe, fling, barndance) and 2 4 (polka) – simple triple: 3 4 (waltz, mazurka) – compound duple: 6 8 (jigs) and 12 8 (slides) – compound triple: 9 8 (slip jigs) • Infer rhythmic information from audio recordings of Irish traditional tunes • Beauguitte, P., Duggan, B. & Kelleher, J. D. (2018). Rhythm inference from audio recordings of Irish traditional music. Proceedings of the 8th International Workshop on Folk Music Analysis, 26-29 June 2018, Thessaloniki (Greece)
- 34. Rhythm Inference • Relies on the repetitive nature of this musical genre
- 41. Keeping the apps up to date • Example from this year • Android 9 – Rebuild – Switch from Eclipse to Android Studio – Switch to Gradle from ad hock build system – Remove ActionBarSherlock library – A lot of time spent fixing UX bugs and crashes on certain phones in certain conditions – Gmail sending bug – Still random crashes • Europeanna searches failing – Lots of tracking down and dead ends trying to find what the problem was – Trying to build a 3 year old web app with a complex toolchain, java scriot, ecs6, node, npm, bower, git
- 43. • Show the map! • Show the visualisation!
- 44. Monthly usage over 10 years
- 45. Listening to ClareFM.ie this morning. Heard a great tune…..interrupted by the Galway Races Used Tunepal……..it’s called The Parting………got it from The Session.org. So much fun……….. listening and learning Irish Trad. Thank you so much for the good you have done and are doing! It is ironic, but one of my mates has proudly proclaimed she has never had a cell phone, computer, nor opened an email account. She is utterly fascinated with Tunepal “I’m sure you’re getting emails every day saying how much folks love your tunepal site and iphone app. I have the app and LOVVVVVVVVVE it! I use it every day. Awesome awesome awesome! Getting a lot of use out of this app now. Everybody I have told so far is very impressed.” It’s pure magic. Thanks a million for having made this diamond and share it with everyone. If there was a God, I’m quite sure that he would have you blessed ! I just wanted to thank you for the great idea and the great job you’ve done with Tunepal. It’s an excellent and very useful tool. Just what I whished for quite often. I am really impressed with the app. It was 100% correct in identifying tunes when I played them on flute or whistle. Having worked in a previous life in ultrasonic testing of materials, I can appreciate some of the problems involved. Thank you, Tunepal!!! You’ve saved lives!!!!!! Fair play to Bryan it’s a wonderful App I have learned lots of tunes from it and Names of Tunes as well Amazing! Congrats! I must be at least 30k of the tune searches 🤔☺ I use it so much. I love it
- 46. Conclusions • Technology has transformed the playing and learning of traditional music • Tunepal is beloved by its users – “It changed my life” • 10 years of data on usage • The future – Tune synchronisation – Better transcription accuracy – Key invariant matching – Modern Tunepal app – Redevelop the server – Get involved!
- 47. Contribute! • Get the code and contribute: – http://github.com/skooter500 – https://github.com/pierrebeauguitte/tunes_annot ations – https://github.com/pierrebeauguitte/tuneset • Contact me – @skooter50 – bryan.duggan@dit.ie – bryanduggan.org
Notas do Editor
- Teaching courses on coding game development & VR in the school of computer science Thank Izzy and the Organising committee for inviting me Teach students how to make Tunepal as part of my OOP class Author & inventor of Tunepal
- Store the score in a tunebook Thesession.org run by Jeremy Keith Bill Black’s transcriptions of Ceol Rince na Heireann O’Neills American Marching band Ceol Rince na Heireann O Farrells Bulmer & Sharpley Heinrik Norebecks tune collections
- Psion version late 1990’s Windows version early 2000’s working for Bear Stearns.
- Palmtop User magazine 2002
- 2004 Poster for a mobile learning conference
- MATT2 and Tansey Heinrik Norebecks collection I was working on a different problem Genuine Eurica moment the day I made it work for the first time Sent Tansey a copy of my thesis
- Even by the standards of the time! It was pretty shit looking One cant be good at everything
- 11 Feb 2010 iPhone app 1 July 2010 with Tunepal for Android 15 December 2010 with Tunepal HD for iPad Zeitgeist Phones could do the transcription Also phones were small & unobtrusive enough to bring to a session No music score display
- Fuzzy Histogram clusters notes together based on similar durations!! Ornamentation Filtering Harmonicity changes All detailed in my PhD thesis and various publications ABCJS by Paul Rosen and Gregory Dyke ABC2PS by Michael Methfessel ABC2PS iPhone port by Jeremy Huxtable ABC4J by Lionel Gueganton. ABC2MIDI by Seymour Shlien
- Marco Cascorino Morgan Conlon Jamie Osler Lisa Shields Jonathan Lynch & Scott Baker Siobhán Nic Gaoithín, Oifig na Gaeilge
- New website Geotagged maps (no longer work!) Translations New tunebooks iPad version
- 2011, 2012
- Sunday Times top 50 cultural apps Mention Avery Wang from Shazam
- Top 50 apps of all time
- Met Avery Wang of Shazam who offered me a job
- Wrote some articles for this Kept the apps updated
- dynamically track and publish records of folk music playing It captures, stores, and posts the names of tunes played in Irish traditional music sessions on a public website Disambiguate between noise and music based on spectral flux Tunes were annotated based on majority voting of several 12 second samples 20-50% accuracy , longitudinal statistics) of tune playing practices, the changing repertoire of a session. to examine whether instrumentation of traditional and folk arts is and should be possible Semi structured interviews and less formal during session with anchor musicians At the same time, TuneTracker seeks to respect the privacy of musicians by not recording or streaming any actual audio. Deployed for five months in Dublin, Ireland at The Cobblestone pub Amateurs can intelligently practice certain (e.g., most popular) tunes in a session. Professionals will be able to reflect on the repertoire of their sessions. TuneTracker is also an academic exercise in intervention
- Some of the anchor musicians of a particular night requested that we turn off TuneTracker during their session time semi-structured [24] interviews (N=7) and less formal interviews during session observations (N=4) with anchor musicians. In other words, TuneTracker is imprinting on amateurs a deterministic image of what a session is—that it follows a formula One anchor told us he received a text from a friend saying “I see you’re in The Cobblestone tonight from the tunes that are being played.” Through the tunes, the musicians were being tracked. When seeing the low percent of polkas (3%) in her session, an anchor told us, “Now, we’ll have to do something about that! I don’t think technology should be pushing the music. I think the musicians should be pushing the music...It’s a bit [like] a genie out of the bottle though. In the second quote, the musicians see TuneTracker as a tool, not a substitute for the tradition. For them, any artifact that can encourage more people to play and reflect on traditional music, thereby preserving the tradition, is beneficial. TuneTracker will encourage visiting musicians to join in sessions. TuneTracker has agency [5]—it deliberately introduces a power asymmetry [4] by wresting control of sessions away from the musicians and into a technological artifact. TuneTracker will be prescriptive: spelling out what tunes people should demand in sessions. Because TuneTracker is authoritatively clothed in digital garb, it asserts itself as the final truth—whether it be regarding tune names, tune notes, the order of tunes in a set, etc. Thus, TuneTracker threatens to destroy tradition by marginalizing the human element in the traditional process Just like any resource (e.g., sheet music), it can be misused
- Bryan Duggan Lise DenBrock Breandán Knowlton Chris Xue 50 million cultural heritage items (from books and paintings to 3D objects and audiovisual material) that celebrate over 3,500 cultural institutions across Europe. Comhaltas 30 second extracts Replace the Java applet with record and transcription functionality implemented in HTML5 Return recordings of music, not just music scores. Make all the functionality of the Tunepal, including query-by-playing work similarly across all devices including smartphones. Open-source Tunepal and make an API server available to other projects. A Search Through Time: Connecting Live Playing to Archive Recordings of Traditional Music 6th. International Workshop on Folk Music Analysis, 15-17 June, 2016. Hosted in TU Dublin Europeanna Sounds Conference in Paris
- user trials for the week of Feadh Ceoil na hEireann in various locations in Sligo in August 2015. In total 40 users tested the new version of Tunepal Limitations of existing Tunepal Support for different instruments such as banjo, concertina Not key invariant searches. Many of the searches were failing because the players were playing ionj 2 of the challenges that were taken on my Pierre Beauguitte in 2015 Approaches these problems with great academic rigour Coming the completion of his PhD
- 2015 Pierre Beauguitte co-supervised with Professor John Kelleher Immersed himself in traditional music & culture Attended classes in the conservatory O’Donohues pub Traditional music ensemble, Peter Browne, Ciaran Hanrahan, Thomas Dorely Gig with Mark Redmond, piper,
- Similar work done by IZZY Co supervising 5 years of outstandiung scholarship to bear on these problems 30 tunes Foinn Seisiun´ books published by the Comhaltas Ceoltoir ´ ´ı Eireann ´ organisation, available with Creative Commons licence. These offer good quality, homogeneous examples of the heterophony inherent to an Irish traditional dance music session. • Grey Larsen’s MP3s for 300 Gems of Irish Music for All Instruments, commercially available. These consist of studio quality recordings of tunes played on Irish flute, tin whistle and anglo concertina. • personal recordings of the second author, a renowned musician, on the Irish flute. These are available together with the annotations. Tony software [14]. After importing an audio file, Tony offers estimates using the pYIN algorithm (notelevel) presented in the next section. These were then manually corrected, by adding new notes, merging repeated notes and adjusting frequencies. The annotations were finally post-processed to convert these frequencies to the closest MIDI note references Exported to CSV files a note is considered correctly transcribed if its onset is distant from the reference note by less than 50ms and its pitch by less than a quarter of a tone
- Precision, recall and F-measures are computed with the mir eval framework It is interesting to observe that MATT achieves the highest F-measures on the solo recordings. However, only the difference with Silvet is statistically significant On the session recordings, Silvet achieves significantly higher pitch precisions Why?
- musical key consists of a tonic note, represented by a pitch class (C, C#, D...), and a mode, or rather mode family, which can be minor or major 24 candidate keys, for the 12 semitones of the octave and the two considered modes The standard approach to identifying keys in a musical piece is to use key-profiles They can be seen as vectors assigning weights to the twelve semitones The paper compares different approaches to generating keyprofiles from the literature with two new approaches introduced by the author based on CADENCES and modes in traditional music Pierre annotated an audio dataset 636 items from the fionn session tunebook, Grey Larsen’s 300 gems, also a symbolic dataset drawn from thesession.org Ran an experiment which extracted the pitch class histogram from the tune in audio or symbolic format Audio files used deep chromas Key was evaluated by comparing against the 24 key profiles one for each semitone. Cross product Highest score Hence it appears that inferring keys from heterophonic or polyphonic audio is easier than on monophonic recordings. An explanation for this is that the harmonic content is richer in heterophonic and polyphonic signals. Cadences outperform the existing key profiles Propose an improvement by adding weights 10 fold cross validation is used to estimate the size of a grid search to evaluate the optimum weights Using the evaluated weights Pierre is able to improve the mirex score again Parametric profiles
- Two new types of error neighbour key errors and relative key errors. The high frequencies of these two types of errors can be explained by the specific characteristics of Irish traditional music minor keys are harder to detect than major keys The key annotations on the datasets were made by the first author, and it is possible that other annotators could annotate some of the tunes differently Ambiguous
- Simple and compound refer to the beat subdivision, duple and triple refer to the grouping of beats Show examples Down the broom Slide Slide Brosna Slip Jig – Drops of Brandy
- obtain an onset detection function by a method of spectral difference The autocorrelation functions is then computed on a 5s window of the SD function. The quaver duration q is determined by the fuzzy histogram algorithm, a clustering algorithm A quantised lag vector is calculated what exactly is? Fed into a logistic regression classifier 10 fold cross validation Used a dataset the collection of recordings accompanying the Foinn Seisiun books published by the Comhaltas Ceoltoirı Eireann organisation, which he labelled with Labelled with tune type and beat subdivision 1st experiment easy, Tune type similar to meter detection, reel, hoprnpipe and barndance 326 unique recordings The models of both experiments predict a label for a 5s window 2 experiments 1. simple, compound 2. Tunetype
- 15787 tunes from thesession,org Audio files from the previous experiment tune id in the search space Key deviation used in the ground truth manually annotated to provide ground truth 4 random offsets are chosen on each tune, spaced by a minimum of 5 seconds such that overlap is allowed. Our dataset thus consists of 2000 12 second excerpts https://dl.acm.org/citation.cfm?id=2660188 pitch class is be represented by integers Ornamentation notes are filtered out Transcribes to chromas, pYin, Melodia, Silvet Quantised using the fuzzy histogram algorithm
- Pitch class histograms are generated for the symbolic audio 120 valued vector. A Gaussian kernel of size 150 cents . The vector is normalised to sum to 1. Makes the representation close to the audio one Audio PCH STFT, summed, 120 valued vector Aligned using Bhattacharyya coefficient. Select the shift that has the highest score that provides the best alignment these symbolic PCH can be pre-computed. Once this optimal shift k is found, the symbolic quaver sequence is transposed by round(k/10) semitones, as the unit for k is 10 cents
- Wu and Manber (1992) present a bit-parallel algorithm for finding occurrences QbSH tMIREX Top-10 hit rate (1 point is scored for a hit in the top 10 and 0 is scored otherwise) Best (Top 1) Hit rate a new way of ranking is now introduced worst possible rank (WPR) worst possible rank (WPR), is a way of ranking results that appropriately deals with draws Relative difference? Original with a tip – glass ceiling All – Go through all possible semitone transpositions and pick the best – brute force
- PCH to the best a novel approach. Existing literature has often focused on audio-to-audio alignment in the context of cover song identification (EllisandPoliner, 2007;Serraetal.,2008), or has relied on keyannota tion when dealing with symbolic music The method introduced here performs audio-to-symbolic alignment without the need for key annotation, thus alleviating the difficulty of defining key in ITM Without reducing accuracy and by adding 50% to the tune identification task compared to 12 times more for the brute force approach Pierres work shows us the road ahead for tunepal
- Other examples move from Webworld to Azure hosting Need to move to ssl Browser updates Europeanna search API changed Caused the archive searches to fail for the past 6 months or so, despite my best efforts to track it down
- Tunepal Heros! Chris Jianghan Xue Mahmoud Almahroum DT282 Computer Science student
- Katie Kilroy is a Senior Analyst with Arlo Smart Home Technologies who works with all things data, process, and systems on a daily basis button accordion at Irish trad sessions around Cork City
- Average of 65,505 per month 1.4 searches per minute
- A generation of musicians it was how they learned the names of tunes Used my teachers, professionals, amatures alike Find a tune in any key on any instrument Tunepal is too important for one person! 4 million interactions. 2 million music searches! Thanks to Pierre 70% geotagged