The first WoRMS workshop (@ ISMIR)

20.9.2018 was historical day as then the 1st international workshop on reading music systems was held. From the National Library of Finland there were two participants in order to hear about music recognition and to get new insights on what kind of new methods researchers have come up with machine learning or with optical music recognition (OMR).

The lightweight proceedings of papers can be found from Worms homepage. All of the papers went through a light-weight review in the OpenReview-platform, after which authors could adjust they papers accordingly.

Notes from organizers’ (Jorge Calvo-Zaragoza) opening words about background of the workshop

He mentioned, that despite OMR has been researched a while, but there is still no focused community. The of independent research seems to stay independent even though there are used different approaches, data formats, evaluation criteria and even datasets,  but very few re-uses of previous works.  This is also something that all hackathons in general seem to end up – hackathons always end up with new ideas, so then all the old ideas are left in the sides.

History of WorMS: In 12th IAPR intl workshop on graphics recognition (GREC’17) , a number of IMR people happened to meet. There after some discussions, they came to conclusion that OMR should define itself, and also probably find its proper place in the crux of the different research areas.

So in order to widen the user-base, the workshop was defined to be  about systems that read documents of written music. Not strictly exclusive of OMR, but instead to cover any computational task for which the objects of study are related to written music document fits:

  • omr
  • information extraction
  • keyword spotting

Workshop on Reading music systems

  • The first edition is held as a satellite of ISMIR’18
  • there exists several events about machine learning, which can be applied to document analysis.

Session 1.

Jan Hajič jr. from Charles University had a thought-provoking talk on the intrinsic evaluation of the OMR. Sheet music has many aspects to monitor and there might not be a one common metric to rule them all. For example, one eye-opening example was that there is no good “edit-distance” for music scores.

In a way, many of the things did sound so very similar as the text recognition errors, which is quite familiar field from the post-processing of digitization or which you might have seen when analysing texts of the old Finnish newspapers.

Alexander Pacha from TU Wien aimed to the building of a community of OMR and strived to support reproducible research via coding conventions and just by promoting the idea of sharing datasets and code with appropriate licenses. These OMR datasets can be found here: , which should cover at least some state of the art quite well. As a summary he emphasized these rules as a staring point:

“Publish your datasets , strive for compatibility, prefer simple encodings and formats”


Session 2: Applications and Interactive Systems


  • HISPAMUS: Handwritten Spanish Music Heritage Preservation by Automatic Transcription by José M. Iñesta, Pedro J. Ponce de León, David Rizo, José Oncina, Luisa Micó, Juan Ramón Rico, Carlos Pérez-Sancho, Antonio Pertusa
Interesting software from a group doing pattern recognition and ML who then advanced to OMR. The medieval scores are not easy to understand by modern musicians. The system does layout analysis, symbol detection & recognition.
  • Developing an environment for teaching computers to read music by Gabriel Vigliensoni, Jorge Calvo-Zaragoza, Ichiro Fujinaga
Pixel.js is a web based app to correct the output of image segmentation processes (Saleh et al.)
  • Optical Music Recognition and Human-in-the-loop Computation by Liang Chen, Christopher Raphael

Aims to recognize symbols, chords, beam groups etc. Involving end-user to get to more granular level.

[1] Choi et al. figure 2. Interface of human-interactive OMR system.

Session 3: Technical Solutions

A Starting Point for Handwritten Music Recognition by Arnau Baró, Pau Riba, Alicia Fornés
Music Symbol Detection with Faster R-CNN Using Synthetic Annotations by Kwon-Young Choi, Bertrand Coüasnon, Yann Ricquebourg, Richard Zanibbi Idea was to do symbol recognition and utilize faster R-CNN in that task. SmuFL was used for the music font layout. Also mentioned was DocCreator which was a system developed for creation of ground truth.
DeepScores and Deep Watershed Detection: current state and open issues by Ismail Elezi, Lukas Tuggener, Marcello Pelillo, Thilo Stadelmann The target data used multiple ligthning conditions and multiple different printers for example, to get variance, which can be difficult for software.

and then finally the session 4. about user perspective.

It was interesting to be in a ‘user track’ as also library as such as users as all digitisation is done to support the public and the researchers. But in a way it was very accurate as we would very much like to use existing research, methods and datasets and just find ways how to efficiently integrate them to our own internal workflows.

Digitisation and Digital Library Presentation System – Sheet Music to the Mix Tuula Pääkkönen, Jukka Kervinen, Kimmo Kettunen Our talk to explain initial concept how enrichment workflow could operate with sheet music and how to integrate “locating” of sheet music to the existing search of the contents of the digital materials.
Music Search Engine from Noisy OMR Data by Sanu Pulimootil Achankunju An state-of-the art description how Bavarian State Library has enabling improved search and use of famous German composers.
How can Machine Learning make Optical Music Recognition more relevant for practicing musicians? by Heinz Roggenkemper, Ryan Roggenkemper An eye-opening talk about the need of musicians. The OMR, pitch recognition should be practically perfect to be fully useful.


Final thoughts

All in all very good workshop. In the side discussion with some of the participants we got also hints about possible datasets and code to use, so thank you everyone for those. As one interesting topic also the IIIF protocol popped up in the final wrap-up as there was just recently created an AV module for it, which enables the vision that was explained in our talk – having everything integrated, the music sheet, the tracking of played notes and even video of having someone playing that said note. IIIF is turning up to be very potential platform, which makes it possible to connect several libraries materials via one common “pipeline”. Existing digitizations get new push via it.




[1] Choi, K.-Y., Coüasnon, B., Ricquebourg, Yann, & Zanibbi, R. (2018). Music Symbol Detection with Faster R-CNN Using Synthetic Annotations. In Calvo-Zaragoza, Jorge, J. H. jr, & Pacha, Alexander (Eds.), Proceedings of the 1st International Workshop on Reading Music Systems (pp. 9–10). Paris.

ISMIR2018 – tieteellinen ohjelma

ISMIR:n tutoriaalipäivän jälkeen konferenssin tieteellinen ohjelma alkoi. Hyväksymisprosentti konferenssiin oli n. 43% , joka kertoman mukaan oli samaa luokkaa, kuin aiemmissa konfrensseissa. Tämä konferenssi oli 19. kerta kun se järjestettiin ja olipa ainakin yksi musiikin käsittelyn ja erityisesti OMR:n kanssa työskennellyt henkilö ollut paikalla kaikissa konfferensseissa. ISMIR:ssä myös osallistujamaita oli Euroopasta, Aasiasta ja Amerikoista. Osallistujamaista oli myös liki samoilla osuuksilla läpimenneiä esityksiä. Muutama maa, erityisesti Saksan ja Espanjan pari yliopistoa oli tutkimuksen kärjessä sillä niistä oli useampikin esitys, mitä esitysten logoista kerkesi huomaamaan.

Konferenssin rakenne

Kaikki konfferenssipaperit löytyvät verkosta, ja kaikki ovat käyneet tuplasokkokatselmoinnin jossa oli mukana kolme katselmoijaa ja yksi järjestäjien edustaja. Kiinnostavaa oli, että paperit esiteltiin neljän minuutin pikapuheenvuoroissa ja lisäksi samat esitykset olivat julistesessiossa, jossa sitten pystyi kysymään tutkimuksesta suoraan tekijällä. Pikapuheenvuorot lienevätkin yleistymässä, sillä näinhän toimittiin jo viime vuoden Heldig Summitissa. Osallistujat pysyivät hyvin aikataulussa, sillä kello oli kaikilla näkyvissä ja ajan ollessa “pykälässä” puhuja sai koneelliset aplodit, joka oli viimeistään merkki, että oli aika lopettaa.


Konferenssin sisällöistä

Tarvitsisi koneoppimista, että saisi tiivistettyä kaiken mitä eri tutkimuksissa oli työn alla. Yleishuomiona voisi ehkä sanoa, että neuroverkkoja käytettiin varmaankin kaikissa, joko luokittelussa tai klusteroinnissa. Toinen huomio jonka voi tehdä oli, että ideana oli manipuloida musiikkia , nuotteja, ääniä eri tavoin ja kylläpä niitä vaihtoehtoja löytyi. Digitoiduista nuoteista tehtiin nuottien tunnistusta (optical music recognition, OMR), jossa oli osittain samoja tuttuja ongelmia kuin tekstintunnistuksessa (OCR), mutta osittain musiikissa ongelmat olivat moninaisemmat sillä tahti ja rytmi, esitystapa, musiikin kokemus olivat muita näkökulmia, joita luotiin. Kuitenkin tuntui, että kirjastoväkeä paikalla oli vain pari yksittäistä, joten kirjastopuolella digitaalinen musiikin käsittely lienee alussa (ainakin tämän konferenssin perusteella). Saksassa oltiin tehty nuottien hakua (s.23) siten, että kysyjä voi antaa tiettyjä nuotteja ja haku etsii kyseisiä kokoelmista ja Englannissa keskiaikaisten nuottien kuvahakua, mutta tämä ei liene vielä päätynyt heillä esitysjärjestelmään asti. Kiinnostavaa oli, että BLAST ja biotieteiden menetelmät joita myös Kansalliskirjaston COMHIS-projektin Turun yliopiston toteuttajapäässä on käytettävänä, tuli myös esille tässä, joten eri puolilla päädytään hiljalleen samoihin tutkimussuuntiin. Samaa lie odotettavissa myös NewsEYE-projektin suunnalta, koska IIIF-rajapinta vilahteli myös muutamassa esityksessä, potentiaalisena ratkaisuna – näimmepä jopa OMR:n gurulta ex tempore-esityksen jossa hän esitteli IIIF-serveriä, joka pystyi juusi havainnollistamaan nuotteja, soittajaa yhtäaikaa samassa verkkojärjestelmässä.

Hetkosen vilahti jo mielessä, että OMR ja musiikintiedeonhaun ongelmat on kaikki jo ratkaistu, mutta tätä toki tutoriaaleissa toistettiin, että tutkimussarkaa vielä on. Aina eri aineistoista voi tulla uusia tutkimusaiheita ja aina tutkijat pyrkivät löytämään aina parempia algoritmeja, joilla aineistoja käsitellä entistä paremmin. Vaikka joitakin järjestelmiä oli, jotka toimivat joillakin saroilla ja osuuksissa, siinä on kuitenkin tekemistä opettaa neuroverkot omalle aineistoille ja löytää malleille parametrit. Aina kuitenkin jäi ihmisille tekemistä vaikka koneistakin on paljon apua.Ja koneetkin vaativat paljon ihmistyötä sekä aineiston seulonnassa kunnollisen opetusaineiston tekemistä varten tahi annotoidun aineistojoukon tekoon.

ISMIR2018/OpenSource and reproducible MIR Research

ORMR was the topic of Sunday’s tutorial . There Brian McFee and Thor Kell went through good coding basics, git, python, CI, tests and flake8 in order to show participants how to make code, which yourself can read after summer holidays, but which would be something that others can pick up.

Tutorial 23.9. 9-12.30

There was quite similar tenet as in the earlier WORMS workshop from the researcher point of view. Utilize version control , document stuff, and explain things. As an avid user of many github repositories and tools shared there both of these talks were important to have – at least if you can influence the newcomers coming to the field, then the future will be brighter or at least more open.  Even if some of the materials would be familiar, do peek it at :  , the material was nicely structured and the pace was nice for the most of the participants. For fast-goers, the teachers reminded of option to go through materials beforehand, so there was something to do while waiting for others. As a handy practical thing, the organizers controlled time spent via two sticky notes: blue : if the checkpoint was completed, and red if there was some issues or questions. With two teachers it worked quite well. Only nag was that they didn’t mention the laptop requirement beforehand, so couple of people got also the pair programming experiment.

But as in one after discussion was told to me – do not just pick up a python module from internet, but do read the code and attempt to understand how it works. In the long run it will pay up as then when the hard times come ahead you have options to tweak system further to the desired direction.

Lehti- ja data-aineistoista meiltä ja muualta


Ruotsissa historiallisten aineistojen yleinen saatavuus kasvaa, kun siellä päästään digitoimaan loputkin välin 1734-1906 välin lehdistä.  Projekti laajentaa Ruotsin Kungliga Biblioteketin ja Ruotsin Kansallisarkiston aineistoja myös paikallislehtiin. Digitiointi on ollut tähän asti enemmän suurissa kaupunkien lehdissä, jolloin paikallisten uutislehtien osuus on ollut pienempi. Voikin miettiä tutkimuksia lukiessaan missä jutussa mainittu lehti onkaan julkaistu ja onko vaikka toiselta puolen Suomea olemassa vastaavanlaista tutkimusta. KB:n digitoinnin tekee mahdolliseksi Arcadia tutkimusinstituutin mittava lahjoitus, joka varmasti parantaa aineistojen saatavuutta ja tutkimuskäyttöä Ruotsissa ja ehkä jopa Pohjoismaissakin mittavasti. Suuret korpukset palvelevat tutkimusta ja erilaisia tutkimusaloja monipuolisesti, mahdollistaen keskittymisen joko tiettyyn nimekkeeseen, alueeseen tai jopa kaikkeen aineistoon ja ovat laajuudessaan yleiskäyttöisiä.

Datasettien haku

Uusi “datamaailma” on myös heräämässä, sillä nyt myös Google on lisännyt aiempien data-aineistojen portaalien lisäksi oman datasettien hakupalvelun . Hakupalvelu ei vielä sisällä kaikkea vaan esimerkiksi suomalaista materiaalia etsiessä, lähteeksi mainitaan  European data portal-palvelu, tai esim. CEICData. Onkin kiinnostavaa nähdä löytävätkö eri ground truth aineistot, joita käytetään algoritmien vertailussa ja voisikin epäillä, että  Suomessa tai Etsin-palvelujen datasetit päätyvät aikanaan Googlen palveluun, Googlen tekemien ohjeiden avulla. Datalle löytyy monia säilytyspaikkoja, joten jotakin dataa voi joutua etsimään useampiakin paikkoja, joten palvelulle voi löytyä uskollisia käyttäjiä.


“– voisi julkisin varoin ylläpidetty julkaisuarkisto olla aivan yhtä kelvollinen primäärinen julkaisupaikka kuin tieteellinen lehtikin”

kirjoittaa Heidi Laine tekstissään, jossa katsotaan avoimen julkaisemisen Plan S-raporttia, johon eurooppalaisten tutkimusrahoittajien yhteistyöelin on määritellyt pitkän tähtäimen tavoitteita tieteellisten julkaisujen julkaisemiseksi.  Julkaisuarkisto ei kuitenkaan veisi pois katselmointivaihetta ja esim. vaikka versiohallintaa (Github) voisi käyttää katselmointiin aivan hyvin, ja tuoreena kokemus OpenReview-palvelusta oli miellyttävä, kun artikkeliteksti oli näkyvillä, ja kommentit sai siihen kommentteina, jotka näkyivät kaikille, kuin myös omat vastaukset ja uudet versiot artikkeliehdotuksesta. Tosin toisaalta nyt kentällä on paljon irtopalloja, joten ehkä aluksi hyvä on avoin keskustelu ja ajatusten vaihto jonka myötä voidaan hiljalleen löytää yhteisesti uusia toimintamalleja.

Digitalia at summer school of document analysis & recognition (SSDA2018-TC10/TC11)

Digitalia project participated to the #SSDA2018 conference, where aim was to gather up researchers who work with document analysis of various materials, and this time the seminar consisted of roughly 40 people, from all over the world.

Summer school was started with Jean-Marc Ogier, who set the table with interesting examples of past and present in document analysis and how the change in media cause new research fields to appear. From digitized materials to born-digital, from physical signatures to online signatures and so forth, which enable also capturing new information. presented the International Association for Pattern Recognition (IAPR) working groups. There IAPR TC-10 is Technical Committee on Graphics Recognition  and IAPR TC-11 is about Reading Systems. Finland is part of the IAPR via the Pattern Recognition Society of Finland.

There was also an introduction round to the L3i laboratories, where various researchers gave us bit of time and presented their work via demos. We got a glimpse of equipment that was used in the lab, and various research done at the lab.

One demo was about segmentation and text capturing of comics , i.e. recognition of individual panels and then the text and also characters who is speaking in the comic. The material for which it was developed were various online comics and some digitized ones. It was patterns all the way, first detecting contours of the panel, then the characters, and also the speech balloons, which were also then processed in order to get the text from the balloons.

Second demo was about transcribing video, the system picked 1 frame from each second of the given input video, and it then transcribed both the content on the captured image and also the audio. Then with help of search engine it was possible to find certain text just via search.

One demo was done as an experiment to monitor the traffic in the harbor. The system detected the moving ships coming or leaving : it detected the boat from the moving image and thusly enabled creation of statistics, how many boats arrived and how many left. This worked in real-time, highlighting boat in the video feed, but also mapping them to the map of the harbor area, which could be very useful in a dashboard. Second live video feed system recognized faces, and aimed to capture sentiments, for example : surprised, content . There were 6 basic feelings, which the system gave ‘bars’ based on the probability.

Bart Lamiroy from University of Lorain talked about familiar concern “Honest, Reproduciple Reporting of Results is Difficult … an Analysis and some Good Practices” about how to do good research and how to spot it when you see it. There the familiar issues, which software, various modules and how fast things change, make it harder for the next researcher to reproduce the results – there are many tiny details, which the publication might not include and there also the tenet of “publish and perish” can create pressures for the researchers, which make them unable to use enough time to documentation, which could be invaluable for the next reader.

Andreas Fischer from University of Fribourg, Switzerland then took the participants to the structural methods of handwriting analysis. Their research team had worked e.g. with George Washington’s letters from Library of Congress. In the Lab the Diva Services attempted to create a platform where a researcher could give data/methods for other researchers to use, thus easing the amount of software needed to be installed to own machine. Diva Services enable processing remotely via specified APIs, first researcher can upload documents (i.e. page images) to the service and then call API of a method and to get results via that api. Very nifty idea of a system and one more implementation for these analysis systems for researchers. Currently can be used for small experiments as the service is currently powered by one machine at the moment. Anyhow enough so that one participant could use its methods to distinct between different Japanese script characters, which in the end won her the excellence award of the summer school.

Poster session

Poster session was also eye-opening – so different kinds of problems where machine learning was used when possible. There were OCR issues, very fragile materials like palm leaves, script and languages for which there does not even exist unicode fonts. Also team sizes varied, some were alone in their university, but then in one lab there was 25 people doing research. In any case, hearing about research done elsewhere was interesting, and some things need to be followed later on, too.


Deep Neural Networks and data

Seiichi Uchida from Kyushu University (Japan) walked us in to the deep neural networks in document analysis and actively promoted the thinking to go beyond 100% accuracy. His slides can be viewed from below:


Vincent Poulain-d’Andecy from YOOZ (France) gave the industrial talk with multitude of examples of all kinds of document types and formats we actually live with. For example IAPR’s TC10 and TC11 has a number of datasets, which researchers can use as a baseline to check the capabilities of own algorithm, against given data set , thusly giving a joint baseline and benchmark which can help in evaluating how well a specific algorithm works.

David Doermann from University of Buffalo (USA) talked about the Document Forensics in document analysis, and about the current situation, where images, videos and audio can be created in such a way, that a real thing can be impossible to distinct from a fake. He defined the terminology, and gave various examples from different centuries how forgeries and tamperings have been tried, and on the other hand how they are and were being prevented. A comporting thought anyway was that :

“It only takes one piece of inconsistent evidence to prove something is not authentic”


He highlighted also how unique each persons handwriting actually is, as each writer has hard time to change ingrained habits, as he mentioned that letter design, spacing and proportions rarely change, and for untrue cases, there are properties which experts can follow.

Marçal Rusiñol from CVC, Universitat Autònoma de Barcelona,Spain told the details of Large-scale document indexing, going bit deeper how the indexing actually works. He had good points on how to make things work to the end-user, i.e. how to improve human computer interaction, and how to rank documents about semantic similarity for a query which goes bit beyond just the search word given by the user.

Dimosthenis Karatzas from CVC, Universitat Autònoma de Barcelona,Spain then talked about urban world and the scene text understanding – how to detect text from noisy and vibrant environment, which is important as he said that over 50% of urban imaginery contains text in some form.  Turned out that trick is in a way in a context, text has some typical features like alignment and characters, which make them distinct even if the observer would not necessarily know the language in question. Many machine-learning algorithms are suitable for the word and object spotting, but the desire was to get to higher level to the true information spotting. There the idea is that machine could answer questions like “what is the price of a 1 litre of milk” from any shop. For this they had generated simulated 3D traffic environment where the autonomous cars can be immersed to various weather, and traffic simulations to test it and now the idea was to create simulated “supermarket” where before mentioned information spotting could be trained upon.

Jean-Yves Ramel, Lifat, University of Tours (France)  talk was titled “Interactive approaches and techniques for Document Image Analysis” compared the computer vision and the document image analysis problems and solutions, which turned out to have quite a many similarities. He highlighted the service aspect of a tool or algorithm, it would be beneficial to let domain experts to fine-tune the algorithms and define parameters, which are meaningful in certain context. By introducing the user, which in fact is the other researcher, would enable them to do more, in a limited time, with a given dataset. In short, researcher should be part of the defining of services which are meant for them to ensure that the features and data is available as needed.

Jean-Christophe Burie from University of La Rochelle kept then the last talk and a workshop where the idea was to experiment with comics, and their segmentation to the speech bubbles. This was one of the demos of the tour of the L3i laboratory of the university so it was interesting to experiment with it even for a short while. Python, opencv, were in use for the image segmentation, and Tesseract was used for then extracting the text part from the speech balloons in the comics.