Master Thesis »music Tag Embeddings Using

The Fraunhofer Institute for Digital Media Technology IDMT is part of the Fraunhofer-Gesellschaft. Headquartered in Ilmenau, Germany, the institute is internationally recognized for its expertise in applied electroacoustics and audio engineering, AI-based signal analysis and machine learning, and data privacy and security.

At the headquarters, on the campus of “Technische Universität Ilmenau” researchers work on technologies for robust, trustworthy AI-based analysis and classification of audio and video data. These are used, among other things, to monitor industrial production processes, but also in traffic monitoring or in the media context, for example when it comes to automatic metadata extraction and audio manipulation detection. Another focus is the development of algorithms for the areas of virtual product development, intelligent actuator-sensor systems and audio for the automotive sector.

There are currently around 70 employees working at Fraunhofer IDMT in Ilmenau.

**What you will do**

In recent years, large, general, pre-trained models, which users fine-tune for specific tasks have become the norm in Natural Language Processing. Among these are sentence embedding models (e.g., [1]), which take a sequence of words and output a vector representing the sequence's meaning in high-dimensional space.

The models are trained on huge collections of text, orders of magnitude larger than currently available collections of music. Nonetheless, the creation of a similar semantic space for music is highly desirable. One potential method for doing so is by leveraging the existing language embedding spaces. By taking a large collection of music classification tags (e.g., mood, genre, etc.), and embedding them using the pre-trained sentence embeddings, dimensionality reduction (e.g., Principal Component Analysis) can be applied to capture the relevant dimensions and ignore those which are superfluous to the musical space.

In particular, in this Master's Thesis, the following objectives should be accomplished:
(1) Perform a literature review of approaches towards creating joint music and text embedding spaces, as well as methods for dimensionality reduction.

(2) Download a pre-trained sentence embedding model and configure an environment in which it can successfully run.

(3) Download and configure a method for generating synonyms or paraphrases of a given text (e.g., [2]), as well as methods for textualization of labels.

(4) Collect a set of music labels.

(5) Run the labels (4) through the implemented embedding model (2), using paraphrasing (3) as a data augmentation technique, and perform dimensionality reduction on the resulting vectors, analyzing the resulting space. The analysis should include, for example, clustering as well as an analysis of the distance between labels in the space in terms of established musicological principles (e.g., similarity or dissimilarity between label pairs). For certain label sets, the analysis should additionally include vector arithmetic similar to the word analogy task in NLP (e.g., Rock - Vocals = Instrumental Rock). A comparison should be made to the non-reduced embedding spaces, and different dimensionality reduction techniques should also be compared.

Finally, the student should write a final thesis document, which well-documents their procedure, allowing the approach to be repeated with arbitrary sets of labels.

References:
[1] Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

[2] Zhou, J., & Bhat, S. (2021, November). Paraphrase generation: A survey of the state of the art. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 5075-5086).

**What you bring to the table**

Prerequisites for this topic are good skills in machine learning and deep learning, as well as a passion for (and some knowledge of) music.

**What you can expect**
- exciting market-related topics with complex issues to be solved - you can be actively involved in shaping the future
- challenges at a high level - on top we offer you excellent opportunities for professional and technical trainings
- space to also implement your own ideas, such as in our quarterly open-topic idea contest
- an excellent technical infrastructure
- renowned partners and customers who work closely with you to develop the technologies of tomorrow
- a very good work-life balance thanks to flexible working hours, a co-child office, the option of digital childcare in case of daycare shortages, and the possibility of mobile working, because family comes first - we know that
- an open-minded and interested team, a tolerant and familiar atmosphere as well as regular team events
- good transport connections and proximity to the state capital Erfurt
- attractive special offers as part of Fraunhofer corporate benefits with numerous enterprise partners
- new work and diversity are not just empty buzz