"Hello, what can I search for you?" asks Taylor, the "face" of the AI (artificial intelligence) behind TailoredMedia. She welcomes the user right at the start of their search in an audiovisual archive. Audiovisual archives are becoming increasingly important for scientific and journalistic research. In order to archive the enormous amount of video content as part of the cultural heritage and enable further use, the content must be described in the best possible way. This requires good and comprehensive labelling of the data. However, due to the increase in data volume, manual labelling has been reaching its limits for some time now. Researchers at our institute DIGITAL, the St. Pölten UAS and Redlink have been working on the "TailoredMedia" project to develop artificial intelligence to support the keywording of large media archives and searches within them. ORF and the Austrian Media Library were also involved in the project: They defined usage scenarios and provided data.
Automatic keywording and improved search
"The project pursued two related goals: On the one hand, the development of powerful artificial intelligence methods for the automatic keywording of audiovisual content, and on the other, the user-centred implementation of user interfaces in order to be able to deal with the large, automatically generated amount of data in a targeted manner," explains project coordinator Georg Thallinger from the DIGITAL Institute. The evaluations by the users have shown that the project was very successful in both respects.
"The biggest challenge for us, and therefore also from the user's perspective, was controlling the labelling performed by the AI. Since an AI 'thinks' in different categories than a human, a common ontology was needed, i.e. a common description logic for the traceability and improvement of the results," reports Peter Judmaier, project manager and deputy head of the Media Computing research group at St. Pölten UAS. To overcome these challenges, the project team developed a personalisation of the AI called "Taylor". Taylor involves the users, explains how results are obtained and can also be negotiated via the ontology. "This not only makes it possible to understand whether Taylor understands a term correctly, knows it at all or is in the right media category, but any misunderstandings can also be easily resolved. This type of 'metacommunication' makes the limits of AI understandable for users and speeds up their research," says Judmaier.
TailoredMedia stands for "Tailored and Agile enrIchment and Linking fOR sEmantic Description of multiMedia" and researches and develops methods for the automatic analysis of audiovisual content on the basis of artificial intelligence (AI). The project was funded by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) as part of the "ICT of the Future" programme.
Here is the project website
TailoredMedia
Further links
