"Hello, what can I search for you?" asks Taylor, the "face" of the AI (artificial intelligence) behind TayloredMedia. She welcomes the user to an audiovisual archive at the very beginning of their search. Audiovisual archives are becoming increasingly important for scientific and journalistic research. In order to archive the enormous amount of video content as part of the cultural heritage and to enable further uses, the content must be described in the best possible way. This requires good and comprehensive keywording of the data. However, due to the increase in the amount of data, manual tagging has been reaching its limits for quite some time. Researchers from our DIGITAL Institute, the University of Applied Sciences St. Pölten and Redlink worked on the "TailoredMedia" project to develop an artificial intelligence to support the keywording of large media archives and the search within them. ORF and the Austrian Media Library were also involved in the project: They defined usage scenarios and provided data.
Automatic keywording and improved search
"The project pursued two related goals: On the one hand, the development of powerful artificial intelligence methods for the automatic tagging of audiovisual content; on the other hand, the user-centered implementation of user interfaces in order to be able to deal with the large, automatically generated amount of data in a targeted manner," explains project coordinator Georg Thallinger from the DIGITAL Institute. The evaluations by the users showed that the project succeeded very well in both areas.
"The biggest challenge for us, and thus also from the point of view of the users, was to control the keywording carried out by the AI. Since an AI 'thinks' in different categories than a human, a common ontology was needed, i.e. a common description logic for the traceability and improvement of the results," reports Peter Judmaier, project manager and deputy head of the Media Computing research group at St. Pölten University of Applied Sciences. To overcome these challenges, the project team developed a personalization of the AI called "Taylor". Taylor involves the users*, explains how results come about and also lets them negotiate with it about the ontology. "This not only makes it possible to understand whether Taylor understands a term correctly, knows it at all, or is in the appropriate media category, but also makes it easy to resolve any misunderstandings. This kind of 'metacommunication' makes the limits of AI understandable to users* and speeds up their research," Judmaier says.
TailoredMedia stands for" Tailored and Agile enrIchment and Linking fOR sEmantic Description of multiMedia" and researches and develops methods for the automatic analysis of audiovisual content based on artificial intelligence (AI). The project was funded by the Austrian Federal Ministry for Climate Protection, Environment, Energy, Mobility, Innovation and Technology (BMK) as part of the "ICT of the Future" program.
Click here for the project website
TailoredMedia
Further Links
