Multimedia extraction tool

This deliverable provides details of the Semantic Multimedia Analysis (SMA) Tool design and operations. Multimedia is extensively used in social networks nowadays and is gaining popularity among the users with the increasing growth in the network capacity, connectivity, and speed. Moreover, affordable prices of data plans, especially mobile data packages, have considerably increased the use of multimedia by different users. This includes terrorists who use social media platforms to promote their ideology and intimidate their adversaries. It is therefore very important to develop automated solutions to semantically analyse given multimedia contents. The SMA Tool is developed to provide following functionalities to the RED-Alert architecture:
1. Separation of audio elements into speech, music and events (such as gunfire, explosions, crowd noises),
2. Extraction of speech audio for input into speech to text engines, and
3. Extraction and identification of image and video scene elements such as logos, flags, weapons, faces.
We conducted a thorough state of the art review of existing multimedia analysis libraries that provide these required functionalities. The results of this tool are sent to the other key components of the project such as Natural Language Processing (NLP) and Complex Event Processing (CEP).
The SMA Tool is currently supporting 8 of the 10 languages identified by the LEAs as high priority. It will support 2 additional languages (Hebrew and Ukrainian) towards the end of 2018. A total of 7 functional requirements of the RED-Alert solution is addressed by the SMA Tool as listed in Section 3. This is a technical document that covers the design details of the SMA Tool and provides its installation details and dependencies.

D2 5 Semantic Multimedia Analysis Tool FINAL