Multimedia/multimodal search engine

The vast majority of search engines (either small or large-scale) are based on search by textual queries. This can be inefficient taking into account the fact that some keywords cannot always represent what the user has in mind. Moreover, “a picture is worth a thousand words”, which means that a query can be better expressed by an image than a set of keywords.

The proposed multimodal search engine enables search and retrieval of relevant multimedia using as query different types of media (either separately or simultaneously). As an example, if the user is searching for the 3D object of a car, he/she can enter as query either text, or a 3D object of a similar car, or an image of a similar car, or the sound of the car, or even all these queries simultaneously. This gives users the freedom to choose which type of query is more convenient and retrieve relevant multimedia results.

Demonstrations

Visit our search engine at:

http://vcl.iti.gr/is/isearch/client/musebag/#

Relevant Projects

isearchI-SEARCH
A UnIfied framework for multimodal content SEARCH

 

Relevant Publications

 P. Daras, S. Manolopoulou, A. Axenopoulos, Search and Retrieval of Rich Media Objects Supporting Multiple Multimodal Queries“, IEEE Transactions on Multimedia, Vol. 14, No. 3, Page(s): 734 – 746, June 2012

P. Daras, A. Axenopoulos, G. Litos, Investigating the Effects of Multiple Factors towards more Accurate 3D Object Retrieval“, IEEE Transactions on Multimedia, Vol. 14, No. 2, Page(s): 374 – 388, April 2012

M. Lazaridis, A. Axenopoulos, D. Rafailidis, P. DarasMultimedia Search and Retrieval using Multimodal Annotation Propagation and Indexing Techniques“, in Elsevier Journal Signal Processing: Image Communication, Special Issue on Image Search and Augmented Reality, April 2012, DOI: http://dx.doi.org/10.1016/j.image.2012.04.001

A. Axenopoulos, P. Daras, S. Malassiotis, V. Croce, M. Lazzaro, J. Etzold, P. Grimm, A. Massari, A. Camurri, T. Steiner, D. Tzovaras, I-SEARCH: a Unified Framework for Multimodal Search and Retrieval“, Future Internet Assembly 2012: From Technological Promises to Reality (FIA Book 2012), Lecture Notes in Computer Science, Volume 7281, DOI: 10.1007/978-3-642-30241-1, pp 130-141, 2012

P. Daras, A. Axenopoulos, V. Darlagiannis, D. Tzovaras, X. Le Bourdon, L. Joyeux, A. Verroust-Blondet, V. Croce, T. Steiner, A. Massari, A. Camurri, S. Morin, A-D. Mezaour, L. Sutton, S. Spiller, Introducing a Unified Framework for Content Object Description“, International Journal of Multimedia Intelligence and Security, Special Issue on “Challenges in Scalable Context Aware Multimedia Computing”, Volume 2, Number 3–4/2011, DOI 10.1504/IJMIS.2011.044765, Pages: 351-375, January 2012

T. Steiner, L. Sutton, S. Spiller, M. Lazzaro, F. Nucci, V. Croce, A. Massari, A. Camurri, A. Verroust-Blondet, L. Joyeux, J. Etzold, P. Grimm, A. Mademlis, S. Malassiotis, P. Daras, A. Axenopoulos, D. Tzovaras, One Size Does Not Fit All – Multimodal Search on Mobile and Desktop Devices with the I-SEARCH Search Engine“, ACM International Conference on Multimedia Retrieval (ICMR 2012), June 5-8, 2012, Hong Kong.

T. Steiner, L. Sutton, S. Spiller, M. Lazzaro, F. S. Nucci, V. C., A. Massari, A. Camurri, A. Verroust-Blondet, L. Joyeux, J., P. Grimm, A. Mademlis, S. Malassiotis, P. Daras, A. Axenopoulos, D. Tzovaras, I-SEARCH – A Multimodal Search Engine based on Rich Unified Content Description (RUCoD)“, World Wide Web 2012, EU projects track, 16-20 April 2012, Lyon, France, accepted for publication.

A. Axenopoulos, S. Manolopoulou, P. DarasOptimizing Multimedia Retrieval using Multimodal Fusion and Relevance Feedback Techniques“, 18th International Conference on MultiMedia Modeling (MMM 2012), January 4-6, 2012, Klagenfurt, Austria.

A. Axenopoulos, S. Manolopoulou, P. DarasMultimodal Search and Retrieval using Manifold Learning and Query Formulation“, ACM International Conference on 3D Web Technology, June 20-22, 2011, Paris, France

A. Axenopoulos, G. Litos, P. Daras3D Model Retrieval using Accurate Pose Estimation and View-based Similarity“, 1st ACM International Conference on Multimedia Retrieval (ICMR2011), April 17-20, Trento, Italy.

A. Axenopoulos, P. Daras3D Object Retrieval: Challenges and Research Directions“, IEEE Communications Society MMTC E-letter, Vol. 6, No. 11, November 2011.

H. Dutagaci, A. Godil, P. Daras, A. Axenopoulos, G. Litos, S. Manolopoulou, K. Goto, T. Yanagimachi, Y. Kurita, S. Kawamura, T. Furuya, R. Ohbuchi, B. Gong, J. Liu, X. Tang, SHREC’11 Track: Generic Shape Retrieval“, 4th Eurographics Workshop on 3D Object Retrieval (3DOR 2011), April 10, 2011, Llandudno (UK).

A. Axenopoulos, P. Daras, D. Tzovaras, Towards the Creation of a Unified Framework for Multimodal Search and Retrieval” 2nd International ICST Conference on User Centric Media – UCMedia 2010, Palma de Mallorca, September 2010