A Framework for the Integration of Multimedia Data

Hyon Hee Kim, LG Electronics, Seoul, Korea
Seung Soo Park, Department of Computer Science and Engineering, Ewha Womans University, Seoul, Korea
Won Kim, SamSung Electronics, Seoul, Korea

COLUMN

PDF Version

Abstract

In this paper, we propose a unified framework, that we call UniMedia, for semantics-based integration and management of multimedia data. The framework is an adaptation and extension of the federated database technology that has been developed during the past three decades for integrating disparate data sources of alphanumeric data. In particular, we propose a multimedia data model, that we call XML/M, which encompasses diverse types of multimedia data and captures semantic relationships among them. We develop the concept of container objects which cluster relevant multimedia data and customize them according to users’ preferences. Further, we augment the concept of container objects with a version management technique. To validate utility of the UniMedia framework, we have implemented a prototype multimedia news management system.

Keywords: XML-based multimedia data model, container mechanism for multimedia data, version management of multimedia data, multimedia application development framework

1 INTRODUCTION

With recent progress in computing technologies, multimedia data such as images, video clips, animations, graphics, and audio have proliferated over the past several years. Most Web-based applications have been developed including diverse types of multimedia data. Users have begun to expect that multimedia contents should be just as easily accessed as alphanumeric data. They want to see video clips related to text article they read, listen to music contained in a video clip they see, and find relevant photo images that appear in a movie or news video clips. To support such user needs, it is important to provide integrated access to diverse types of multimedia data stored in disparate data sources. However, many multimedia applications today deal with multimedia data from disparate sources separately.

The aim of this paper is to propose an XML-based framework, that we call UniMedia, for semantic integration and management of multimedia data to support the development of a wide variety of multimedia applications. The UniMedia framework is similar to the Chamois framework [8] which has been developed at Ewha Womans University. While the Chamois knowledge engineering framework helps development of enterprise business intelligence applications such as customer-relationship management, electronic commerce, business analystics, and information personalization, the UniMedia framework supports multimedia applications such as news-on-demand applications, e-learning systems, and digital libraries.

The contribution of this paper is in the development (and validation) of the architecture of a federated database system that allows dynamic integration of disparate multimedia data sources, such as database systems, multimedia data servers, multimedia applications, asset management systems, etc. Our UniMedia framework represents an adaptation and extension of the federated database technology that has been established during the past three decades for integrating alphanumeric data from disparate data sources, such as database systems and file systems [7].

The framework is based on a multimedia data model, that we call, XML/M (M for multimedia). XML/M is composed of media objects, relationship objects, and container objects. Media objects describe diverse multimedia contents, and relationship objects maintain explicit relationships among the media objects. Container objects are containers of semantically related media objects. They are customized according to user needs and profiles, and then delivered to the users. Further, the container objects are augmented with version management capabilities to support efficient authoring tasks. We implemented a prototype multimedia news management system, that we call UniMediaNews, to validate utility of the UniMedia framework.

XML-based standards for multimedia data have been defined. MPEG-7 [9] is a standard for describing multimedia content. Semantic descriptions in content description tools are closely related to the XML/M data model. The main differences between the semantic description of MPEG-7 and that of our data model is that a basic object in MPEG-7 is a real-world object such as a person in images or a mountain in a video clip, while a base object in XML/M is a media object such as an image or a video clip. Therefore, while spatio-temporal relationships in MPEG-7 are about constituent objects in a single multimedia object, the semantic relationships in our model focus on relations among diverse multimedia objects. MPEG-21 [10] is expected to become an open standard framework for multimedia delivery and consumption. While multimedia data are integrated into a digital item in a digital item declaration statically in MPEG-21, the UniMedia framework integrates semantically relevant multimedia data dynamically using container objects.

The remainder of this paper is organized as follows: In Section 2, we give an overview of the UniMedia framework. In Sections 3 and 4, we discuss the XML/M data model and container management, two cornerstones of the UniMedia framework. In Section 5, we present the UniMediaNews system for validating the UniMedia framework. In Section 6, we provide concluding remarks.

2 OVERVIEW OF THE UNIMEDIA FRAMEWORK

In this Section, we first present the architecture of the UniMedia framework. Figure 1 illustrates the basic architecture of the UniMedia Framework. The framework is largely composed of three components: multimedia data sources and adapters, a container management component, and a metadata management component. The bottom of Figure 1 depicts heterogeneous multimedia data sources and adapters. Multimedia data sources can be database systems, multimedia data servers, multimedia applications, asset management systems, etc. An adaptor is the intermediary between UniMedia and particular data sources, which does the translation of a query and update statement to naive retrieval and update commands.

Figure 1. Architecture of the UniMedia Framework

The left side of Figure 1 shows the container management component which is composed of a container base object manager and a container object manager. The container base object manager is composed of a container generator and a version manager. The container generator creates container objects and stores them in the repository. The container objects are divided into single-media and multiple-media container objects. The single-media container objects are generated by restructuring and integrating homogeneous media objects, and are classified into image container objects, audio container objects, video container objects, and text container objects. On the other hand, the multiple-media container objects are generated by clustering different types of media objects based on relationships among media objects, and are classified into spatial container objects, temporal container objects, and semantic container objects.

Above the container base object manager, there is a container object manager, which is composed of a container customizer and a version manager. The container customizer customizes the container objects based on users’ profiles. It transforms the customized container objects into a standard XML document or an HTML form using a pre-defined template. The version manager creates and manages versions of container objects through both the container generation and customization processes.

The right side of Figure 1 shows metadata management component. Metadata repository is the central storage area for UniMedia metadata, which provides UniMedia users with information about data sources. For management of the metadata, there are query facilities, security/authorization, and backup/recovery functionalities.

3 THE XML/M DATA MODEL

Due to its semistructured and self-describing characteristics, XML [1] has been widely used as a common data model for integrating heterogeneous data sources. Since XML is a markup language that allows user-defined tags, it is also useful to represent the content of various types of multimedia data. Therefore, we adopt XML as a common multimedia data model. To model multimedia data as objects using XML, we take two main features of the standard object-oriented data model [3]: object identity and object nesting. There are three types of objects in the multimedia content repository: media objects, relationship objects, and container objects.

Media Objects: A media object is the basic unit of multimedia data. For example, an image, an audio stream, and a video clip containing a meaningful scene are each modeled as a media object. Each media object is represented by an XML tree with a unique object identifier. The XML tree does not need a fixed schema, because it is self-describing. An image object might be described by the content of the image and metadata like color and shape, while a video object might be described by the content of the video clip like event, object and so on and metadata like duration and date.

Relationship Objects: A relationship object specifies the relationships among media objects. Managing relationships is one of the key features in many multimedia applications. The object-oriented data model captures the Is-A and Is-Part-of relationships between classes using a class hierarchy and a composition hierarchy, respectively [6]. However, multimedia data have more diverse relationships among media objects. We need to manage the relationship object as a first-class object, because we should be able to capture diverse relationships such as spatial relations, temporal relations, and non-spatio-temporal semantic relations and to represent 1-to-many and many-to-many relations as well as n-ary relations flexibly. While media objects focus on describing their contents, relationship objects are managed independently of media objects.

As with a media object, a relationship object is represented by a tree structure with a unique object identifier for the object. Whereas media objects do not have types for the tree structure, relationship objects do. A tree T is described by SpatialRelation, TemporalRelation, or SemanticRelation and each relation is described by a relation name and one or more participating media objects. A participating object (Pobject) is described by one or more audio objects (Aobject), image objects (Iobject), video objects (Vobject), or text objects (Tobject). Each participating object can be a media object or a container object that is a cluster of media objects. If a relationship object specifies a binary relation, it has two participating objects. According to the XML element order, the first object becomes the source object and the second object is the target object. If a relationship object specifies an n-ary relation, it can have several participating objects. In this case, we do not consider the element order.

Container Objects: In our previous paper [4], we designed a mechanism that can be applied to multimedia data. We have extended the mechanism to container objects. A container object is a cluster of semantically related media objects. There are two types of container objects: single-media and multiple-media container objects. Single-media container objects cluster semantically related homogeneous media objects, whereas multiple-media container objects integrate semantically related heterogeneous media objects. Like a relationship object, a single-media container object has a type for the tree structure. The type for the tree T is composed of two subtrees, and the subtrees describe the contents of the container object with a semantics element and the object identifiers of the participating media objects with Pobject element, respectively. Unlike those of relationship objects, the subtrees do not have fixed types.

Like single-media container objects, a multiple-media container object has a type for tree T. The type for a tree T is composed of two subtrees which describe a target object with TargetObj element and related objects with RelatedObj element, respectively. The TargetObj tree is composed of the object identifier of the target object and a semantics element, while RelatedObj is composed of diverse semantic relationship objects.

4 THE VERSION MANAGER

There is a general consensus that version control is one of the important functions in multimedia applications [2]. In our previous paper [5], we proposed a versioning scheme for efficient partial updates of XML documents. We adapt Chou and Kim’s version model [5] to support a versioning scheme appropriate for UniMedia. The model does not include consideration of differential versioning, that is, keeping only the differences between the original and a version derived from it.

In order to allow partial update of a container object, a container object is transformed into two subtrees, that is, one to be changed (versionable subtree) and the other not to be changed (non-versionable subtree). Our approach is to share the non-versionable subtree and to apply versioning only to the versionable subtree. Version history is represented as version trees. UniMedia users can define an XSLT template for the transformation of a selected container object, and before versioning, execute the XSLT script. Versions of the versionable subtree are stored in the repository, and users can retrieve a version from the repository using XQuery just as ordinary objects are retrieved. If there are several versions that satisfy the query, the latest version is selected as a default. After a version is selected, users construct a full version of a container object by merging the non-versionable subtree and the selected version of the versionable subtree.

5 VALIDATION: A MULTIMEDIA NEWS MANAGEMENT SYSTEM

In order to validate the UniMedia framework, we have developed a multimedia news management system that integrates, updates, customizes, and delivers multimedia news items to customers. A news agency wants to build a multimedia news management system which integrates, manages, customizes, and delivers relevant multimedia news items. It has text articles, photo news, and video news in related database systems or video servers, separately. They want to integrate relevant multimedia news about a gold medalist at the Athens 2004 Olympic Games from the separate data sources, and to deliver the integrated multimedia news to different types of users such as newspapers, portal sites or groups of individual users.

Figure 2 shows an example customized container object. The video clip played in Figure 2 is a target object, a video clip containing an interview scene for an Olympic gold medalist, “Dae Sung Moon”, and the target object has three related objects, Image_C_03, Video_O_03, and Text_O_13. Image_C_03 consists of 5 photos of the athlete, and Video_O_03 is a video clip containing the athlete’s final game scene in the Olympics. Text_O_13 is a text article about the athlete’s interview.

Figure 6. An Example of a customized container object

The system is implemented on top of a commercial native XML server, Tamino version 4.1 in Windows 2000 professional environment. To support versioning of container objects, WebDAV server is integrated with a database system. User interfaces are implemented using JavaServer Pages (JSP), and the entire UniMedia framework has been implemented with 3,000 lines of JSP. Using the UniMedia framework, a developer with JSP programming skills can implement this example in three hours.

6 CONCLUSIONS AND FUTURE WORK

In this paper, we presented an XML-based framework for semantics-based integration and management of multimedia data as an adaptation and extension of the federated database technology. The framework is based on the XML/M data model composed of media objects, relationship objects, and container objects. The data model is designed to cluster relevant multimedia data and to customize them according to users’ preferences. Further, the concept of container objects is augmented with a version management model.

While most existing multimedia modeling techniques focus on modeling specific types of multimedia data, our data model unifies different types of multimedia data. Due to the flexibility and self-describing features of our data model, multimedia contents are easily described, and relationship objects capture diverse relationships among different types of media objects explicitly. Container objects integrate semantically relevant media objects, and provide users with rich semantic information about the underlying multimedia contents. To validate utility if our framework, we implemented a multimedia news management system on top of our framework. Our validation system, UniMediaNews, showed that the UniMedia framework enables significant productivity advantages in developing the news management application. The productivity advantages result from three facilities of UniMedia: personalization, semantic support, and versioning.

We are currently enhancing the multimedia semantics with ontology technologies by reasoning about implied relationships among the physical objects appearing in the media objects. Work on XML warehouses or XML-based mediators is still not mature, with many problems remaining to be solved [3]. In the context of UniMedia, performance issues such as indexing techniques and query optimization techniques should be further studied.

ACKNOWLEDGEMENTS

This work was supported in part by the Brain Korea 21 Project of Korean Ministry of Education and done while the first author was a visiting student at the Univervisy of Stuttgart, Stuttgart, Germany. She thanks Prof. Bernhard Mitschang for facility support and useful comments.

REFERENCES

[1] T. Bray, J. Paoli and C. M. Sperberg-McQueen, Extensible Markup Language, http://www.w3c.org/TR/XML

[2] H. Chou and W. Kim, A Unifying Framework for Version Control in a CAD Environment, In Proceedings of the VLDB Conference, pp. 336-344, 1986.

[3] H. Garcia-Molina et al. The TSIMMIS approach to mediation: data models and languages, Journal of Intelligent Information Systems, pp. 117-132, Vol. 8, 1997.

[4] H. H. Kim and S. S. Park, Mediaviews: A Layered View Mechanism for Integrating Multimedia Data, In Proceedings of 9th International Conference on Object-Oriented Information Systems, LNCS 2817, pp. 250-261, 2003.

[5] H. H. Kim and S. S. Park, A Semantics-based Versioning Scheme for Multimedia Data, In Proceedings of DASFAA Conference, LNCS 2973, pp. 277-288, 2004.

[6] W. Kim, Object-Oriented Databases: Definition and Research Directions, IEEE Transactions on Knowledge and Data Engineering, Vol. 2, No. 3, pp. 327-341, 1990.

[7] W. Kim, Modern Database Systems: The Object Model, Interoperability, and Beyond, ACM Press and Addison Wesley, 1995.

[8] W. Kim and et al., The Chamois Component-Based Knowledge Engineering Framework, IEEE Computer, Vol. 35, No. 5, pp. 46-54, 2002.

[9] MPEG-7, http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm

[10] MPEG-21, http://www.chiariglione.org/mpeg/standards/mpeg-21/

About the author

		Won Kim is Senior Advisor at SamSung Electronics, Korea. He is Editor-in-Chief of ACM Transactions on Internet Technology (htttp://www.acm.org/toit), and Chair of ACM Special Interest Group on Knowledge Discovery and Data Mining (http://www.acm.org/sigkdd). He is the recipient of the ACM 2001 Distinguished Service Award. He can be reached at wonkim@austin.rr.com
		Hyon Hee Kim is a senior research engineer in the digital media laboratory at LG Electronics, Seoul, Korea. Her research interests include digital multimedia broadcasting, audio/video codec technologies, and XML. She received a Ph. D degree in computer science and engineering from Ewha Women’s University, Seoul, Korea.
		Seung Soo Park is professor of computer science and engineering. His areas of interest include artificial intelligence, data mining and bioinformatics. He received a Ph.D. degree in computer science from the University of Texas at Austin, USA.

Cite this column as follows: Won Kim: “A Framework for the Integration of Multimedia Data", in Journal of Object Technology, vol. 4, no. 5, July-August 2005, pp. 27-35 http://www.jot.fm/issues/issue_2005_07/column3

Previous column

Next column