A Stepwise Modeling Approach for Personal Media Semantics

1. A Stepwise Modeling Approach for Individual Media Semantics Annett Mitschick , Klaus Meiner TU Dresden, Department of Computer Science, Multimedia Technology Group Applications of Semantic Technologies (AST' 06)

2. 06.10.2006 AST 2006 2 Outline Introduction Approach: - Semantics for Personal Media Management - Information Instantiation and Integration - Implemented System Architecture Conclusion and Future Work

3. 06.10.2006 AST 2006 3 Introduction Media Management @ Home: - Growing amount of digital assets (text, images, video, audio) complex, high-dimensional - User: non-/semi-professional individual conceptualizations usually minimum efforts to annotate/organize - Tools mostly apply administrative (objective) metadata vs. user interpretation/conceptualization Goal: automated modeling of essential semantic descriptions from existing metadata to ease annotation work

4. 06.10.2006 AST 2006 4 Introduction Solutions: - Automated feature extraction and indexing techniques to manage complexity of huge multimedia collections low-level features S imilarity-based Search (Query-by-Example, etc.) higher-level features Pattern Recognition/Classification (objects, people, genres, etc.) requires prototype features (user context, user feedback) Semantic Gap - Media descriptions based on common metadata standards to enable interoperability & comparability MPEG-7 Multimedia Description Schema RDF/OWL representation for semantic modeling - Semantic Desktop Personal Information Management based on Semantic Web Technologies

5. 06.10.2006 AST 2006 5 Approach :: Semantics for Personal Media Management Knowledge representation - Allow description of background-knowledge (context information like events, activities, actors, locations, etc.) - Allow integration of domain models and individual extensions (e.g. integration of MPEG-7 MDS, FOAF for description of people, etc.) - Manageable, straightforward for non-professionals ABC [Hunter, et al.] - facilitate interoperability between metadata ontologies , conceptual basis for automated mapping , guidance to communities - Base classes to attach domain-specific entities MPEG-7 MDS - Temporal semantics and related actualities ( who, what, when, where ) evolution/transition of objects

6. 06.10.2006 AST 2006 6 Approach :: Semantics for Personal Media Management Acquisition of as much information as possible from content + context - File information, administrative metadata, structural features, etc. Information sources depend on application context - naming, availability/accessibility, semantics, etc. of features/attributes e.g. Date Time Original/Date Created/, Creator/Author/, etc. - properties might be relevant, or redundant in some context e.g. EXIF:ExposureTime and EXIF:ShutterSpeedValue

7. 06.10.2006 AST 2006 7 First step: - Apply sets of mapping rules to construct media-centric description (MDS) - Mapping rules expandable regarding supported metadata schemes ( developer) Approach :: Information Instantiation and Integration Make=FUJIFILM Model=FinePix F601 [(urn:Exif:Make rdf:value ?j) (urn:Exif:Model rdf:value ?k) (urn:Filepath rdf:value ?l) hashCode(?l, ?t) makeTemp(?x) -> (?t mds:creationTool ?x) (?x mds:make ?j) (?x mds:model ?k) (?x rdf:type mds:Device)] FUJIFILM FinePix F601 ... ... FUJIFILM FinePix F601

8. 06.10.2006 AST 2006 8 Approach :: Information Instantiation and Integration Next steps: - Apply derivation rules (schema level) to establish relationships to actualities and temporalities ( developer, user) - Apply classification rules (instance level) to refine description model regarding known pattern ( user)

9. 06.10.2006 AST 2006 9 Approach :: Implemented System Architecture Design rationale: - Automated import and indexing of media items triggered by file system actions - Registry of media type specific analyzing components providing metadata and feature extraction - Processing of semantics and ontology model wrapped by generic model API independent from RDF/OWL storage and querying solution (Jena, Sesame, ) - Extensible, flexible service architecture plug-in based setup, extension at run-time ( OSGi service platform)

10. 06.10.2006 AST 2006 10 Conclusion and Future Work Our approach: - Part of the K-IMM (Knowledge through Intelligent Media Management) project managing private media collections with the help of semantic technologies - Stepwise modeling of resource descriptions to separate distinct problems of construction of media semantics Building objective media descriptions from various sources Deriving subjective, user-oriented semantics Future Work: - Strategies to extend and refine basic rule sets by means of user feedback, context capture - Distribution of modeling steps (Web Services/Agents) - Detailed, comprehensive evaluation

11. 06.10.2006 AST 2006 11 Thank you for your attention! Annett Mitschick annett.mitschick@inf.tu-dresden.de Multimedia Technology Group, Department of Computer Science, TU Dresden http://www-mmt.inf.tu-dresden.de