Topic and Goals
Cultural heritage (CH) content is often multilingual and multimedia (e.g. text, photographs, images, audio recordings, and videos), usually described with metadata in multiple formats and of different levels of complexity. CH institutions have different approaches to managing information and serve diverse user communities, often with specialized needs. The targeted audience for the proposed lab and its task are those content producers that have to cope with the tension between experts (e.g. museum officials) and non-experts (e.g. the general public).
Evaluation approaches (particularly system-oriented evaluation) in this domain have been fragmentary and often non-standardized. Therefore, the CHiC 2012 pilot evaluation lab aims at moving towards a systematic and large-scale evaluation of cultural heritage digital libraries and information access systems.
The lab's goal is to increase our understanding on how to integrate examples from the cultural heritage community into a CLEF-style evaluation framework and how results can be fed back into the CH community.
Data test collections and queries will come from the cultural heritage domain and tasks will contain a mix of conventional system-oriented evaluation scenarios (e.g. ad-hoc retrieval) for comparison with other domains and uniquely customized scenarios for the CH domain (e.g. variability task).