Multilingual Task

This task is a continuation of the 2012 CHiC lab, using similar task scenarios, but requiring multilingual retrieval and results. This task has a predefined and fine-tuned set of requirements from last year's lab, but will assessall runs against the multilingual collection.

The CHiC Europeana collection will be re-used from last year (only the 13 language-based sub-collections, not the "other" collection!). The combination of 13 language-based sub-collections will form the multilingual collection. A detailed description of the collection can be found here: http://www.promise-noe.eu/chic-2013/collections

A challenge in this task will be to perform relevance assessments on a multilingual document corpus. Participants are asked to submit monolingual experiments and multilingual experiments or experiments that use only parts of the full multilingual collection.

The following languages will be available in the multilingual collection: Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Polish, Slovenian, Spanish, and Swedish.

Two subtasks will be offered:

Multilingual ad-hoc retrieval

Task definition: This task is a standard ad-hoc retrieval task, which measures information retrieval effectiveness with respect to user input in the form of queries. No further user-system interaction is assumed although automatic blind feedback or query expansion mechanisms are allowed to improve the system ranking. The ad-hoc setting is the standard setting for an information retrieval system - without prior knowledge about the user need or context, the system is required to produce a relevance-ranked list of documents based entirely on the query and the features of the collection documents.

Requirement for submission: Participants are expected to submit relevance-ranked result lists for all 50 topics in TREC-style format. Participants will be asked to submit at least two multilingual runs (using as many collection languages as possible, optimally all 13, minimally 2). While multilingual retrieval is the focus of this year's track, participants will also be asked to submit at least two monolingual runs in any of the 13 sub-collection languages. The monolingual runs should be in different sub-collection languages. (These runs will be used mainly for pooling and relevance assessments.)

In the multilingual runs, the focus is NOT on topic translation: the organizers will provide topic translations for all 13 sub-collection languages. This means that your submitted runs should not involve any query translation experiments; please use the official topic translations if you want to use multilingual topics. However, you can, of course, use the multilingual topic set to test your query translation algorithms in any additional experiments you wish to run.

Topics: The topic set for the 2013 edition consists of 50 topics sampled both from Europeana logs and intellectually developed. We down-sampled the proportion of named-entity topics in this year's topic set (from 60% in 2012 to about 20% this year) to put more emphasis on the more difficult topical queries. For comparison purposes, we also included topics from CHiC 2012. All 50 topics for CHiC multilingual ad-hoc will be translated into all 13 sub-collection languages. The topics from last year can be viewed here: http://www.promise-noe.eu/topics

There are no formal requirements for which topic language(s) to use in the multilingual experiments as long experiments are run on at least two of the sub-collection languages combined. This could include (but is not limited to):

(1) picking a single topic language and running this against two or more sub-collection indexes (e.g. using the German topics to run against the German, Italian and English sub-collections),

(2) merging two or more monolingual runs (using the respective topic languages) to produce multilingual results lists (e.g. using the German topics to run against the German collection and the Finnish topics to run against the Finnish collection and then to merge them), or

(3) running a combination of two or more topic languages against a single index containing all sub-collections (e.g. combining all 13 topic languages to run against a combination of all 13 sub-collections).

Relevance assessments: Documents will be assessed individually by language for relevance according to the query and information need, and then pooled into a multilingual relevance pool. An assumed information need for the query was generated and described collaboratively (which will be used for later editions). Here, the perspective of an average user is assumed—we assume the majority of users typing that particular query would have that particular information).

Evaluation metrics: The evaluation metrics for the ad-hoc task will be the standard information retrieval measures of precision and recall, particularly the standard measure Mean Average Precision (MAP) and precision@k.

More details on submission requirements can be found here:  http://www.promise-noe.eu/chic-2013/guidelines-for-participation-and-submission/multilingual-task#adhoc

Multilingual semantic enrichment

Task definition: The task requires systems to present a ranked list of at most 10 related concepts for a query to semantically enrich the topic and/or guess the user's information need or original query intent. Related concepts can be extracted from Europeana data (internal information) or from other resources in the LOD cloud or other external resources (e.g. Wikipedia). The related concepts should then be weighted (assigning weights between 0 and 1) in order to represent the importance of a concept in query expansion.

Expansion concepts may come from any combination of the 13 sub-collection languages. The original CHiC topics will then be expanded with these enrichment concepts to determine their retrieval effectiveness. Original query terms receive a weight of 1, while the enrichment concepts receive the participant-specified weights.

Europeana already enriches about 30% of its metadata objects with concept names and place (included in the test collection). It uses the following vocabularies for its included semantic enrichments, which can be explored further as well:

  • GeoNames
  • GEMET
  • DBPedia

Semantic enrichment is an important task in information systems with short and therefore ambiguous queries like Europeana, which will support the information retrieval process either interactively (the user is asked for clarification, e.g. "Did you mean?") or automatically (the query is automatically expanded with semantically related concepts to increase the likely search success). For CHiC, this task resembles a typical user interaction, where the system should react to an ambiguous query with a clarification request (or a result output as required in the variability task).

Requirement for submission: Participants are required to submit at least ONE monolingual OR multilingual enrichment file using the CHiC Europeana collection. Monolingual enrichment files contain enrichment concepts in only one of the sub-collection languages; multilingual enrichment files contain enrichment concepts in two or more sub-collection languages. Each file should contain at most 10 related concepts per topic, along with a weight for each term (between 0 and 1) that represents how important the term is as an expansion term for that topic.

In the multilingual runs, the focus is NOT on topic translation: the organizers will provide topic translations for all 13 sub-collection languages which can be used for semantic enrichment.

Topics: The topic set for the 2013 edition consists of 25 topics sampled from Europeana logs, including 10 named entity queries. For comparison purposes, we also included topics from CHiC 2012. The topics for CHiC multilingual semantic enrichments will be in all of the 13 sub-collection languages.

Relevance assessments: Relevance will be assessed in 2 phases:

(1) All submitted enrichments will be assessed manually for use in an interactive query expansion environment (e.g. "does this suggestion make sense with respect to the original query?").

(2) All submitted enrichments will be used to expand the original topics (using the concept weights) and compared with the unexpanded runs, evaluated against the ad-hoc track's relevance assessments.

Evaluation metrics: The evaluation metrics for the semantic enrichment  task will be the standard information retrieval measure of precision (precision@1, @3, @10) for the first phase of assessing the submitted enrichments and the standard ad-hoc information retrieval measures (MAP and precision@k)for the second phase of assessing the submitted enrichments as query expansion variations.

More details on submission requirements can be found here: http://www.promise-noe.eu/chic-2013/guidelines-for-participation-and-submission/multilingual-task#sem