PROMISE NoE &nbps; &nbps;

Guidelines for Participation and Submission

(Note: These guidelines have been adapted from previous CLEF submission guidelines. Please read carefully before submitting results for a task.)

In these Guidelines, we provide information on the CLEF 2012 CHiC queries, data manipulation, query construction and results submission for all 3 CHiC tracks.

CHiC 2012 consists of 3 distinct tasks with different requirements for query and data processing. Tasks can have sub-tasks according to run and language variations. Run variations are described as: topics in language X are searched (à) in document collection Y, e.g. EN àDE means the English queries are used to search the German document collection.
 

Language-dependent subtasks

3 subtasks per task are offered: (i) monolingual (query and document language are the same), (ii) bilingual (query and document languages are different), (iii) multilingual (documents in multiple languages, i.e. the whole Europeana collection will be searched).

The topics (or queries) are offered in 3 languages: English (EN), French (FR) and German (DE). The document collections are offered in English (EN), French (FR), German (DE) and all languages (MUL)

Possible monolingual runs: DE-->DE, EN-->EN, FR-->FR

Possible bilingual runs: X-->DE, X-->EN, X-->FR, whereas X can be one of two topic languages the document language is not in

Possible multilingual runs: X-->MUL, whereas X is either DE, FR or EN

Participants can choose which languages or language pairs they want to use for their experiments. At least one monolingual run in one language is required for participation in a task.

Topic format

Topics are taken from real Europeana query logs and comprise queries for people, places, work titles (e.g. Mona Lisa), events or subjects. Topics have the following format:

<topic lang="en">

<identifier>CHIC-021</identifier>

<title>chardonne</title>

<description>Jacques Chardonne, Writer (FR) OR place in Switzerland</description>

</topic>

Where:

<topic lang="en"> marks the begin and language of a topic.

<identifier> is the query identifier

<title> is the actual query

<description> is a description of the content that will be used for relevance assessment. Not every topic contains text in the description field.

THE DESCRIPTION FIELD MAY NOT BE USED FOR RETRIEVAL EXPERIMENTS.

Ad-hoc task

Goal: To retrieve relevant documents for a given query (a result list of 1000 documents is expected). 50 topics will be provided (CHIC-001 – CHIC-050).

Collection processing: all document fields can be used for retrieval. The collections may not be altered in response to the CHiC 2012 topics, that is, new content may not be added specifically adapted to the topics. Other alterations (e.g. document translation or expansion) that are non-specific to the queries are permitted. Other external resources are also permitted, but must be noted in the run description later.

Conditions for participation: all groups submit at least one base monolingual run in the target language(s) chosen. A maximum number of 4 monolingual runs in one language (i.e. DE) and a maximum number of 4 bilingual runs for one language pair (i.e. ENàDE) and a maximum number of 4 multilingual runs in one language are permitted.

Result format: Results have to be submitted in ASCII format, with one line per document retrieved. The lines have to be formatted as follows:

The fields must be separated by ONE blank and have the following meanings:

1) Query identifier. INPUT MUST BE SORTED NUMERICALLY BY QUERY NUMBER.

2) Query iteration (will be ignored. Please choose "Q0" for all experiments).

3) Document number (content of the <europeana:uri> tag.).

4) Rank 0..n (0 is best matching document. If you retrieve 1000 documents per query, rank will be 0..999, with 0 best and 999 worst). Note that rank starts at 0 (zero) and not 1 (one). MUST BE SORTED IN INCREASING ORDER PER QUERY.

5) RSV value (system specific value that expresses how relevant your system deems a document to be. This is a floating point value. High relevance should be expressed with a high value). If a document D1 is considered more relevant than a document D2, this must be reflected in the fact that RSV1 > RSV2. If RSV1 = RSV2, the documents may be randomly reordered during calculation of the evaluation measures. Please use a decimal point ".", not a comma. Do not use any form of separators for thousands. RSV values must NOT be negative numbers. The only legal characters for the RSV values are 0-9 and the decimal point. MUST BE SORTED IN DECREASING ORDER PER QUERY.

6) Run identifier (please chose an unique ID for each experiment you submit). Only use a-z, A-Z and 0-9. No special characters, accents, etc.

The fields are separated by a single space. The result file contains nothing but lines formatted in the way described above.

You are expected to retrieve 1000 documents per query. An experiment that retrieves a maximum of 1000 documents each for 50 queries therefore produces a file that contains a maximum of 50000 lines.

You should know that the effectiveness measures used in CLEF evaluate the performance of systems at various points of recall. Participants must thus return at most 1000 documents per query in their results. Please note that by its nature, the average precision measure does not penalize systems that return extra irrelevant documents at the bottom of their result lists. Therefore, you will usually want to use the maximum number of allowable documents in your official submissions. If you knowingly retrieved less than 1000 documents for a topic, please take note of that and check your numbers with those reported by the system during the submission.

Submission: Please submit your runs to DIRECT system, which will be opened soon (a username and password will be sent to you). Result files should be uploaded as zip files and validated through the DIRECT system before the final submission. Runs can be deleted or added as necessary.

Variability task

Goal: To retrieve 12 relevant but diverse documents for a given query. Variability or diversity is measured according to media type of object (text, image, audio, video), content provider, query category, field match (which metadata field contains a query term) or other features to be described / suggested by participants, e.g. other query categories. 25 topics will be provided (CHIC-001 – CHIC-025).

Collection processing: all document fields can be used for retrieval. The collections may not be altered in response to the CHiC 2012 topics, that is, new content may not be added specifically adapted to the topics. Other alterations (e.g. document translation or expansion) that are non-specific to the queries are permitted. Other external resources are also permitted, but must be noted in the run description later.

Conditions for participation: all groups submit at least one base monolingual run in the target language(s) chosen. A maximum number of 4 monolingual runs in one language (i.e. DE) and a maximum number of 4 bilingual runs for one language pair (i.e. EN-->DE) and a maximum number of 4 multilingual runs in one language are permitted.

Result format:Results have to be submitted in ASCII format, with one line per document retrieved. The lines have to be formatted as follows:

The fields must be separated by ONE blank and have the following meanings:

1) Query identifier. INPUT MUST BE SORTED NUMERICALLY BY QUERY NUMBER.

2) Query iteration (will be ignored. Please choose "Q0" for all experiments).

3) Document number (content of the <europeana:uri> tag.).

4) Rank 0..n (0 is best matching document. If you retrieve 12 documents per query, rank will be 0..11, with 0 best and 11 worst). Note that rank starts at 0 (zero) and not 1 (one). MUST BE SORTED IN INCREASING ORDER PER QUERY.

5) RSV value (system specific value that expresses how relevant your system deems a document to be. This is a floating point value. High relevance should be expressed with a high value). If a document D1 is considered more relevant than a document D2, this must be reflected in the fact that RSV1 > RSV2. If RSV1 = RSV2, the documents may be randomly reordered during calculation of the evaluation measures. Please use a decimal point ".", not a comma. Do not use any form of separators for thousands. RSV values must NOT be negative numbers. The only legal characters for the RSV values are 0-9 and the decimal point. MUST BE SORTED IN DECREASING ORDER PER QUERY.

6) Run identifier (please chose an unique ID for each experiment you submit). Only use a-z, A-Z and 0-9. No special characters, accents, etc.

The fields are separated by a single space. The result file contains nothing but lines formatted in the way described above.

You are expected to retrieve 12 documents per query. An experiment that retrieves a maximum of 12 documents each for 25 queries therefore produces a file that contains a maximum of 300 lines.

Submission: Please submit your runs to DIRECT system, which will be opened soon (a username and password will be sent to you). Result files should be uploaded as zip files and validated through the DIRECT system before the final submission. Runs can be deleted or added as necessary.

Semantic enrichment task

Goal: To retrieve 10 related concepts (terms or phrases) for a given query to semantically enrich the query and / or guess the user's information need or original query intent.25 topics will be provided (CHIC-001 – CHIC-025).

Collection processing: Theoretically, Europeana collections don't need to be used for this task as query enrichment can also be done with external resources. If the Europeana collections are used for enrichment, all document fields can be used.

Conditions for participation: all groups submit at least one base monolingual run in the target language(s) chosen. A maximum number of 4 monolingual runs in one language (i.e. DE) and a maximum number of 4 bilingual runs for one language pair (i.e. ENàDE) and a maximum number of 4 multilingual runs in one language are permitted.

Result format:

Topic number <tab> term or phrase

Example:

You are expected to present 10 enrichments per query. An experiment that retrieves a maximum of 10 enrichments each for 25 queries therefore produces a file that contains a maximum of 250 lines.

Submission :Please submit your runs to DIRECT system, which will be opened soon (a username and password will be sent to you). Result files should be uploaded as zip files and validated through the DIRECT system before the final submission. Runs can be deleted or added as necessary.

Deadline for submission

DEADLINE FOR RESULT SUBMISSION FOR ALL CHIC TASK IS JULY, 10, 2012 (TUESDAY), 12:00AM CET (MIDNIGHT IN BERLIN).