This is an old revision of the document!

# Data sets for the evaluation of word space models

This page contains a developing list of tasks, sub-tasks and corresponding (sub-)data-sets.

Other tasks or sub-tasks might be added in the near future.

## Ordered by task categories

### Task 1: Free Association

It is tempting to make a connection between the statistical association patterns of words – both first-order associations (collocations) and higher-order associations (word space) – and human free associations – the first words that come to mind when native speakers are presented with a stimulus word. In this task, we will explore to what extent such free associations can be explained and predicted by statistically salient patterns in the linguistic experience of speakers, possibly offering a simple and straightforward cognitive interpretation of distributional similarity (i.e. higher-order association). However, this is not merely a “baseline” task: it also touches on intriguing research problems such as the interaction of first-order and higher-order information in human associative memory.

NB: On March 29th, we fixed a small (but serious) bug in script eval_task3.perl. If you obtained a copy at an earlier time, please download the most recent version of the package and use it for your evaluation.

### Task 2: Categorization

Categorization tasks play a prominent role in cognitive research on concepts. In this type of tasks, subjects are typically asked to assign experimental items - objects, images, words - to a given category or to group together items belonging to the same category. Since categorization presupposes an understanding of the relationship between the items in a category, it is regarded as a key source of evidence on the organization and structure of the human conceptual system.

In the present task, computational models will be tested on their ability to properly group words into semantic categories. The task is organized into three sub-tasks, focussing on different areas of the lexicon and/or semantic dimensions:

### Task 3: Property Generation

The ability to describe a concept in terms of its salient properties is an important feature of human conceptual cognition. In this task, we compare human-generated norms collected by psychologists to the properties generated by computational models.

NB: ON MARCH 7, WE MADE A SMALL CORRECTION TO THE PROPERTY EXPANSION FILE USED FOR THIS TASK; IF YOU DOWNLOADED THE RELEVANT ARCHIVE BEFORE THIS DATE, PLEASE DOWNLOAD IT AGAIN

## Source corpus

You can train your word space on your favorite corpus. However, we also invite you, if this is suitable, to experiment with the ukWaC corpus, so that we will be able to compare different word spaces trained on the same corpus (for information on how to obtain the corpus, write to this address). ukWaC is a very large (about 2 billion tokens) Web-derived corpus. It is split into sub-sections containing randomly chosen documents. Thus, if your algorithm has problems scaling up to 2 billion tokens, you can train it on one or more sub-sections, that will constitute a document-based random sub-sample of ukWaC.