Differences

This shows you the differences between two versions of the page.

Link to this comparison view

data:start [2008/01/17 22:24]
marco
data:start [2010/11/01 14:07]
Line 1: Line 1:
-====== Data sets for the evaluation of word space models ====== 
- 
-This page contains a developing list of tasks, sub-tasks and corresponding (sub-)data-sets. 
- 
-Other tasks or sub-tasks might be added in the near future. 
- 
- 
-===== Ordered by task categories ===== 
- 
- 
- 
- 
- 
-==== Task 1: Categorization ==== 
- 
-Categorization tasks play a prominent role in cognitive research on concepts. In this type of tasks, subjects 
-are typically asked to assign experimental items - objects, images, words - 
-to a given category or to group together items belonging to the same category. 
-Since categorization presupposes an understanding of the relationship between the items in a category, it is regarded as a key source of evidence on the organization and structure of the human conceptual system. 
- 
-In the present task, computational models will be tested on their ability to properly group 
-words into semantic categories. The task is organized into three sub-tasks, focussing on different areas 
-of the lexicon and/or semantic dimensions: 
- 
-  * [[Concrete Nouns Categorization]] 
-  * [[Abstract/Concrete Nouns Discrimination]] 
-  * [[Verb Categorization]] 
- 
- 
- 
- 
-==== Task 2: Free Association ==== 
- 
-It is tempting to make a connection between the **statistical association** patterns of words --  first-order (//collocations//) as well as higher order (//word space//) -- and **human free associations** -- the first words that come to mind when native speakers are presented with a stimulus word.  In this  task, we will explore to what extent such free associations can be explained and predicted by statistically salient patterns in the linguistic experience of speakers, possibly offering a simple and straightforward interpretation of distributional similarity (i.e. higher-order association), in contrast to the symbolic aspects of meaning targeted by the other tasks.  However, this is not merely a "baseline" task: it also touches on intriguing research problems such as the interaction of first-order and higher-order information in human associative memory. 
- 
- 
-  * [[Correlation with Free Association Norms]] 
- 
-==== Task 3: Property Generation ==== 
- 
-The ability to describe a concept in terms of its salient properties is an important feature of human conceptual cognition. In this task, we compare human-generated //norms// collected by psychologists to the properties generated by computational models. 
- 
-  * [[Comparison with Speaker-Generated Features]] (**preliminary gold standard and evaluation script available!**) 
- 
- 
-===== Source corpus ===== 
- 
-While participants are welcome to use models they trained on their corpora, we also invite them, whenever this it suitable, to use the [[http://wacky.sslmit.unibo.it|ukWaC]] corpus (for information on how to obtain the corpus, write to [[mailto:wacky@sslmit.unibo.it]].