Differences
This shows you the differences between two versions of the page.
data:esslli2008:concrete_noun_categorization [2010/02/08 00:26] schtepf |
data:esslli2008:concrete_noun_categorization [2010/11/01 14:07] |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Task 1.a - Concrete Noun Categorization ====== | ||
- | |||
- | |||
- | |||
- | ==== Introduction ==== | ||
- | |||
- | The goal of the sub-task is to group concrete nouns into semantic categories. | ||
- | |||
- | The {{concnouns.categorization.dataset.txt.gz |data set}} consists of 44 concrete nouns, belonging to 6 semantic categories (four natural and two man-made). The nouns are included in the feature norms described in McRae et al. (2005) (cf. [[comparison_with_speaker-generated_features|Task 3]]). | ||
- | |||
- | |||
- | |||
- | |||
- | |||
- | ==== Task Operationalization ==== | ||
- | |||
- | We operationalize concrete nouns categorization as a clustering task. Since the data set is organized hierarchically, | ||
- | we will run three clustering experiments, | ||
- | |||
- | * **6-way clustering** - models will be tested on their ability to categorize the nouns into the most fine-grained classes of the dataset: //bird// (" | ||
- | * **3-way clustering** - models will be tested on their ability to categorize the nouns into 3 classes supported by robust neuro-cognitive evidence (see, e.g., Caramazza, 2000, "The Organization of Conceptual Knowledge in the Brain", | ||
- | |||
- | * **2-way clustering** - models will be tested on their ability to categorize the nouns into the two top classes: //natural// (superordinate of //animal// and // | ||
- | |||
- | To abstract away from differences stemming from any specific clustering method, you are asked to run your experiments with the //Repeated Bisections// | ||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | |||
- | ==== Task Evaluation ==== | ||
- | |||
- | Evaluation will be carried out in two stages: | ||
- | |||
- | 1. **quantitative evaluation** - results will be evaluated with respect to the two measures for cluster quality available in CLUTO: //purity// and //entropy// (cf. Zhao, Y. and G. Karypis (2002), " | ||
- | |||
- | 2. **qualitative evaluation** - participants will be asked to perform a fine-grained error analysis, focussing on critical nouns, hard classes, etc. ({{qualitativeanalysis.nouncat.zip| recommended qualitative evaluation criteria}}) | ||
- | |||