Task 1.a - Concrete Noun Categorization

The goal of the sub-task is to group concrete nouns into semantic categories.

The data set consists of 44 concrete nouns, belonging to 6 semantic categories (four natural and two man-made). The nouns are included in the feature norms described in McRae et al. (2005) (cf. Task 3).

We operationalize concrete nouns categorization as a clustering task. Since the data set is organized hierarchically, we will run three clustering experiments, varying the number of classes and consequently their level of generality:

• 6-way clustering - models will be tested on their ability to categorize the nouns into the most fine-grained classes of the dataset: bird ("peacock"), groundAnimal ("lion"), fruitTree ("cherry"), green ("potato"), tool ("hammer"), vehicle ("car");
• 3-way clustering - models will be tested on their ability to categorize the nouns into 3 classes supported by robust neuro-cognitive evidence (see, e.g., Caramazza, 2000, "The Organization of Conceptual Knowledge in the Brain", in Gazzaniga (ed.): The New Cognitive Neurosciences): animal (superordinate of bird and groundAnimal), vegetable (superordinate of fruitTree and green), artifact (superordinate of tool and vehicle);

• 2-way clustering - models will be tested on their ability to categorize the nouns into the two top classes: natural (superordinate of animal and vegetable) and artifact (superordinate of tool and vehicle)

To abstract away from differences stemming from any specific clustering method, you are asked to run your experiments with the Repeated Bisections clustering algorithm in CLUTO (rbr value of the -clmethod option). In case you can not run CLUTO on your system, the workshop organizers will carry out the clustering for you. In this case, data should be prepared according to a format that will be specified later on. Participants are also invited to experiment with other clustering methods and to compare the results with those obtained with CLUTO.

Evaluation will be carried out in two stages:

1. quantitative evaluation - results will be evaluated with respect to the two measures for cluster quality available in CLUTO: purity and entropy (cf. Zhao, Y. and G. Karypis (2002), "Evaluation of Hierarchical Clustering Algorithms for Document Datasets", in CIKM 2002).

2. qualitative evaluation - participants will be asked to perform a fine-grained error analysis, focussing on critical nouns, hard classes, etc. ( recommended qualitative evaluation criteria)

Back to the the Categorization Task

Back to Start