Table of Contents
Task 2c - Verb Categorization
Introduction
The goal of the sub-task is to group verbs into semantic categories.
The data set consists of 45 verbs, belonging to 9 semantic classes. The classification scheme is inspired by P. Vinson & G. Vigliocco (2007), “Semantic Feature Production Norms for a Large Set of Objects and Events”, Behavior Research Methods, which in turn closely follows the classification proposed in Levin (1993).
Task Operationalization
We operationalize verb categorization as a clustering task. Since the data set is organized hierarchically, we will run two clustering experiments, varying the number of classes and consequently their level of generality:
- 9-way clustering - models will be tested on their ability to categorize the verbs into the most fine-grained classes of the dataset: communication ("talk"), mentalState ("know"), motionManner ("run"), motionDirection ("arrive"), changeLocation ("carry"), bodySense ("smell"), bodyAction ("eat"), exchange ("buy"), changeState ("destroy");
- 5-way clustering - models will be tested on their ability to categorize the verbs into 5 classes: cognition (superordinate of communication and mentalState), motion (superordinate of motionManner, motionDirection, changeLocation), body (superordinate of bodySense and bodyAction), exchange, and changeState;
To abstract away from differences stemming from any specific clustering method, you are asked to run your experiments with the Repeated Bisections clustering algorithm available in CLUTO. See the page on the concrete noun categorization task for details.
Task Evaluation
Evaluation will be carried out in two stages:
1. coarse-grained evaluation - results will be evaluated with respect to the standard measures for cluster quality available in CLUTO (again, see the page on the concrete noun categorization task).
2. fine-grained evaluation - verb semantic classification is notoriously hard. Any a priori classification scheme runs the risk of being defied by the highly polysemous and multidimensional character of verbs. In this second stage, evaluation will therefore focus on specific verbs selected as "hard cases", because they are "excentric" members of a given class or they can be classified into more than one class. Participants will be asked to perform a fine-grained error analysis on such verbs ( recommended qualitative evaluation criteria)
