Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
data:correlation_with_free_association_norms [2008/03/07 10:48]
marco 		data:correlation_with_free_association_norms [2008/06/23 22:19]
schtepf
Line 1: Line 1:
 ====== Correlation of the statistical distribution of words with human free associations ====== ====== Correlation of the statistical distribution of words with human free associations ======
 +
 +
  
  
Line 11: Line 13:
 In the shared task, we wish to find out to what extent free associations can be explained and predicted by statistical association measures computed from corpus data.  The scientific goals of this experiment are twofold: In the shared task, we wish to find out to what extent free associations can be explained and predicted by statistical association measures computed from corpus data.  The scientific goals of this experiment are twofold:
  
-  - **Improve our understanding of free associations.**  In particular, we are interested in the interplay between **first-order and higher-order statistical associations** in human associative memory (e.g. //bear// evokes the hypernym //insect// and //brown//, but //mouse// evokes the compound //mouse trap//).  In future shared tasks, we will also attempt to model the **asymmetry** of many free associations (e.g. //bowler// strongly evokes //hat//, but not vice versa).+  - **Improve our understanding of free associations.**  In particular, we are interested in the interplay between **first-order and higher-order statistical associations** in human associative memory (e.g. //bear// evokes the hypernym //animal// and the property //brown//, but //mouse// evokes the compound //mouse trap//).  In future shared tasks, we will also attempt to model the **asymmetry** of many free associations (e.g. //bowler// strongly evokes //hat//, but not vice versa).
   - **Evaluate free associations as a straightforward "baseline" interpretation of distributional similarity.**  If word space proves to be a good **model of human associative memory**, then we should perhaps focus more on the relation between such free associations and theoretical linguistic categories rather than studying the linguistic aspects of word space models directly.  ((We fully expect a negative answer here, and this is certainly the desirable outcome for many researchers. However, it will be interesting to see how close the relation between word space and associative memory really is.))   - **Evaluate free associations as a straightforward "baseline" interpretation of distributional similarity.**  If word space proves to be a good **model of human associative memory**, then we should perhaps focus more on the relation between such free associations and theoretical linguistic categories rather than studying the linguistic aspects of word space models directly.  ((We fully expect a negative answer here, and this is certainly the desirable outcome for many researchers. However, it will be interesting to see how close the relation between word space and associative memory really is.))
  
Line 17: Line 19:
  
 ===== Data preparation ===== ===== Data preparation =====
 +
  
  
Line 22: Line 25:
  
 Psychologists measure free association with so-called **association norms**:  Native speakers are presented with stimulus words and are asked to write down the first word that comes to mind for each stimulus.  The degree of free association between a stimulus (//S//) and response (//R//) is then quantified by the percentage of test subjects who produced //R// when presented with //S//.  The data sets for this task are based on a large, freely available database of English association norms, the **Edinburgh Associative Thesaurus** ([[http://www.eat.rl.ac.uk/]]). Psychologists measure free association with so-called **association norms**:  Native speakers are presented with stimulus words and are asked to write down the first word that comes to mind for each stimulus.  The degree of free association between a stimulus (//S//) and response (//R//) is then quantified by the percentage of test subjects who produced //R// when presented with //S//.  The data sets for this task are based on a large, freely available database of English association norms, the **Edinburgh Associative Thesaurus** ([[http://www.eat.rl.ac.uk/]]).
-((We also considered using the **USF Free Association Database** ([[http://www.usf.edu/FreeAssociation]]), but it was not suitable for our purposes due to the exclusion of hapax responses.  More information on the USF database can be found in: Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998).  //The University of South Florida word association, rhyme, and word fragment norms.//))+((We also considered using the **USF Free Association Database** ([[http://w3.usf.edu/FreeAssociation]]), but found it more difficult to adapt to our purposes.  One reason is that hapax responses (those generated only by a single subject) were originally excluded from the database and are now available only in separate files with a different format.  More information on the USF database can be found in: Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (1998).  //The University of South Florida word association, rhyme, and word fragment norms.//))
  
   * Kiss, G.R., Armstrong, C., Milroy, R., and Piper, J. (1973).  An associative thesaurus of English and its computer analysis. In Aitken, A.J., Bailey, R.W. and Hamilton-Smith, N. (Eds.), //The Computer and Literary Studies//. Edinburgh: Edinburgh University Press.   * Kiss, G.R., Armstrong, C., Milroy, R., and Piper, J. (1973).  An associative thesaurus of English and its computer analysis. In Aitken, A.J., Bailey, R.W. and Hamilton-Smith, N. (Eds.), //The Computer and Literary Studies//. Edinburgh: Edinburgh University Press.
Line 121: Line 124:
   am.MI       MI (pointwise mutual information) score   am.MI       MI (pointwise mutual information) score
   am.Dice     Dice coefficient association score   am.Dice     Dice coefficient association score
 +
 +
  
 ===== Evaluation ===== ===== Evaluation =====
  
-Since our focus is not on competition, each team will be responsible for evaluating their own model and reporting the results in their paper submission, following the recommendations in the task descriptions above.  Participants are strongly encouraged to make model predictions available for downloads to allow further analysis and discussion by other researchers.+Since our focus is not on competition, each team will be responsible for evaluating their own model and reporting the results in their paper submission, following the recommendations in the task descriptions above.  Participants are strongly encouraged to make the full model output available for download to allow further analysis and discussion by other researchers. 
 + 
 +**NB: bug in script eval_task3.perl fixed as of March 29: if you downloaded earlier, please re-download**  
 + 
 +Evaluation package: {{data:eval_package_free_association.zip}}
  
-We intend to provide [[http://www.r-project.org/|R]] scripts for a basic evaluation of each task (to be made available by end of February). +  * sample output generated by FOO model ((**F**irst-**O**rder associations **O**nly)) 
 +  * sample evaluation scripts written in [[http://www.r-project.org/|R]] and [[http://www.perl.org/|Perl]] 
 +  * includes complete implementation of FOO model

You are here: start » data » correlation_with_free_association_norms