| Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
data:correlation_with_free_association_norms [2008/03/09 18:44]
schtepf |
data:correlation_with_free_association_norms [2008/06/23 22:19]
schtepf |
| ====== Correlation of the statistical distribution of words with human free associations ====== | ====== Correlation of the statistical distribution of words with human free associations ====== |
| | |
| | |
| |
| |
| In the shared task, we wish to find out to what extent free associations can be explained and predicted by statistical association measures computed from corpus data. The scientific goals of this experiment are twofold: | In the shared task, we wish to find out to what extent free associations can be explained and predicted by statistical association measures computed from corpus data. The scientific goals of this experiment are twofold: |
| |
| - **Improve our understanding of free associations.** In particular, we are interested in the interplay between **first-order and higher-order statistical associations** in human associative memory (e.g. //bear// evokes the hypernym //insect// and //brown//, but //mouse// evokes the compound //mouse trap//). In future shared tasks, we will also attempt to model the **asymmetry** of many free associations (e.g. //bowler// strongly evokes //hat//, but not vice versa). | - **Improve our understanding of free associations.** In particular, we are interested in the interplay between **first-order and higher-order statistical associations** in human associative memory (e.g. //bear// evokes the hypernym //animal// and the property //brown//, but //mouse// evokes the compound //mouse trap//). In future shared tasks, we will also attempt to model the **asymmetry** of many free associations (e.g. //bowler// strongly evokes //hat//, but not vice versa). |
| - **Evaluate free associations as a straightforward "baseline" interpretation of distributional similarity.** If word space proves to be a good **model of human associative memory**, then we should perhaps focus more on the relation between such free associations and theoretical linguistic categories rather than studying the linguistic aspects of word space models directly. ((We fully expect a negative answer here, and this is certainly the desirable outcome for many researchers. However, it will be interesting to see how close the relation between word space and associative memory really is.)) | - **Evaluate free associations as a straightforward "baseline" interpretation of distributional similarity.** If word space proves to be a good **model of human associative memory**, then we should perhaps focus more on the relation between such free associations and theoretical linguistic categories rather than studying the linguistic aspects of word space models directly. ((We fully expect a negative answer here, and this is certainly the desirable outcome for many researchers. However, it will be interesting to see how close the relation between word space and associative memory really is.)) |
| |
| am.MI MI (pointwise mutual information) score | am.MI MI (pointwise mutual information) score |
| am.Dice Dice coefficient association score | am.Dice Dice coefficient association score |
| | |
| | |
| |
| ===== Evaluation ===== | ===== Evaluation ===== |
| |
| Since our focus is not on competition, each team will be responsible for evaluating their own model and reporting the results in their paper submission, following the recommendations in the task descriptions above. Participants are strongly encouraged to make model predictions available for downloads to allow further analysis and discussion by other researchers. | Since our focus is not on competition, each team will be responsible for evaluating their own model and reporting the results in their paper submission, following the recommendations in the task descriptions above. Participants are strongly encouraged to make the full model output available for download to allow further analysis and discussion by other researchers. |
| | |
| | **NB: bug in script eval_task3.perl fixed as of March 29: if you downloaded earlier, please re-download** |
| | |
| | Evaluation package: {{data:eval_package_free_association.zip}} |
| |
| We intend to provide [[http://www.r-project.org/|R]] scripts for a basic evaluation of each task (to be made available by end of February). | * sample output generated by FOO model ((**F**irst-**O**rder associations **O**nly)) |
| | * sample evaluation scripts written in [[http://www.r-project.org/|R]] and [[http://www.perl.org/|Perl]] |
| | * includes complete implementation of FOO model |
