Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
data:esslli2008:comparison_with_speaker-generated_features [2008/03/07 00:00] 127.0.0.1 external edit |
data:esslli2008:comparison_with_speaker-generated_features [2010/11/01 14:07] (current) |
||
|---|---|---|---|
| Line 22: | Line 22: | ||
| We operationalize the property generation task as follows. | We operationalize the property generation task as follows. | ||
| - | We focus on the same set of 44 concepts used in the [[http:// | + | We focus on the same set of 44 concepts used in the [[:data:esslli2008: |
| For each target concept, we pick the top 10 properties from the McRae norms (ranked | For each target concept, we pick the top 10 properties from the McRae norms (ranked | ||
| Line 64: | Line 64: | ||
| match, and we ignore the lower ones (i.e., lower matches are not treated as | match, and we ignore the lower ones (i.e., lower matches are not treated as | ||
| hits, but they do not contribute to the n-best count either). | hits, but they do not contribute to the n-best count either). | ||
| - | |||
| - | |||
| - | Back to [[data: | ||
| - | |||
| - | |||
| - | |||
| Line 78: | Line 72: | ||
| ==== Gold standard and evaluation script ==== | ==== Gold standard and evaluation script ==== | ||
| - | ** | + | **NB: on March 7, we made a small correction to the property expansion file; if you downloaded the archive before this date, please download it again** |
| - | NB: ON MARCH 7, WE MADE A SMALL CORRECTION TO THE PROPERTY EXPANSION FILE; IF YOU DOWNLOADED THE ARCHIVE BEFORE THIS DATE, PLEASE DOWNLOAD IT AGAIN** | + | |
| - | This {{data:propgen.tar.gz|archive}} contains the gold standard (with property expansions as described above) and an evaluation script that computes average precision at various n-best thresholds. | + | This {{propgen.tar.gz|archive}} contains the gold standard (with property expansions as described above) and an evaluation script that computes average precision at various n-best thresholds. |
| Detailed information about the script can be accessed by running it with the '' | Detailed information about the script can be accessed by running it with the '' | ||
| Line 97: | Line 90: | ||
| We provide this script to have a common benchmark when comparing models, but we also encourage you to explore the McRae et al.'s database for other possible ways to evaluate the models. | We provide this script to have a common benchmark when comparing models, but we also encourage you to explore the McRae et al.'s database for other possible ways to evaluate the models. | ||
| - | |||
| - | Back to [[data: | ||
| - | |||
| - | Back to [[Start]] | ||
