CoNLL-2005 Shared Task: 

Semantic Role Labeling: Systems & Results

Introduction    F.A.Q.    References    CoNLL Conferences

CoNLL-2005 :       Description&Goal       Examples       Data&Software      Systems&Results 

CoNLL-2004 :      Summary Page (data, systems & results)


The CoNLL-2005 Shared Task on Semantic Role Labeling took place during the period January to May of 2005. Participant systems and results were presented at the CoNLL-2005 conference in Ann Arbor (Michigan, USA) in June 30, 2005.

Nineteen systems participated in the closed challenge. No system was presented at the open challenge. A description of the task, with a discussion of systems and results, can be found below in the introduction paper [CM05]. Furthermore, this webpage includes the papers and outputs of all systems, as well as the slides of the talks presented at the Shared Task session.

Talks at the CoNLL-2005 Shared Task Session


Results

                 +-----------------------+-----------------------+-----------------------+---------------------------+
                 |  Development          |  Test WSJ             |  Test Brown           |  Test WSJ+Brown           |
                 +-----------------------+-----------------------+-----------------------+---------------------------+
                 |  P      R      F1     |  P      R      F1     |  P      R      F1     |  P      R      F1         |
 +---------------+-----------------------+-----------------------+-----------------------+---------------------------+
 | punyakanok    |  80.05  74.83  77.35  |  82.28  76.78  79.44  |  73.38  62.93  67.75  |  81.18  74.92  77.92 ±0.7 |
 | pradhan (*)   |  81.91  75.08  78.34  |  82.95  74.75  78.63  |  74.49  63.30  68.44  |  81.87  73.21  77.30 ±0.7 |
 | haghighi      |  77.66  75.72  76.68  |  79.54  77.39  78.45  |  70.24  65.37  67.71  |  78.34  75.78  77.04 ±0.7 |
 | marquez       |  78.39  75.53  76.93  |  79.55  76.45  77.97  |  70.79  64.35  67.42  |  78.44  74.83  76.59 ±0.7 |
 | pradhan       |  80.90  75.38  78.04  |  81.97  73.27  77.37  |  73.73  61.51  67.07  |  80.93  71.69  76.03 ±0.7 |
 | surdeanu      |  79.14  71.57  75.17  |  80.32  72.95  76.46  |  72.41  59.67  65.42  |  79.35  71.17  75.04 ±0.7 |
 | tsai          |  81.13  72.42  76.53  |  82.77  70.90  76.38  |  73.21  59.49  65.64  |  81.55  69.37  74.97 ±0.6 |
 | che           |  79.65  71.34  75.27  |  80.48  72.79  76.44  |  71.13  59.99  65.09  |  79.30  71.08  74.97 ±0.7 |
 | moschitti     |  74.95  73.10  74.01  |  76.55  75.24  75.89  |  65.92  61.83  63.81  |  75.19  73.45  74.31 ±0.7 |
 | tjongkimsang  |  76.79  70.01  73.24  |  79.03  72.03  75.37  |  70.45  60.13  64.88  |  77.94  70.44  74.00 ±0.7 |
 | yi            |  75.70  69.99  72.73  |  77.51  72.97  75.17  |  67.88  59.03  63.14  |  76.31  71.10  73.61 ±0.7 |
 | ozgencil      |  73.57  71.87  72.71  |  74.66  74.21  74.44  |  65.52  62.93  64.20  |  73.48  72.70  73.09 ±0.8 |
 | johansson     |  73.40  70.85  72.10  |  75.46  73.18  74.30  |  65.17  60.59  62.79  |  74.13  71.50  72.79 ±0.8 |
 | cohn          |  73.51  68.98  71.17  |  75.81  70.58  73.10  |  67.63  60.08  63.63  |  74.76  69.17  71.86 ±0.7 |
 | park          |  72.68  69.16  70.87  |  74.69  70.78  72.68  |  64.58  60.31  62.38  |  73.35  69.37  71.31 ±0.7 |
 | mitsumori     |  71.68  64.93  68.14  |  74.15  68.25  71.08  |  63.24  54.20  58.37  |  72.77  66.37  69.43 ±0.8 |
 | venkatapathy  |  71.88  64.76  68.14  |  73.76  65.52  69.40  |  65.25  55.72  60.11  |  72.66  64.21  68.17 ±0.7 |
 | ponzetto      |  71.82  61.60  66.32  |  75.05  64.81  69.56  |  66.69  52.14  58.52  |  74.02  63.12  68.13 ±0.8 |
 | lin           |  70.11  61.96  65.78  |  71.49  64.67  67.91  |  65.75  52.82  58.58  |  70.80  63.09  66.72 ±0.8 |
 | sutton        |  64.43  63.11  63.76  |  68.57  64.99  66.73  |  62.91  54.85  58.60  |  67.86  63.63  65.68 ±0.8 |
 +---------------+-----------------------+-----------------------+-----------------------+---------------------------+
 | baseline      |  50.00  28.98  36.70  |  51.13  29.16  37.14  |  62.66  33.07  43.30  |  52.58  29.69  37.95 ±0.8 |
 +---------------+-----------------------+-----------------------+-----------------------+---------------------------+

  Results of CoNLL-2005 evaluation, on the four evalutation sets. The WSJ+Brown test is actually a concatenation of 
  two tests sets (WSJ and Brown). 

  Systems are ranked by their performance on the WSJ+Brown test. The significance intervals for the F rates in
  that column have been obtained with bootstrap resampling [Nor89].  F rates outside of these intervals are 
  assumed to be significantly different from the related F rate (p<0.05).

  Systems marked with (*) correspond to post-conference updates.


Post-conference contributions: If you have a system developed in the setting of the CoNLL-2005 Shared Task, and a description of it in an official document (technical report or published paper/article), please contact us at srlconll <at> lsi.upc.edu. We'll be glad to add an entry for your system in the results table.


CoNLL-2005 Introduction Paper:

CoNLL-2005 Papers :

Last Update: September 16, 2005. Xavier Carreras, Lluís Màrquez.