SEW-2009 - Semantic Evaluations: Recent Achievements and Future Directions

A NAACL HLT 2009 Workshop, June 4, 2009, Boulder, Colorado, USA

Main    Call for Papers    Program Committee    Submission Instructions   Program

(November 4, 2008)
PDF version

Workshop Description
The main purpose of this workshop is to review, analyse and discuss
the latest developments in semantic analysis of text. The fact that this workshop happens between the last Semantic Evaluation exercise (SemEval-2007) and the preparation for the next SemEval in 2010, presents an exciting opportunity to discuss practical and foundational aspects of semantic processing of text.  The workshop targets papers describing both semantic processing systems and evaluation exercises, with special attention to foundational issues in both lexical and propositional semantics, including semantic representation and semantic corpus construction problems.

There are now many computer systems that do automatic semantic analysis of text. The purpose of SemEval-2007 was to evaluate the strengths and weaknesses of such systems with respect to different words, relations, types of texts, different varieties of language, and different languages.  This workshop is a follow-up to the SemEval-2007 and Senseval series of workshops on semantic evaluation and a preparation for the next SemEval workshop to be held in 2010.

Senseval-1 included semantic evaluation tasks in English, French and Italian and culminating workshop held at Herstmonceux Castle, Sussex, England in September, 1998. The Senseval-2 evaluations took place in the summer of 2001, and was followed by a workshop held in July 2001 in Toulouse, in conjunction with ACL-2001.  It included tasks for 12 languages. A follow-up workshop on the recent successes and future directions of word sense disambiguation was held at ACL 2002. The Senseval-3 evaluations took place in the spring of 2004, and was followed by a workshop held in July 2004 in Barcelona, in conjunction with ACL-2004. More than 55 teams participated with over 160 systems in its 16 tasks. SemEval-2007 organized 18 tasks, with over 100 teams and 125 systems participating. The results of the tasks and systems were presented in a workshop held in conjunction with ACL-2007.

SemEval-2007 was motivated by a desire to broaden the spectrum of accepted works to all aspects of computational semantic analysis of language. It especially encouraged the proposal of tasks for different languages, cross-lingual tasks, and tasks that are relevant to particular NLP applications such as machine translation, information retrieval and information extraction.

In recent years, the deployment of multiple semantic lexicons and accordingly tagged corpora (WordNet-SemCor, VerbNet-PropBank, FrameNet Prague Dependency Treebank and OntoNotes, to name a few) have radically changed the way semantic analysis is performed, especially at the disambiguation stage. Some of the tasks at SemEval-2007 tried to take this one step further by proposing multiple layers of semantic annotation of the same corpus (e.g., senses, roles, name-entity classes), allowing for novel research to be performed.

SemEval-2007 had a tight schedule, and the results and systems were usually not known until the workshop was held. This fact, and the
growing number of tasks and systems made it difficult to fully analyse the tasks and systems.  The Language Resources and Evaluation journal devoted an special issue to SemEval-2007, attracting 20 submissions. This workshop will also serve as a complementary
follow-up analysis of that evaluation exercise, with special attention to foundational issues in both lexical and propositional semantics,
including semantic representations and semantic corpus construction problems.

In the meantime SemEval-2010 is already rolling, with a public call for task proposals due last October 19, 2008. By the time of this workshop the SemEval panel will have already defined the list of tasks for SemEval-2010, and we expect many of those tasks to be presented in this workshop.

The workshop invites original submissions of papers on systems for semantic processing and evaluable evaluation exercises on semantic processing in general, including, but not limited to, the following topics:
We welcome papers on the above from all theoretical, practical, algorithmic and corpus construction perspectives.

Authors are invited to submit two kinds of papers:
Authors will be able to express their preference for short/long papers but the final decision is on the program chairs. Short papers will be presented in a poster session. Full papers should be up to 8 pages in length, plus one page for references. Short papers should be up to 6 pages in length. Both kinds of papers will appear in the proceedings indistinguishably.

As reviewing will be blind, the paper should not include the authors' names and affiliations. Furthermore, self-references that reveal the author's identity, e.g., "We previously showed (Smith, 1991) ...", should be avoided.

All submissions must be electronic in PDF. Please see the conference website for detailed typesetting specifications. Authors are strongly encouraged to use the LaTeX or Microsoft Word style files available at the conference website (

Submission will be electronic, using the START paper submission website. The START submission page will be available from

Important Dates
March 6, 2009        Paper Submission due
March 30, 2009      Notification of acceptance
April 12, 2009        Camera ready papers due
June 4, 2009           SEW-2009 Workshop

Program chairs:
Eneko Agirre (University of the Basque Country - UPV/EHU, Basque Country)
Lluís Màrquez (Technical University of Catalonia - UPC, Catalonia)
Richard Wicentowski (Swarthmore College, USA)

Address any queries regarding the workshop to:

Program Committee

Last update: April 24, 2009