|
The IQMT
Framework for Automatic MT Evaluation
|
|
download now ULC v 0.4.3 (Perl)
|
Here you can find information about the IQMT package, an open source Automatic MT Evaluation Framework, released under the GNU Lesser General Public License (LGPL) of the Free Software Foundation. This tool is a joint research effort by the UNED NLP & IR research group and the TALP research center NLP group at UPC.
The IQMT package is based on the QARLA Framework (Amigó et al., ACL'2005). Rather than defining a new supermetric 'XXX' our tool follows a 'divide and conquer' strategy. You can define a set of metrics and then combine them into a single measure of MT quality, in a robust and elegant manner, avoiding scaling problems and metric weightings.
Using IQMT offers a number of advantages over previous MT evaluation packages. First, individual metrics improve their level of correlation with human judgements when they are applied inside QARLA. Second, it permits to avoid the 'metric bias' problem, by allowing you to tune your system on a combination of metrics instead of on a single metric. Third, it allows you to define a set of subtle metrics focusing on partial aspects of MT quality possibly at different linguistic levels, and then combine them into a single measure.
Several well-known and freely available current MT evaluation tools have been incorporated so far:
All these metrics work at the lexical level. Additionally ROUGE and METEOR perform some stemming. METEOR looks up for synonyms in WordNet, too. We plan to incorporate new metrics working at linguistic levels other than lexical overlap in a near future. However, on of the most innovative features of the IQMT package is that user-supplied evaluation metrics can be easily incorporated to the system. Do not hesitate to try your own metrics inside QARLA!
Components
The IQMT package consists of two main components:
(1) IQsetup
The IQsetup component is responsible for applying a given set of
metrics to a given set of translations by different systems. Thus it
requires a set of human references.
(2) IQeval
The IQeval component computes KING / QUEEN / JACK scores on top of the scores
generated by IQsetup. It allows the user to choose the set of
metrics, systems, references and segments to be used.
The user may define these sets in the IQeval config file.
Download now...
application docs scripts
Install
This software requires several Perl packages which can be obtained at the CPAN website:
Moreover, since IQMT may call external software packages, before starting to play with IQMT, if you plan to use these metrics (BLEU, GTM, METEOR, NIST, ROUGE) you first might want to check they work properly.
IQMT Discussion Group
Discussion on features and bugs of this software as well as information
about oncoming updates takes place on the IQMT group, to which
you can subscribe at:
http://groups-beta.google.com/group/IQMT
and post messages at:
IQMT at googlegroups.com
Contributing
The IQMT framework is licensed under LGPL , which means that it may be linked to and used by commercial software packages. But the license also enforces that any changes or improvements made to the library must be redistributed under LGPL terms.
Thus, if you improve the software, either adding new functionalities, fixing bugs, or adding new metrics, you can not distribute them under different conditions than those stated in the license (i.e. freely and with no usage restrictions).
If you want that your changes and improvements become useful to
many other people using this free software, please contact us:
References
Please reference this tool in your academic works citing the following paper: