\chapter{The AGFL tools}
In this chapter, a number of tools or utilities are described that
are included with the AGFL system because we found them useful
for supporting the processes of developing, testing, modifying,
evaluating and using AGFL grammars and lexica. They are written in
Perl or PROLOG, and can be installed with little effort under both
UNIX and Windows (but it is easier to use them under UNIX).

%Grammar WorkBench (GWB) system
%\cite{DKNZ91,DNS92,NK92,NKDZ92}, a software engineering environment
%which supported the task of grammar development rather than
%conventional program development, with the following tasks:
%\begin{enumerate}
%\item {[} to serve as a graphical user interface to the parser generator
%and the parsers generated {]};
%\item {[} to provide an incremental syntax check in reading a grammar ];
%\item {[} to allow the user to navigate, inspect and modify a grammar,
%using a dedicated editor ];
%\item to perform consistency checks on the grammar;
%\item to compute grammar properties;
%\item {[} to perform profiling of a parser ];
%\item {[} to allow incremental tracing of a parser's execution ];
%\item to generate example sentences;
%\item {[} to assist in performing grammar transformations ].
%\end{enumerate}
%Tasks 4 and 5 are presently performed by the parser generator,
%for task 8 a separate utility esists (see \ref{se:exgen}).
%
%Unfortunately, the GWB could not yet be carried
%along in the course of time for lack of manpower (and mow has to be
%completely re-implemented), but in the mean time we have constructed
%certain utilities that support the more important development
%tasks.

\section{Lexicon Management utilities}
As described in section \ref{lexfreq}, a lexicon consists of
comment lines and lexicon entries, like
\begin{grammar}
#  auxverb TO BE
"am"    to be(sing,first)
"are"   to be(plur,first|third)
"are"   to be(NUMB,secnd)
"is"    to be(sing,third)
\end{grammar}
Although the input is in free format style, it is strongly advised
to put only a single tab after the
terminal symbol and no spaces in the call that serves as Part-Of-Speech
(\X{POS}). This convention ensures that spaces occur only within
terminal symbols, which makes it easy to use UNIX utilities like {\tt
grep} and {\tt sort}. Examples:
\begin{verbatim}
   sort -u noun.dat > result.dat % eliminate duplicate entries
   grep ' ' noun.dat | wc        % count noun collocations
\end{verbatim}
The following example creates a list of all words in  file1,
one per line in file2, where a word is taken to be a maximal
string of letters. Then it shows the fequency count of
the different words in this list:
\begin{grammar}
   tr -cs "[:alpha:]" "[\verb+\+n*]" <file1 >file2
   sort file2 | uniq -c | sort -nr | show
\end{grammar}
It is very helpful in getting a first impression of word frequencies.

A number of simple lexicon management utilities
(Perl scripts) are distributed along with the AGFL system, which
can be used with lexicon files obeying this convention:
\begin{itemize}
\item {\tt subtract.pl file1 file2 > result}\\
    Generates a copy of {\tt file1} in which all lines that start with a
    terminal symbol which also occurs in {\tt file2} have been eliminated
    (\XX{lexicon}{subtraction})
    As an example, removing from the adjectives of the EP4IR grammar
    (see distribution) all forms that are also present participles:
    \begin{verbatim}
       grep VERBG verb.dat > ingform
       subtract.pl adje.dat ingform > result
    \end{verbatim}
\item {\tt intersect.pl file1 file2 > result}\\
    Outputs all lines starting with a terminal symbol that occurs in
    both {\tt file1} and {\tt file2} (\XX{lexicon}{intersection}).
    An example, determining the number of adjectives that are also
    listed as nouns:
    \begin{verbatim}
       intersect.pl adje.dat noun.dat | wc
    \end{verbatim}
\item The \X{sentencizer}, to be found at\\ \label{se:sentenci}
{\tt http://search.cpan.org/~shlomoy/Lingua-EN-Sentence-0.25/lib/Lingua/EN/Sentence.pm},
    serves to cut a running text into sentences. It is particularly
    useful in combination with segment mode parsing, since in this mode
    short paragraphs are parsed much faster than long paragraphs.
\end{itemize}
Together with the technique to generate a lexicon, described in
section \ref{genlexi}, these tools help in making lexicon
construction and maintenance manageable.

\section{The Example Generator}
The Example generator, contributed by Doru Arfire from Rumania,
is found in the distro (ftp site) at {\tt agfl-2.5/utils/generator}.
It can be used with a (correct and complete)
grammar and lexicon to generate random examples of terminal
productions of the root of the grammar -- random sentences, or
examples of simpler constructs.
First the grammar (and eventual lexica) must be compiled
into a PROLOG-program. Then sentences can be generated, by running
this PROLOG program.

Generating examples is a wellknown technique to investigate the
plausibility of the grammar and to get an idea of its liberality
(testing for overgeneration). It is of no help in finding the
reason why some input is unexpectedly not parsed (testing for
undergeneration). As an example, we shall show how to generate
examples from the {\tt meko} grammar (which is included in the
AGFL distribution). We use swi PROLOG, but any other PROLOG system
should also do.

\begin{enumerate}
\item  start the swi-prolog CLI.
\begin{verbatim}
   $swi-pl
\end{verbatim}

\item  compile and load the program
The generator is compiled (sorry for some curious identifiers)
\begin{verbatim}
   ?- ['grammar.pl'].
   %  transducer compiled into transducer 0.00 sec, 10,916 bytes
   % grammar.pl compiled 0.03 sec, 98,280 bytes

   Yes
   ?-
\end{verbatim}
Loading the grammar is done using the predicate {\tt load\_grammar}.
This predicate loads all grammar files included by the main grammar
file, lexicon files, and builds affix domains and lexicon sets.
\begin{verbatim}
   ?- load_grammar('meko.gra').
   Loading grammar from file meko.gra
   The End
   Loading lexicon smallex.dat
   The End
   Building affix domains
   Finished creating lexsets

   Yes
   ?-
\end{verbatim}
\item the predicate {\tt generate\_nonterminal} can now be used to
generate a terminal production for one of the nonterminals of the
grammar. It takes 3 parameters: the name of the nonterminal,
the variable that takes the generated string and a list of affix
values for this nonterminal.

\item Suppose we have the following definitions:
\begin{verbatim}
   NUMB :: sing | plur.
   PERS :: first | second | third.

verbpart(NUMB, PERS): ....
\end{verbatim}
\begin{itemize}
\item
To generate an example without any restriction on the affix values:
\begin{verbatim}
   ?- generate_nonterminal(verbpart, I, ['NUMB', 'PERS']).
\end{verbatim}
\item
In order to generate an example of a {\tt verbpart} of any number
and first or second person we issue the following predicate call:
\begin{verbatim}
   ?- generate_nonterminal(verbpart, I, ['NUMB', [first,second]]).
\end{verbatim}
\item
To generate a singular verbpart for the first or third person:
\begin{verbatim}
   ?- generate_nonterminal(verbpart, I, [[sing],[first,third]]).
\end{verbatim}
\end{itemize}
\item
To generate a complete sentence we may use the predicate
{\tt generate\_nonterminal}
if we know the root nonterminal, or use the predicate
{\tt generate\_sentence}, that takes only one argument,
the variable that will take the generated string.
\begin{verbatim}
   ?- generate_sentence(I).
\end{verbatim}
\item
To control the amount of information displayed during generation we use
a number of globally declared clauses, named attributes. To change the
value of an attribute issue the following call:
\begin{verbatim}
   ?- set_attribute(<Name>, <Value>).
\end{verbatim}
Current attributes values are:
\begin{itemize}
\item {\tt stepbystep} - allows the user to watch closely every step of
the generation process.
It requires the user to press Enter for the next step to occur. The user
can exit the stepbystep mode by pressing space. Values accepted:
true, false.  Default value: false.
\item
{\tt show\_trace} - activates/deactivates the printing of the
generation trace. Accepted values: true, false. Default: true.
\item
{\tt step\_on\_backtrack} - allows the user to specify that, when
a rule can't be satisfied with the current affix values, the generator
should automatically turn into stepbystep mode, so that the process of
backtracking to other alternatives can be inspected.
	Accepted values: true, false. Default: false.
\end{itemize}
\end{enumerate}

\section{Measuring grammar accuracy}
In developing a grammar, an objective measure is needed for the
accuracy of the grammar, in order to do regression testing after
each major modification to the grammar, and to assess the suitability
of the parsers resulting from it for their intended purpose.

We can not use the ubiquitous Bracket Crossing (BC) measure \cite{Mar94}.
To begin with,
it has a large number of weakness and shortcomings \cite{Car99}:
it is not suited for partial parses, only useful for constituency
based parsers, not fine grained enough for some specific syntactic
phenomena, and there is no clear agreement on the granularity of
bracketing. According to \cite{Lin95}, in BC a mis-attachment can
be punished more than once, so that a
shallow parse with less syntactic information scores better than a
``richer" analysis. Furthermore, the bracket-crossing approach needs an
extensive syntactically analyzed corpus, which causes a
chicken-and-egg problem, and it is very much oriented to constituent
trees, whereas we are interested in Dependency Trees.

The Dependency Triples generated from a text represent precisely the
relations that we are interested in, and closely reflect the
semantical dependencies in the text. If we can extract the right Dependency
Tree from a sentence, we can also derive the complete sentence structure.
Therefore we propose to express the accuracy of the grammar/transucer in
terms of the Dependency Triples produced by it when applied to a
reference corpus for which the correct Dependency Triples are known.

This DT-annotated reference corpus is to be based on a collection of
sentences which are representative enough for the intended application
domain and for the syntactic constructions occurring in it.
This collection need not be very big (in comparison with a syntax-annotated
treebank) but the number of Dependency Triples generated must be enopugh
to allow a measurement at the required granularity - say two thousand DT's
if we want 3 decimals of accuracy, just a few hundred sentences.

The manual annotation of the reference corpus with the
correct DT's is tedious and error prone, but it can be performed in
interaction with the parser to be tested, presenting the sentences in
larger or smaller fragments and verifying the results by inspection.
By skillfully exploiting the compositional character of grammar, this
can be done in an efficient and reliable way.

Two persons may mark the
same corpus and discuss the points of difference in order to guarantee
the correctness of the reference corpus.
The measurement procedure is as follows:

\begin{enumerate}
\item generate a parser/trans\-ducer from the grammar and its lexicon
\item collect the reference corpus
\item manually derive the Dependency Triples to be generated from it
\item let the parser generate Dependency Triples from it
\item compare these, computing precision and recall in the usual way.
\end{enumerate}

\subsection{Constructing a Reference Corpus}
We want to evaluate the {\tt npx} parser, operating in line
mode (not in segment mode, see \ref{se:segmode}).
Assume we have a collection of example sentences, one per line, in the
file named {\em examples}, looking like
\begin{verbatim}
this morning I met him in the garden
the vatican has spoken
IBM has been sponsoring a sequence of software engineering conferences
he is asking the president for clemency
unfortunately you're just too late to see him
\end{verbatim}
We first analyse all sentences and transduce them to their Dependency
Trees, retaining the original sentence as a comment, by means of
\begin{verbatim}
   npx -HtP1 -v lexer < examples > examples.dts
\end{verbatim}
The result looks like
\begin{verbatim}
#  0 0--1|this morning I met him in the garden
{P:I ,[SUBJ [V:met ,OBJ P:him |in N:garden ],|TIME this morning ]}

#  1 0--1|the vatican has spoken
{N:vatican ,SUBJ [V:spoken ,]}

#  2 0--1|IBM has been sponsoring a sequence of software engineering conferences
{N:IBM ,SUBJ [V:sponsoring ,OBJ [N:sequence ,|of [N:conferences ,of N:software
engineering ]]]}

#  3 0--1|he is asking the president for clemency
{P:he ,SUBJ [V:asking ,OBJ [N:president ,|for N:clemency ]]}

#  4 0--1|unfortunately you're just too late to see him
{P:you,[PRED [[A:late ,MOD X:too ],MOD X:just ]|SUBJ [V:see ,OBJ P:him ],|MOD
X:unfortunately ]}
\end{verbatim}
Using one of the UNIX-editors, we replace the empty line between examples
by a line containing a control-A character. We unnest the resulting dependency
trees\Y{unnesting} into dependency triples (DT's). The comments retaining
the original sentences pass unchanged through the unnester.
According to the \XX{control-A}{convention} obeyed by the unnester, the parts
between the control-A's are unnested separately (so that each input line is
followed by the DT's generated from it).
\begin{verbatim}
   unnest < example.dts > example.tst
\end{verbatim}
produces the following result:
\begin{verbatim}
#  0 0--1|this morning I met him in the garden
1   [ P:I, SUBJ V:met ]
1   [ SUBJ V:met, TIME this morning ]
1   [ V:met, OBJ P:him ]
1   [ V:met, in N:garden ]
^A
#  1 0--1|the vatican has spoken
1   [ N:vatican, SUBJ V:spoken ]
^A
#  2 0--1|IBM has been sponsoring a sequence of software engineering conferences
1   [ N:IBM, SUBJ V:sponsoring ]
1   [ N:conferences, of N:software engineering ]
1   [ N:sequence, of N:conferences ]
1   [ V:sponsoring, OBJ N:sequence ]
^A
#  3 0--1|he is asking the president for clemency
1   [ N:president, for N:clemency ]
1   [ P:he, SUBJ V:asking ]
1   [ V:asking, OBJ N:president ]
^A
#  4 0--1|unfortunately you're just too late to see him
1   [ A:late, MOD X:just ]
1   [ A:late, MOD X:too ]
1   [ P:you, PRED A:late ]
1   [ P:you, SUBJ V:see ]
1   [ PRED A:late, MOD X:unfortunately ]
1   [ SUBJ V:see, MOD X:unfortunately ]
1   [ V:see, OBJ P:him ]
\end{verbatim}
Now we painstakingly go through this whole testfile correcting erroneous
analyses. This task is best performed by two people separately, who resolve any
discrepancies found by discussion.

We have semi-interactively constructed a \XX{reference}{corpus}, a
\XX{gold}{standard} for measuring the
accuracy of {\tt npx}. This reference corpus may be used, among other things,
to measure the effect of changes (\XX{regression}{testing}).

Two tips:
\begin{itemize}
\item it is possible to use segment parsing instead of line parsing
(testing also the robustness of the parser) but in that case in the file
of examples (see the beginningnof this section) an empty line must be put
after each example
\item the two calls above can be combined via a pipe
\begin{verbatim}
   npx -HtP1 -v lexer < examples | unnest > example.tst
\end{verbatim}
omitting the intermediate file.
\end{itemize}

\subsection{Measuring parser accuracy}
\YY{parser}{accuracy}
Assume that we have a Dependency Parser P and an associated (manually
verified) reference corpus C.
Let DT$_{gold,i}$ be the set of DT's following the i-th sentence in C.
We want to know what is the accuracy of the (current) parser P with respect
to C, expressed in terms of the DT'S produced. Let DT$_{P,i}$ be the set of
DT's produced from the i-th sentence of C by the parser P.

According to typical terminology from Information Retrieval,
\begin{itemize}
\item the \X{Precision} of P\YY{parser}{precision} is measured
as the fraction of the triples produced by P that are correct according to C
\[ \sum_i\frac{|DT_{P,i}\cap DT_{gold,i}|}{|DT_{P,i}|} \]
\item the \X{Recall} of P\YY{parser}{recall} is measured as the
fraction of the triples in DT that are also produced by P
\[ \sum_i\frac{|DT_{P,i}\cap DT_{gold,i}|}{|DT_{gold,i}|} \]
\item as a single figure for measuring the Accuracy, we shall use the
so-called \X{F1-value}, the harmonic means of precision and recall
\[ F_1 = \frac{2}{\frac{1}{Precision} + \frac{1}{Recall}} \]
\end{itemize}
For more explanation, see \cite{Lin95,Kos04,Kos04a}.

For measuring the accuracy we can use the utility {\tt dtcompare.pl},
which is called with as input the reference corpus
\begin{verbatim}
   dtcompare < gold
\end{verbatim}
Assuming for simplicity that the reference corpus {\tt gold} contains only
one sentence together with its DT's
\begin{verbatim}
#  16 0--1|A control circuit switches the various transmission lines so that the
impedance at the input remains balanced no matter how many of the first transmission
lines are in the high impedance state.
 [ A:balanced, MOD X:no matter ]
 [ N:A:many, in N:state ]
 [ N:A:many, of N:transmission lines ]
 [ N:control circuit, SUBJ V:switches ]
 [ N:impedance, PRED A:balanced ]
 [ N:impedance, at N:input ]
 [ N:state, of N:high impedance ]
 [ N:transmission lines, ATTR A:first ]
 [ N:transmission lines, ATTR A:various ]
 [ V:switches, OBJ N:transmission lines ]
\end{verbatim}
the comparison may produce something like
\begin{verbatim}
A control circuit switches the various transmission lines so that the impedance at
the input remains balanced no matter how many of the first transmission lines are
in the high impedance state.

                                       >[ A:balanced, MOD X:no matter ]
                                       >[ N:A:many, in N:state ]
                                       >[ N:A:many, of N:transmission lines ]
 [ N:control circuit, SUBJ V:switches ]=[ N:control circuit, SUBJ V:switches ]
                                       >[ N:impedance, at N:input ]
              [ N:impedance, DET that ]<
                                       >[ N:impedance, PRED A:balanced ]
                                       >[ N:state, of N:high impedance ]
                                       >[ N:transmission lines, ATTR A:first ]
[ N:transmission lines, ATTR A:various =[ N:transmission lines, ATTR A:various ]
               [ V:switches, MOD X:so ]<
[ V:switches, OBJ N:transmission lines =[ V:switches, OBJ N:transmission lines ]
3 common lines; 2 wrong; 7 missed.

Total: 3 common lines;2 wrong;7 missed.
precision = 0.6; recall = 0.3
\end{verbatim}
The left column shows the results of the current parser, the right column the
gold standard, and DT's are aligned for as far as possible. Obviously, this parser
is in bad state.

At the end of the comparison, some statistics over all DT's are given :
\begin{verbatim}
relator total/missed/wrong  precision/recall/F1
  all      12/    7/    2   0.600 / 0.300 / 0.400
 PREP       4/    4/    0   0.000 / 0.000 / 0.000
 ATTR       2/    1/    0   1.000 / 0.500 / 0.667
  MOD       2/    1/    1   0.000 / 0.000 / 0.000
  OBJ       1/    0/    0   1.000 / 1.000 / 1.000
 PRED       1/    1/    0   0.000 / 0.000 / 0.000
 SUBJ       1/    0/    0   1.000 / 1.000 / 1.000
  DET       1/    0/    1   0.000 / 0.000 / 0.000
\end{verbatim}
