\chapter{Lexicalization and Transduction}\label{ch:inout}
In this chapter the lexicalization process and the transduction mechanism
are described in more detail, with some (hopefully stimulating) examples.

Parsers generated from \agfl\ grammars are intended to be used in widely
differing applications, in particular in:
\begin{itemize}
\item linguistic research, and
\item Information Retrieval applications.
\end{itemize}
Most of the necessary flexibility is provided by the lexical and
transduction mechanisms, and a better understanding of those mechanisms
will make many little {\sc perl} scripts superfluous.

\section{Lexicalization}
By \X{lexicalization} we mean the process of analyzing the input into
terminal symbols from the grammar and the lexicon. An attempt has been
made to make this process so flexible and parametrizeable that in most
cases no preprocessing of the input is needed:
\begin{itemize}
\item No \XX{word}{segmenter} (segmenting the input into words)
is needed -- using the hyphen convention (\S\ref{hyphconv})
the parser is quite capable of segmenting the unformatted input text
into a trellis of (possibly composed) words
\item No \XX{sentence}{segmenter} (segmenting the input into sentences,
while taking abbreviations into account) is needed -- in line mode
(\S\ref{se:linemode}) the parser itself can split the running text into
sentences (but for efficiency reasons segmentation of the running text
may be useful even in segment mode -- see \ref{se:sentenci})
\item No \X{tagger} (assigning POS to words) is needed -- the parser with
its lexicon is the best tagger
\item No separate component for \XX{named entity}{recognition} is
needed -- the syntax of named entities can be described in the grammar,
along with other lexical robustness measures.
\end{itemize}
In the lexicalization phase, the input is split into a trellis of words
(\S\ref{se:words}) separated by terminator characters (\S\ref{se:term}).
The lexical graph (\S\ref{se:lexgraph}) helps in visualizing the result
of the lexicalization process.

The two main application areas of \agfl\ mentioned make different demands
on the parsers, therefore two different styles of input are accepted:
line mode (next section) and segment mode (\S\ref{se:segmode}).

\subsection{Line mode}
\label{se:linemode}
In traditional \XX{linguistic}{applications}, e.g. the syntactic analysis
of corpora, the input consists of sentences, and the goal of the parser
is to find all (or the best $k$) analyses for each sentence.
Penalties may be used to indicate a preference for certain
analyses over others. In this case, the unit of analysis should be
the (complete) {\em sentence}, which is indicated by including in the
grammar the pragmat \XX{line}{mode}.

The input should be presented in a file with one sentence per line.
{\em For each line of input, the parser attempts to find the best analyses
covering it completely}. In case the time for analysing a sentence is
larger than the limit (a \XX{time}{break}), the result consists in the best
parsings found until now.

Line parsing can be effectuated by giving the pragmat
\begin{grammar}
   |line mode|
\end{grammar}
at the head of the grammar. It is the default.

\subsection{Segment parsing}
\label{se:segmode}
In \XX{Information}{Retrieval} applications, the raw input is not nicely
segmented into sentences, and it is to be expected that in many cases the
input is not according to the grammar, or contains unknown words.
A simple segmenter is only a few lines of {\sc perl}, but there are so
many complications (like non-standard abbreviations, missing or spurious
punctuation marks and unreliable capitalization of words) that even the
best segmentation programs achieve little more than 95\% precision on raw
newspaper prose. The best segmenter is in fact the parser -- as long as the
input is syntactically correct. In IR applications, we are interested in
extracting from left to right the largest comprehensible
{\em segments} in a robust way, which
will in many cases not be complete sentences. Furthermore we usually want
only one analysis, viz. the best we can achieve.

For this case the \XX{segment}{mode} was introduced, which
causes the parser to find {\em the best among the longest segments
covering a prefix of the input}; if no segment is found, one word is
skipped. After processing one segment, the parser starts again, on the
rest of the input. A single newline in the input is ignored, so that
segments may extend over more than one line. A multiple newline is seen as
the start of a new segment, and ends the preceding segment. Now the
robust recognition intended can be described by
\begin{alltt}
  | segment mode |
  RULE root: segment.
  RULE segment:
     $PENALTY(50), $MATCH (".*");
     $PENALTY(20), fragment;
     $PENALTY(10), noun phrase;
     sentence.
\end{alltt}
The penalties indicate the quality of the segment.
Notice that the order of the productions for {\tt segment} has been
reversed with respect to the previous example:
the suboptimal analyses are now found
before the optimal ones, so that in case of a time break always at least
one suboptimal analysis has been found - thus achieving \X{robustness}.

Of course, such a strictly left-to-right segmentation may not be optimal
in comparison with a segmentation which may start at \emph{any} point of
the input, but this may be compensated for in the grammar.

\subsection{Terminators}\label{se:term}
The word \Xs{terminator} are characters serving to terminate a word: the
layout characters and certain other characters enumerated in the
\XX{terminator}{string}. The default is
\begin{grammar}
   ' <TAB><NL><CR>,.?;:()!"'
\end{grammar}
The default may be changed by a pragmat.

Some of the terminators are also \XXs{invisible}{character}, meaning that
they will be ignored unless they are matched explicitly as (part of) a
terminal).

\subsection{Words}
\label{se:words}
A lexical element may be
\begin{enumerate}
\item a \X{word}, a string of characters not containing a blank or a
hyphen, e.g. {\tt "not"}
\item a \XX{multi}{word}, a string containing  a number of words with
blanks between them to denote white space, e.g. {\tt "software engineering"}
\item a \XX{part}{word}, a word with a \X{hyphen} {\tt -} at its beginning
or end or both.
\end{enumerate}
Since at any position of the input, more than one of these may be
applicable, the input is not considered as a sequence of lexical elements,
but rather as an ordered and directed graph.

\subsection{The lexical graph}
\label{se:lexgraph}
The result of the lexicalization is the \XX{lexical}{graph} or
\XX{lexical}{trellis}, a graph whose nodes
represent positions in the input and whose arcs are directed and marked
with terminal symbols. Three types of terminal symbols may appear in the
lexical graph:
\begin{enumerate}
\item terminal symbols from the grammar, e.g. {\tt "not"}
\item \XXs{quasi}{terminal} from the grammar, e.g. the argument of
{\tt \$MATCH("[a-z]+")}
\item \XXs{lexical}{terminal} from the lexica
%\item \XXs{tagged}{symbol} from a separate tagger
\end{enumerate}
The lexical graph of the input can be inspected by calling the generated
parser with the option {\tt -G}. An example of a lexical graph, slightly
edited to improve the layout, is:
\begin{verbatim}
1 "Mary" $MATCH("[A-Z][-A-Za-z]*") => 6, "Mary" $MATCH(".*") => 6,
  "mary" NOUN(SING) => 6, "Mary" PNAME => 6
6 "had" $MATCH(".*") => 10, "had" VERBV(FOR|AS, TRAV) => 10, "had"
  VERBV(NONE,TRAV) => 10, "had" VERBP(FOR|AS,TRAV) => 10, "had"
  VERBP(NONE,TRAV) => 10, "had" VERBV(NONE,TRAV,IN) => 10, "had"
  VERBV(NONE,TRAV,OFF) => 10, "had" VERBV(NONE,TRAV,ON) => 10, "had"
  VERBV(NONE,TRAV,OUT) => 10, "had" VERBV(NONE,TRAV,OVER) => 10, "had"
  VERBV(NONE,TRAV,UP) => 10, "had" VERBP(NONE,TRAV,IN) => 10, "had"
  VERBP(NONE,TRAV,OFF) => 10, "had" VERBP(NONE,TRAV,ON) => 10, "had"
  VERBP(NONE,TRAV,OUT) => 10, "had" VERBP(NONE,TRAV,OVER) => 10, "had"
  VERBP(NONE,TRAV,UP) => 10, "had" TO HAVE(numb,pers) => 10
10 "a little" ADVB => 19, "a" $MATCH(".*") => 12, "a" ART(SING) => 12
12 "little" $MATCH(".*") => 19, "little" ADJE(ABSO) => 19, "little"
  ADVB => 19
19 "lamb" $MATCH(".*") => 23, "lamb" NOUN(SING) => 23, "lamb"
  VERBI(NONE,TRAV) => 23
23 <EOS>
\end{verbatim}
The example shows a rather high level of lexical ambiguity. Notice the
many quasi-terminals ({\tt \$MATCH}).

\subsubsection{Example: matching word suffixes}
For the grammar
\begin{alltt}
  RULE sent:
          "quererlo";
          "querer", lo.
  
  RULE lo: "lo"; "-lo".
\end{alltt}
the lexical graph for the input ``quererlo" shows that the arcs passing
a terminator ({\tt =>})
are distinguished from those not passing a terminator ({\tt ->}):
\begin{verbatim}
   1 "quererlo" => 9, "querer" -> 7
   7 "-lo" => 9
   9 <EOS>
\end{verbatim}
Only the latter will be matched against part-words (in this case {\tt
"-lo"}).

In the examples, we have not shown the frequencies: every transition has
also a frequency associated with it (default = 1).

\subsection{Quasi-terminals}
\label{se:quasi}
The \XXs{quasi}{terminal} are built-in rules for dealing with open
classes of symbols and penalties. Their names start with a dollar sign and
for the rest consist of capital letters, so that they are distinct from
nonterminals, but they are used like nonterminals with a special semantics.

Some of the quasi-terminals are parameterized with a quoted regular
expression (see \S\ref{se:RE}) which is to be matched against (a prefix of)
\YY{RE}{matching}
the input. A regular expression always matches characters up to a
terminator, unless it ends on a hyphen, in which case the \XX{hyphen}
{convention} (See \S\ref{hyphconv}) applies. A blank within the regular
expression matches a terminator. A regular expression cannot match an
empty text.
The default regular expression (\verb,.*,) will recognize any token
in the input. The following quasi-terminals are available:

\begin{flushleft}
\vspace{0.5cm}
\begin{tabular}{ll}
{\tt \$MATCH ("RE")}       & match the RE against the input \\
{\tt \$MATCH ("RE", TEXT)} & match the RE against the input
and unify the matched string with TEXT \\
{\tt \$SKIP ("RE")}        & match the RE against the input, but only \\
                           & if NO other symbol matches at this position \\
{\tt \$SKIP ("RE", TEXT)}  & Like \$SKIP ("RE") and unify the matched string with TEXT \\
{\tt \$OTHER}	           & match the input upto the next terminator, but only \\
		           & if NO other symbol or \$SKIP matches at this position \\
{\tt \$OTHER (TEXT)}       & Like \$OTHER and unify the string with TEXT \\
{\tt \$PENALTY}            & increase the current penalty level by 1 \\
{\tt \$PENALTY (INT)}      & increase the current penalty level by that number \\
\end{tabular}\vspace{0.5cm}
\end{flushleft}

\subsection{Penalties}\label{se:penalties}
A \X{penalty} in an alternative indicates that it is less probable or
desireable. The parser will produce parsings in order of increasing
penalty level, and when limiting the number of parses, the parsing(s) with
the lowest number of penalties is (are) preferred.
\begin{grammar}
RULE sentence:
   subject, transitive verb, object;   # {\tiny [he hits]} {\small me}
   subject, transitive verb, $PENALTY. # {\tiny [he hits]}
RULE subject : "he"; "they";
   $PENALTY, object.                   # {\small [them]} {\tiny likes me}
RULE object : "me"; "them".
RULE transitive verb: "hits": "likes".
\end{grammar}
Penalties may be considered as as rough approximations to (the negative
logarithm of) probabilities, and should be used with discretion.

\subsection{Regular expressions}\label{se:RE}
The \agfl\ formalism uses a restricted form of \XXs{regular}{expression}
with a conventional \XX{greedy}{RE semantics}, i.e.\
the longest possible match is taken, and it will result
in at most one match.

\begin{syntax}
regular-expression:
  character-expression,
    repetition-operator-option,
       concatenated-expression-option.

character-expression:
  character;
  "\(\backslash\)", character.
  ".";
  "[",  character-range-sequence, "]";
  "[^", character-range-sequence, "]".

character-range:
  character;
  character, "-", character.
\end{syntax}

A single character  in a regular expression matches itself, unless it
is a character with a special meaning (an \X{operator}). The backslash
is used as an \XX{escape}{character} in order to denote an operator.
A dot matches any character {\em except} a separator
(as given in the terminator string, see \S\ref{se:term}).
A space matches any separator.
A \XX{character}{range} matches
any character in the range, unless the range is preceded with a
\verb+^+. In the latter case, any character \emph{not} in the range is
matched.

\begin{syntax}
repetition-operator:
  "*";
  "+";
  "?".
\end{syntax}
The \XXs{repetition}{operator} are used for specifying the number of
occurrences of the preceding expression. An asterisk \verb+*+ denotes
zero or more occurrences, a plus \verb_+_ denotes one or more
occurrences, and a question mark \verb+?+ denotes an optional
occurrence.
\begin{grammar}
ORDINAL DIGITS: $MATCH("([0-9]*(1st|2nd|3rd|[4-90]th)").
\end{grammar}

\section{The hyphen convention}\label{hyphconv}
According to the \XX{hyphen}{convention}, a part word
ending (beginning) with a hyphen does not require the presence of
a word terminator following (preceding) it. As an example, either of the
sequences
\begin{verbatim}
     "querer", "-lo"
     "querer-", "lo"
     "querer-", "-lo"
\end{verbatim}
will match the input ``quererlo", but it will not be matched by
\begin{verbatim}
     "querer", "lo"
\end{verbatim}
To escape from the hyphen convention, a minus sign at the start or at the
end of a word has to be escaped by preceding it with a backslash.

\subsection{Dealing with prefixes and suffixes}
The example of ``quererlo" has already demonstrated how to deal with
suffixes. The grammar
\begin{alltt}
  RULE sent:
        "lo", "quiero";
        "querer", lo.

  RULE lo: "lo"; "-lo".
\end{alltt}
does not distinguish between ``quererlo" and ``querer lo"

\subsubsection{Example: negated adjectives}
As an example of the recognition of prefixes, consider negated adjectives
in English: the three forms ``nonexistent", ``non-existent" and ``non
existent" are presumably equivalent, and the last two should be
mapped onto the first. This is achieved by the grammar
\begin{alltt}
  possibly negated adj:
    "non", adj / "non", adj;
    "non-", adj / "non", adj;
    adj.
  
  adj: "existent"; "nonexistent".
\end{alltt}
which uses transduction (see \S\ref{se:transd}) to glue the ``non" to the
front of the adjective. The following is a trace of the corresponding
parser, with the options {\tt -tG} in effect.
\begin{verbatim}
>> nonexistent
   1 "non-" -> 4, "nonexistent" => 12
   4 "existent" => 12
  12 <EOS>
nonexistent 
nonexistent 
>> non existent
   1 "non" => 5
   5 "existent" => 13
  13 <EOS>
nonexistent 
\end{verbatim}
Notice that the first case was ambiguous (easily mended by a commit in the
first rule, by setting the number of analyses to one or by eliminating all
adjectives starting with ``non" from the lexicon).

\subsubsection{Example: Composed nouns}
Suppose we want to find out which nouns in the lexicon can be parsed as a
compound noun consisting of two nouns. We make a lexicon file
(\S\ref{se:lexica}) containing a copy of all nouns from the
lexicon, assigning them to a new class {\tt N} and adding a hyphen at the
end of their representation, e.g.
\begin{alltt}
"table-"	N
\end{alltt}
and write a grammar with as root
\begin{alltt}
  word:
    N, word / N1, "-", word  !
    N, noun / N, "-", noun!
    noun / !
    $MATCH(".*") / "NOT A NOUN :", $MATCH.
\end{alltt}
The lexicon for this grammar is composed of all nouns from the original
lexicon plus all {\tt N}s obtained from them. The parser (called with
{\tt -tP 1} to demand transduction of only the first analysis)
will recognize words composed of two or more nouns, which will be
transduced one-per-line, with a hyphen to separate parts. For non-composed
nouns an empty line is produced. Non-nouns give an error message.

By providing it with all nouns from the lexicon as input (one word per
line) we will obtain a list of all composed nouns.

\section{Different character sets}
The input may be in different \XXs{character}{set}: {\sc ascii}, 
extended {\sc ascii} {[} or {\sc unicode} ], provided the \agfl\ system has
been suitably installed.

\subsection{Invisible characters}
Some characters like quotes and brackets are hard to deal with in
a sensible and robust way.
There is a regular way to use them (e.g. as matching brackets
around certain syntactical constructions) but they may also occur in an
unpredictable way.
In those cases they may be skipped explicitly (expressed in the grammar,
imposing a suitable penalty).

Any characters marked as an \XX{invisible}{character}, will be skipped
automatically when not explicitly recognized by the parser.
By default, the \XX{white}{space} characters (space, tab, new line,
carriage return) are invisible, but if needed they may be matched by a
space in a regular expression (\S\ref{se:RE}). White space also serves
to terminate a line in line mode.

The following example shows how to extend the set of invisible
characters.
\begin{verbatim}
  |invisible characters " \t\n\r\'\`\"\(\)"|
\end{verbatim}

\subsection{Character conversions}
\YY{character}{conversion}
In richer character sets, certain characters may form a family of variants
with different \XXs{diacritical}{mark}, like e, \"e, \'e, \`e. In languages
like Dutch, these diacritical marks may mostly without problem be replaced
by the corresponding letters without diacritics.
But to include all diacritical variants of each word
in the lexicon appears cumbersome.  This particular problem can be solved
by mapping all the versions of the letter e onto e itself, and so on, using
a suitable \XX{alternative}{character} list.

Such a list consists of one or more pairs of numbers, one per line
with white space between the numbers constituting a pair. Such a pair
expresses that in the input each character whose code is equal to the
first member of a pair should be replaced by that character whose code is
the second member. Characters not mentioned in the list remain unchanged.

There are many reasons for a lowercase character to become capitalized,
e.g. at the beginning of a sentence, whereas there are no general reasons
for the opposite. Therefore the default character conversion causes an
upper case character to be a match for its lowercase equivalent (but not
the other way around).

\subsubsection{Example: character conversion}
The following alternative character list describes the default (and is
therefore superfluous): the automatic conversion
of upper case characters into the corresponding lower case ones.
\begin{verbatim}
   65 97    # A --> a
   66 98    # B --> b
   67 99    # C --> c
   68 100   # D --> d 
   69 101   # E --> e
   70 102   # F --> f
   71 103   # G --> g
   72 104   # H --> h
      ...
   90 122   # Z --> z
\end{verbatim}
Suppose that the file containing such a list is called {\tt altchars.dat};
the parser should then be invoked with {\tt -A altchars.dat}. Alternatively,
a pragmat may be given in the grammar, which directs the generated parser
to use a specific file as the default conversion table:
\begin {verbatim}
  |translate "altchars.dat"|
\end{verbatim}
To enforce that a generated parser should not use any conversion by default,
the pragmat
\begin{verbatim}
  |translate off|
\end{verbatim}
should be used.

\section{Lexical frequencies}
As described in paragraph \ref{lexfreq}, entries in a lexicon can have
a frequency as third element, after the part of speech. The default frequency
of an entry is one. The \XX{lexical}{frequencies} can be obtained from
a \X{tree bank} (a collection of validated parse trees), or from text
that has been tagged with a reliable tagger.

When \XX{hybrid}{parsing} is switched on (option {\tt -H} on the
generated parser), all entries with lexical frequency $f$ such that
$2^k \leq f < 2^{k+1}$ get a bonus. Taking a famous example, we
show how to impose a preferred structure on the sentence {\sf time flies
like an arrow}.
\begin{grammar}
GRAMMAR probab.

LEXICON small
DEFINES subject, verb, adject.

ROOT sent .

  sent:
    verb, object/
     "(", verb, ")", "(", object, ")!";
    verb, object,  adject/
     "(", verb, ")", "(", object, ")", "(", adject, ")!";
    subject, verb/
     "(", subject, ")", "(", verb, ").";
    subject, verb, object/
     "(", subject, ")", "(", verb, ")", "(", object, ").";
    subject, verb, adject/
     "(", subject, ")", "(", verb, ")", "(", adject, ").".

  object:
    subject, [adject].
\end{grammar}
The lexicon is as follows:
\begin{verbatim}
"time"  subject   10
"flies" subject   4
"an arrow"  subject  1
"time flies"    subject  1
"time"  verb  1
"flies" verb  5
"like"  verb  5
"like an arrow" adject 1
\end{verbatim}
Running the parser without the {\tt -H} option gives the
parsings in \XX{canonical}{order}, determined by the ordering
of alternatives in the grammar, and without any penalties.
\begin{verbatim}
twoquid.cs.kun.nl: probab -t -v parser
>> time flies like an arrow
# parsing 1 time 0.000 penalty 0
(time )(flies like an arrow )!
# parsing 2 time 0.000 penalty 0
(time )(flies )(like an arrow )!
# parsing 3 time 0.000 penalty 0
(time flies )(like )(an arrow ).
# parsing 4 time 0.000 penalty 0
(time )(flies )(like an arrow ).
>>
\end{verbatim}
The {\tt -v parser} option makes the penalties visible. Now
using hybrid parsing
\begin{verbatim}
 probab -tH -v parser
>> time flies like an arrow
# parsing 1 time 0.000 penalty 0
(time flies )(like )(an arrow ).
# parsing 2 time 0.000 penalty 1
(time )(flies )(like an arrow ).
# parsing 3 time 0.000 penalty 4
(time )(flies like an arrow )!
# parsing 4 time 0.000 penalty 4
(time )(flies )(like an arrow )!
>>
\end{verbatim}
The AGFL system includes some utilities to generate and maintain
lexica with frequencies.

\section{Transduction and its many uses}
\label{se:transd}
The output from an \agfl\ parser can be in one of the following forms:
\begin{enumerate}
\item an indented parse tree (default)
\item no output ({\tt -O option})
\item a label-bracketed tree ({\tt -b} option)
\item a transduced form ({\tt -t} option).
\end{enumerate}
In the last case, the form of the output is wholly determined by the
transduction specified in the grammar (which is ignored in the
other cases).

\subsection{Example: Fancy output}
Using transduction, the output does not have to look like a parse tree,
but it may be tailored in great detail to other purposes.
An example syntax-rule with a transduction-part, which transduces
{\sl a beautiful man} to \texttt{(the NP: [ DET/Q- a MOD- beautiful
HEAD- man ] has number = SINGULAR)}

\begin{grammar}
RULE NP (NUMBER):
  determiner (NUMBER) , adjective , noun (NUMBER) /
	"(the NP: [ DET/Q- ",  determiner,
	    "MOD- ",  adjective,
	      "HEAD- ",  NOUN,
	        "] has number = ",  NUMBER,  ") " .

DETERMINER (singular) : "a".
ADJECTIVE: "beautiful".
NOUN (singular) : "man".
\end{grammar}

\subsection{Example: Filtering for unknown words}
The transduction mechanism may be used to filter from a given text all
words that do not appear in the lexicon. The parser generated from
the following grammar can be used as such a \X{filter}. It makes use of
a lexicon called {\tt wordlex}.
\begin{grammar}
GRAMMAR filter.
LEXICON wordlex DEFINES
  NOUN(NUMB),
  VERB(NUMB,PERS),
  ADJE(GRAD),
  ADVB,
  ART(NUMB),
  DET(NUMB),
  PREP,
  POSSPRON,
  PERSPRON(NUMB, PERS, CASE),
  RELPRON(CASE).

ROOT input.
\end{grammar}
The parser will use segment mode, so that we can input any text in free
format. We allow a rich set of terminators (which will be skipped explicitly).
\begin{grammar}
| segment mode |
|word terminators " -+:,();.!?/%\"'"|

 input: segment, (input! ).

 segment:
   known / ;
   $MATCH(".*")/ $MATCH.
\end{grammar}
A segment is either a known word or an unkown one; in the latter case it
will be transduced literally. Terminators and known words will be skipped
without transduction.

The affixes needed by the lexicon are:
\begin{grammar}
 NUMB :: sing | plur.
 PERS:: first | secnd | third.
 CASE :: nom | gen | dat | acc.
 GRAD :: abso | comp | supl.
\end{grammar}
The next rule will recognize all words that occur in the lexicon.
\begin{grammar}
known:
  NOUN(NUMB);
  VERB(NUMB,PERS);
  ADJE(GRAD);
  ADVB;
  ART(NUMB);
  DET(NUMB);
  PREP;
  POSSPRON;
  PERSPRON(NUMB, PERS, CASE);
  RELPRON(CASE).
\end{grammar}
The parser generated outputs the transductions one-per-line, even if
they are empty. In order to get rid of spurious empty lines the output can
be filtered (under {\sc unix}) by a suitable {\tt grep}.
\begin{verbatim}
./filter -tP 1 | grep -v "^$"
\end{verbatim}

\subsection{Example: generating a lexicon}
\label{genlexi}
As a final example of the use of transduction, we describe a useful
technique for generating (part of) the lexicon for an inflected
language.

The ``traditional" approach in lexical analysis for inflected languages
is to construct a \XX{morphological}{grammar} describing the
structure of word forms in the language, and use a parser generated from
this, either preceding the syntax analysis or as part of the syntax
analysis. Often a tagger is used instead of a morphological parser,

We shall instead consider a technique where the lexicon is {\em generated}
from a concise list of stems, declinations and conjugations, taking the
nouns of the first and second conjugation in Latin as example. We make
a list specifying for each noun the conjugation and one representative
form. As output we want a sequence of entries for a lexicon file.
For the rest, the example should speak for itself.
\begin{verbatim}
GRAMMAR substgen.

 # INPUT PATTERNS:
 #  decl  nom
 #   a     -a
 #   b     -us/um
 # OUTPUT ENTRIES:
 #   LEXS(NUM,GENUS,CASUS)

NUM :: sg | pl.
GENUS :: fem | masc | ntr.
CASUS :: nom | voc | gen | dat | acc | abl.

ROOT substantivum.

 substantivum:
    "a", radix, "a" /
      "\"",radix,"a\"\tLEXS(sg,fem,nom|voc|abl)\n",
      "\"",radix,"ae\"\tLEXS(sg,fem,gen|dat)\n",
      "\"",radix,"am\"\tLEXS(sg,fem,acc)\n",
      "\"",radix,"ae\"\tLEXS(pl,fem,NOMVOC)\n",
      "\"",radix,"arum\"\tLEXS(pl,fem,gen)\n",
      "\"",radix,"is\"\tLEXS(pl,fem,dat|abl)\n",
      "\"",radix,"as\"\tLEXS(pl,fem,acc)\n";
    "b", radix, "us" /
      "\"",radix,"us\"\tLEXS(sg,masc,nom)\n",
      "\"",radix,"e\"\tLEXS(sg,masc,voc)\n",
      "\"",radix,"i\"\tLEXS(sg,masc,gen)\n",
      "\"",radix,"o\"\tLEXS(sg,masc,dat|abl)\n",
      "\"",radix,"um\"\tLEXS(sg,masc,acc)\n",
      "\"",radix,"i\"\tLEXS(pl,masc,NOMVOC)\n",
      "\"",radix,"orum\"\tLEXS(pl,masc,gen)\n",
      "\"",radix,"is\"\tLEXS(pl,masc,dat|abl)\n",
      "\"",radix,"os\"\tLEXS(pl,masc,acc)\n";
    "b", radix, "um" /
      "\"",radix,"um\"\tLEXS(sg,ntr,nom|acc|voc)\n",
      "\"",radix,"i\"\tLEXS(sg,ntr,gen)\n",
      "\"",radix,"o\"\tLEXS(sg,ntr,dat|abl)\n",
      "\"",radix,"a\"\tLEXS(pl,ntr,nom|acc|voc)\n",
      "\"",radix,"orum\"\tLEXS(pl,ntr,gen)\n",
      "\"",radix,"is\"\tLEXS(pl,ntr,dat|abl)\n";
   words / "ERROR:", words.  # to catch incorrect input
 
 words: word, words; .
 word: $MATCH(".*").
 radix:$MATCH(".*-").
\end{verbatim}
Latinists will note that this is a simplification (some
words ending on "-a" are masculine or even neuter), but the
overall approach should be clear enough, and extending this example
into a realistic lexicon generator should be straightforward.

In this way it is possible to generate e.g. hundred thousands of
verb entries from a few thousand stem times. The sheer size of the
generated lexicon presents no problem, since the lexicon system is 
up to it. And lexicon lookup is much faster than (morphological)
parsing.
Since the grammar uses transduction, the parser substgen has to be
started with transduction switched on, e.g.
\begin{verbatim}
substgen -t
>> a dea
"dea"   LEXS(sg,fem,nom|voc|abl)
"deae"  LEXS(sg,fem,gen|dat)
"deam"  LEXS(sg,fem,acc)
"deae"  LEXS(pl,fem,nom|voc)
"dearum"        LEXS(pl,fem,gen)
"deis"  LEXS(pl,fem,dat|abl)
"deas"  LEXS(pl,fem,acc)
>> 
\end{verbatim}
The generative approach described is prone to overgeneration, which can
be solved by filtering the generated words through a word list derived
from a corpus, accepting only those word forms which actually occur in
the corpus. In fact an analyting approach, analyzing stem+suffixes,
would acept precisely the same forms. But as
long as the spuriously generated forms can not be confused with other
words this presents no problem. A few words are too irregular to be
treated in this way and have to be dealt with separately.

\section{Example: Interfacing to a different tagset}
In this example, it is assumed that we have an English lexicon
interface of our own, with quasiterminals like {\tt ADJE(GRAD)},
and with affix rules like
\begin{verbatim}
   GRAD:: abso | comp | supl .
\end{verbatim}
(and many other quasiterminals and affixe rules).
We want to apply it to parse text from the Penn Tree Bank
(http://www.cis.upenn.edu/~treebank/)
that has been pre-tagged with tags like 
\begin{verbatim}
     JJ      adjective               green
     JJR     adjective, comparative  greener
     JJS     adjective, superlative  greenest
\end{verbatim}
A short sentence in tagged form looks like
\begin{verbatim}
(NNS Negotiations)
(, ,)
(RB however)
(, ,)
(AUX were)
(JJ unsuccessful)
(. .)
\end{verbatim}
Originally, our grammar had lexical interface rules like
\begin{verbatim}
  LEX_ADJ (GRAD):
     ADJE(GRAD);
     $PENALTY(2), hyphenated form, {GRAD :: abso};
 # EX that was my never-do-well nephew
     $PENALTY(2), robust adje (GRAD).
 
 hyphenated form:
        $PENALTY(2), $SKIP("[a-zA-Z0-9]+-[-a-zA-Z0-9]+")

 robust ADJE(abso):
    $SKIP("....*[ai]ble");
    $SKIP("....*a[clr]");
    $SKIP("....*ary");
    $SKIP("....*ate");
    $SKIP("....*form");
    $SKIP("....*fold");
    $SKIP("....*ic");
    $SKIP("....*i[lv]e");
    $SKIP("....*like");
    $SKIP("....*oid");
    $SKIP("....*ory");
    $SKIP("....*ous");
    $SKIP("....*tic");
    $SKIP("....*wise");
    $SKIP("....*less").
\end{verbatim}
We would like to
\begin{itemize}
\item simply skip the enclosing brackets as we go along (rather
than removing them from the input first)
\item accept the POS-tag of Penn for adjectives that are also
in our lexicon
\item also accept Penn adjectives according to our robustness rules,
albeit with a penalty
\item mark the other words that Penn considers adjectives in
the transduction by a questionmark.
\end{itemize}
We can achieve this by modifying our interface rule for adjectives
into
\begin{verbatim}
  LEX_ADJ (GRAD):
     "(JJ", ADJE(abso), "-)", {GRAD :: abso};
     "(JJR", ADJE(comp), "-)", {GRAD :: comp};
     "(JJS", ADJE(supl), "-)", {GRAD :: supl};
     "(JJ", $PENALTY(2), hyphenated form, "-)", {GRAD :: abso};
 # EX that was my never-do-well nephew
     "(JJ", $PENALTY(2), robust adje (GRAD), "-)"!
     $MATCH("JJ[RS]?"), $MATCH(".*") /
        "?:", $MATCH.2.
\end{verbatim}
and similarly for the other parts-of-speech.

