\chapter{The AGFL formalism}\label{ch:grammar}
This chapter is the reference document for the syntax and the semantics of
the \agfl\ formalism. This is heavy stuff.
For its understanding previous knowledge of the descriptional
formalism used (see \S\ref{ch:formalism}) is desireable.
This chapter provides important terminology and describes
the constituents of grammars and grammar rules and their semantics
in a Top-Down order. While this order
is reasonable for readers already familiar with \agfl, it is not the best
order for didactic purposes, for which an informal introduction (see
\S\ref{ch:intro}) is advised.

An \XX{affix}{grammar} \cite{KOS71} (such as \agfl) is a
\XX{two-level}{grammar}, consisting of
\begin{itemize}
\item an \XX{underlying}{CF grammar}, defining CF productions (also
known as \XXs{grammar}{rule} or just \Xs{rule})
for certain nonterminal symbols (aka \Xs{nonterminal})
\item a \XX{meta}{grammar}, defining CF production rules
(aka \XXs{affix}{rule}) for
certain nonterminal affixes (aka \XXs{affix}{nonterminal})
\item an annotation of the rules of the underlying grammar with
\Xes{affix} serving as \Xs{parameter}, and
\item \Xs{call} on certain \Xs{operation} embedded in the grammar.
\end{itemize}
\[ \left\{ \begin{array}{rcl}
          affixes &+& domains\\CF grammar &+& operations
          \end{array} \right\} \]
\noindent
Affix Grammars over a Finite Lattice (AGFL) is a particular form of
affix grammars in which the domains are finite and
set-valued, and the operations are set intersection and union (called
implicitly rather than explicitly).

\section{Overall structure of AGFL's}
An \agfl\ grammar consists of a number of files
(also called \Xs{module}):
\begin{enumerate}
\item the \XX{main}{grammar}, containing the \X{root} of the grammar
\item any \XXs{sub}{grammar} used by it
\item any \XXs{lexicon}{file} used by it.
\end{enumerate}
A file containing a (sub)grammar has the extension {\tt .gra},
a lexicon file has the extension {\tt .dat}.

The \agfl\ system translates one grammar, the \XX{main}{grammar},
together with all its sub-grammars and lexicon files, into a \X{parser}.

\section{Grammars}
A grammar has a \XX{grammar}{head} identifying the grammar, a
\XX{grammar}{interface} describing its relations to other grammars, and
a \XX{grammar}{body} defining the \Xs{object} of that grammar.
\begin{syntax}
grammar:
   grammar-head, grammar-body.
\end{syntax}
\begin{syntax}
grammar-head:
   "GRAMMAR", grammar-name, ".".
\end{syntax}
The name of the grammar should be the same as the name of the file
in which it is contained (apart from the extension {\tt .gra}).

\subsection{Grammar body}
The body of a grammar consists of interface descriptions, specifications
and definitions for \Xs{object} (nonterminal symbols and nonterminal affixes).
The order of the specifications and definitions is irrelevant, apart from
the fact that the order in which parses are found may depend on the order
of definitions in the grammar.
\begin{syntax}
grammar-body:
   statement-sequence.
\end{syntax}
\begin{syntax}
statement:
   lexicon-interface;
   includes-part;
   root-statement;
   specification;
   definition.
\end{syntax}
\begin{syntax}
definition:
   syntax-rule;
   affix-rule.
\end{syntax}

\subsection{Interface descriptions}
An \XX{includes}{part} of a grammar specifies \XXs{sub}{grammar}
which must be included in it.
\begin{syntax}
includes-part:
   "INCLUDES",  grammar-name-list, ".".
\end{syntax}

The \XX{root}{statement} in the main grammar indicates which nonterminal
serves as \X{root} of the system. Only the main grammar should have a
root-statement. If no root-statement is given, its first syntax-rule is
taken as root. The root must be a nonterminal with arity zero,
i.e.\ without parameters.
\begin{syntax}
root-statement:
   "ROOT", nonterminal-name, ".".
\end{syntax}

A \XX{lexicon}{interface}, specifies nonterminals which are realized by
means of a lexicon rather than by syntax rules, together with their
parameters and arity.
\begin{syntax}
lexicon-interface:
   "LEXICON", lexicon-name-list, defines-part.
\end{syntax}
The lexicon-name-list of a lexicon interface lists the lexicon
files which it accesses. The defines-part specifies the
\XXs{lexicon}{nonterminal}, implemented by a lexicon rather than by
syntax rules.
\begin{syntax}
defines-part:
   "DEFINES", specification-list, ".".
\end{syntax}

\begin{example}{10.9cm}
\hspace{2mm}All the above elements are shown in the following example:
\begin{grammar}
GRAMMAR example.                      # {\rm name of the grammar}
USES subj, verb.                      # {\rm grammars used}

LEXICON adj, noun, adv, misc, verb    # {\rm lexicon files}
DEFINES
   ADJE(GRAD),                        # {\rm lexicon nonterminals}
   ADJE(GRAD,PREP),
   ADJE_TO(GRAD),
   ADVB,  
   NOUN(NUMB),
   DIMENSION,
   VERBI(PREP,TRAN),
   VERBS(PREP,TRAN),
   VERBV(PREP,TRAN),
   VERBG(PREP,TRAN),
   VERBP(PREP,TRAN).

ROOT sentence.                        # {\rm root of the grammar}

RULE subject (NUMBER).                # {\rm specification}
subject (NUMBER): pronoun (NUMBER).   # {\rm syntax rule}
NUMBER :: plural | singular.          # {\rm affix rule}
\end{grammar}
\end{example}

\section{Affix Rules}\label{affix-rule}
\YY{affix}{rule}
Affix rules (or \XXs{meta}{rule}) can be seen as CF rules on the meta level,
in a slightly different notation, using as delimiters {\tt ::} instead of the
single colon to separate the left- from the right-hand-side, and {\tt |} instead
of {\tt ;} to separate alternatives. (Currently, there is in the meta-level
of \agfl\ no equivalent of the comma composing alternatives).

An affix rule serves to specify the \X{domain} of an
affix nonterminal. This domain is either 
\begin{itemize}
\item one of the predefined \XXs{flat}{lattice} {\tt INT} and
{\tt TEXT} (see \S\ref{lattices}), or
\item a \XX{power-set}{lattice}, consisting of all subsets of a
specified set of \XXs{affix}{terminal}.
%%, or
%%\item a \X{tuple} of domains.
%%   CONCORD :: PERSON NUMBER.                    # {\rm affix tuple}
\end{itemize}
\begin{example}{2.5cm}
\begin{grammar}
VERBFORM  :: FINITE | PARTICP | infinitive . # {\rm hierarchy}
FINITE    :: first | second | third .        # {\rm \hspace*{.4cm}of three}
PARTICP   :: past | present .                # {\rm \hspace*{.4cm}rules}
VERBLEMMA :: TEXT .                          # {\rm flat affix}
\end{grammar}
\end{example}
[In each power-set-lattice, the number of distinct affix terminals is
presently limited to \textbf{31}, but we hope to drop this restriction.]
\begin{syntax}
affix-rule:
   affix-rule-head, "::", affix-expression, ".".
\end{syntax}
\begin{syntax}
affix-rule-head:
   affix-nonterminal-list.
\end{syntax}
Affixes with the same domain are called \XXs{affix}{synonym}.
They may be defined by separate rules, or together in one affix rule:
\begin{example}{3.3cm}
\begin{grammar}
PREP, P :: from | to | by | with | of.
\end{grammar}
\hspace{2mm}is a shorthand for
\begin{grammar}
PREP :: from | to | by | with | of.
P :: PREP.
\end{grammar}
\end{example}
The right-hand-side of an affix rule is an \XX{affix}{expression}; the
same construct can occur as an actual or formal parameter (see
\S\ref{ruletypes}).
\begin{syntax}
affix-expression:
   affix-term.
\end{syntax}
%%The most general structure for an affix domain is a tuple of
%%affix domains, each specified by an affix nonterminal
%%\begin{syntax}
%%affix-tuple:
  %%affix-nonterminal, affix-nonterminal-sequence.
%%\end{syntax}
\begin{syntax}
affix-term:
   affix-nonterminal;
   affix-terminal-set;
   predefined-affix;
   text-denotation;
   number.

affix-nonterminal:
   affix-name.

affix-terminal-set:
   affix-terminal;
   affix-terminal, "\verb+|+", affix-terminal-set. 

predefined-affix:
   "INT";
   "TEXT".
\end{syntax}
An \XX{affix}{terminal} is usually a name, written in small letters and
some special characters (see section \S\ref{naming}). For the predefined
flat domains \texttt{TEXT} and \texttt{INT} it is a text-denotation or a
number, respectively. In \agfl\ {\em recursive affix rules are forbidden},
but a hierarchy of affix rules can be described.

\section{Specifications}\label{specification}
Nonterminal symbols can have affixes as parameters. The number of
parameters is called the \X{arity} of the nonterminal. Since a call of a
nonterminal with a certain arity can refer only to the definitions
for that nonterminal with that arity, a nonterminal is allowed to occur
with different arities, each with their own definitions.

A \X{specification} for a nonterminal with a certain arity serves to
establish
\begin{itemize}
\item the \X{type} of a nonterminal (e.g. \texttt{RULE}), which must be
the same for all its arities, and
\item the \Xs{domain} of the parameters for this arity, i.e. the set of
values which the $i$th parameter can take on, specified by a nonterminal
affix. In this sense {\em affixes also act as types}.
\end{itemize}
{\em Specifications are optional}. If a nonterminal with a certain arity
has no explicit specification, the domains of its parameters are
implicitly specified by affix nonterminals which are the
\X{most general unifier} of the affix domains with which the
nonterminal with that arity is defined anywhere in the grammar.
If there is no such affix nonterminal this is considered an error.
\begin{syntax}
specification:
   syntax-rule-head, ".".
\end{syntax}
A nonterminal may have more than one specification with different arities.
{\em Specifications may be repeated} as long as they do not contradict
one another.
All definitions of a nonterminal with a given arity must have a type
(of the nonterminal) and domains (of its parameters) agreeing with the
specification.
% MS: We moeten echt iets doen aan die example macro's
\begin{example}{2.4cm}
\hspace{2mm}A nonterminal with two different arities:
\begin{grammar}
RULE verb (NUMBER, PERSON, TIME).       # {\rm finite form}
RULE verb (TIME).                       # {\rm nonfinite form}
\end{grammar}
\end{example}
\begin{itemize}
\item the nonterminal {\tt verb} is specified with arities 3 and 1
\item since it is specified as a {\tt RULE}, it may not produce empty
(see \S\ref{ruletypes})
\item the domains of the parameters are specified by affix nonterminals
{\tt NUMBER}, {\tt PERSON}, etc.
\end{itemize}

\noindent
For each call of a {nonterminal}, it is checked whether the actual
parameter domains conform to the {specification} of the parameters for
that arity.

\section{Syntax rules}\label{SyntaxRules}
A nonterminal of a certain arity is defined by one or more
\XXs{syntax}{rule} with that name and arity.
\begin{syntax}
syntax-rule:
   left-hand-side, ":", right-hand-side, ".".
\end{syntax}
Nonterminals with the same {right-hand-side} can be defined together,
according to the following convention:
\begin{syntax}
left-hand-side:
   syntax-rule-head-list.

right-hand-side:
   alternatives.
\end{syntax}
If there are more rules for one {nonterminal} with the same arity,
their alternatives are combined in textual order.

\subsection{Rule heads}\label{ruletypes}
\begin{syntax}
syntax-rule-head:
   syntax-rule-type, nonterminal-name, formal-parameter-list-pack-option.
\end{syntax}
\begin{syntax}
syntax-rule-type:
   "RULE"; ;
   "OPTION";
   "CONDITION".
\end{syntax}
{\tt CONDITION} may be shortened to {\tt COND}. {\tt RULE} is the
default, which may be omitted.

\subsubsection{Rule types}
Syntax rules are typed according to whether they are \X{productive}
in rewriting:
\begin{itemize}
\item a \texttt{RULE} \emph{must} always generate at least one terminal
symbol
\item an \texttt{OPTION} \emph{must be able to} generate empty as well as
a terminal symbol
\item a \texttt{CONDITION} \emph{must never} generate a terminal symbol
(but it may generate empty).
\end{itemize}
The type {\tt RULE} is optional (and assumed as default). The type {\tt
OPTION} is used for indicating rules that may produce empty, and therefore
never fail (aka \XXs{nonfalse}{rule}).
\pagebreak
\begin{example}{3cm}
\hspace{2mm}An example of typed rules:
\begin{grammar}
RULE postmodifiers:
   postmodifier, maybe postmodifiers.
OPTION maybe postmodifiers:
   postmodifier, maybe postmodifiers; .
\end{grammar}
\end{example}
\noindent
The type {\tt CONDITION} is typically used for expressing a \X{predicate},
a test or restriction on
affix values, possibly combined with a penalty (see \S\ref{se:penalties}).
\begin{example}{3.8cm}
\hspace{2mm}Especially in British english, the agreement rules between subject and verb clause are
\hspace{2mm}a bit hazy, because certain singular nouns may be interpreted as a collective name
\hspace{2mm}(\textit{constructio ad sententiam}). This may be expressed by the following agreement predicate
\begin{grammar}
COND agrees (NUMBER, NUMBER):.
COND agrees (singular, plural): $PENALTY.
COND agrees (plural, singular): $PENALTY(2).
\end{grammar}
\end{example}\vspace{.1cm}

\noindent
All definitions
for a nonterminal must have the same type, independent of the arity.

The \XXs{rule}{type} introduce redundancy into the grammar, allowing the
system to check the intentions of the grammar writer (expressed in the
head of the syntax rule) against his/her
deeds (expressed in the body of the syntax rule and in its calls).

\subsubsection{Formal parameters}
\begin{syntax}
formal-parameter:
   parameter.
\end{syntax}
A \XX{formal}{parameter} acts as a variable which is bound against the
corresponding actual parameter (by set unification).
Formal parameters and actual parameters (see \S\ref{actpar}) have the same
syntactic richness.  \emph{Everything} that may be written as an actual
parameter may also be written as a formal parameter.
\begin{syntax}
parameter:
   affix-variable;
   affix-expression.
\end{syntax}
In \XXs{strict}{affix grammar}, all parameters are variables, and the
domains of formal and actual parameters must be the same.
In \XXs{extended}{affix grammar}, like {\sc eag} and \agfl\,
affix expressions may occur at parameter positions, and the domain of the
formal parameter may be larger or smaller than that of the actual
parameter, as long as the intersection is not empty. Having an affix
expression as formal parameter implies an \XX{implicit}{guard} (which is
said to be ``sugared away").\YY{syntactic}{sugar}
\begin{example}{4.2cm}
\hspace{2mm}An example of this syntactic sugar:
\begin{grammar}
RULE to be (NUMBER, second): "you".
\end{grammar}
\hspace{2mm}is equivalent to the (strict) rule
\begin{grammar}
RULE to be (NUMBER, PERSON):
   \{PERSON :: second\}, "you".
\end{grammar}
\end{example}
\begin{syntax}
affix-variable:
   affix-nonterminal, index-option.

index: number.
\end{syntax}
An \XX{affix}{variable} occurring in a syntax rule
must obtain a nonempty value which is no larger
than the domain of the corresponding {affix\-nonterminal}.
The optional
\X{index} of a {affix\-variable} serves to indicate different instances
of an {affix\-nonterminal} in one {syntax\-rule}. Therefore
{affix\-variable}s with a different {index} may have different values.
\begin{example}{2.3cm}
\hspace{2mm}An example of different instances of one {affix\-nonterminal} is:
\begin{grammar}
RULE sentence:
   subject (NUMBER1), verb (NUMBER1), object (NUMBER2).
RULE subject (NUMBER): pronoun (NUMBER).
\end{grammar}
\end{example}
The {index} is only used to distinguish one instance from another within
one alternative, and it has no influence on the use (or name) of the
{parameter} in the rest of the grammar.

\subsection{Rule bodies}\label{alternatives}
Alternatives in a rule may be separated either by a semicolon \verb+;+
or by the \XX{commit}{operator} \verb+!+.
\begin{syntax}
alternatives:
   alternative-series,
      commit-operator,
         alternatives;
   alternative-series.

commit-operator:
   "!".
\end{syntax}
During parsing, the alternatives of a rule will be tried in textual
order by recursive backtrack. For each succesfull alternative the
continuation of the parsing is peformed.
However, if one of the {alternative} preceding a {commit\-operator}
has succeeded, then the {alternative}s following the
{commit\-operator} will be skipped. 

The commit operator may be used as a robustness device (preferring
syntactically correct input but allowing controlled ungrammaticality)
% MS: Same problem as before
\begin{example}{3.0cm}
\begin{grammar}
RULE subject (NUMBER):
   pronoun (NUMBER, nom);      # {\small [he]} {\tiny walks right there}
   NP (NUMBER) !               # {\small [the man]} {\tiny looks at him}
   pronoun (NUMBER, acc).      # {\small [them]} {\tiny did it}

CASE :: nom | gen | dat | acc.
\end{grammar}
\end{example}
An {alternative} consists of a \X{production} with an optional
\XX{transduction}{part}.
\begin{syntax}
alternative:
   production,
      transduction-part-option.
\end{syntax}

\subsection{Productions}
\begin{syntax}
production:
   frequency-part-option, syntax-part.
\end{syntax}
\begin{syntax}
frequency-part:
  "[", number, "]".
\end{syntax}
The optional frequency parts serve to indicate {\bf syntactic
probabilities}\YY{syntactic}{probability} in the grammar.
\YY{syntactic}{frequency}
In case no frequency-part is given, a \XX{default}{frequency} is
taken: 2 for an empty-generating production, and 1 otherwise.
This distinction is made in order to be able to generate [later]
random examples of terminal productions without going too deeply
into recursion.

The \XX{syntax}{part} consists of a (possibly empty) list of members,
optionally followed by a final group, or it consists of free order
members (unfortunately, the two can not be mixed).
\begin{syntax}
syntax-part:
   member-list;
   member-list, ",", final-group;
   success-operator-option;
   free-order-part.
\end{syntax}
Groups may only appear at the end of a production.
An empty alternative may be explicitly marked with the
\XX{succes}{operator} {\tt +}.
\begin{syntax}
member:
   call;
   option;
   guard;
   terminal-symbol.
\end{syntax}

\subsection{Calls}\label{actpar}
Most {member}s  are likely to be be {call}s.
A \X{call} is the invocation of a \X{nonterminal} with an optional list
of \XXs{actual}{parameter}.
\begin{syntax}
call:
   nonterminal-name, actual-parameter-list-pack-option.

actual-parameter:
   parameter.
\end{syntax}
This also includes calls of lexicon terminals (see \S\ref{se:quasi}).

Having an affix expression as actual parameter again implies a restrict
guard which is ``sugared away".\YY{implicit}{guard}\YY{syntactic}{sugar}

\begin{example}{2.5cm}
\hspace{2mm}The following two lines are equivalent
\begin{grammar}
to be (NUMBER, second)
to be (NUMBER, PERSON), \{PERSON :: second\}
\end{grammar}
\end{example}

\subsection{Final groups}
A \XX{final}{group} allows the last member of an alternative to be
(recursively) a choice between alternatives.

\begin{syntax}
final-group:
   "(", alternatives, ")".
\end{syntax}
\begin{example}{3cm}
\hspace{2mm}Example of an final group:
\begin{grammar}
RULE noun group (NUMBER):
   noun (NUMBER),
      ( post adjective;
        "of", np (NUMBER1)).
\end{grammar}
\end{example}

\subsection{Options}
An \X{option} can be seen as a shorthand for an anonymous rule, whose
body consists of the given {alternative} followed by an empty
{alternative}.
\begin{syntax}
option:
   "[", alternatives, "]".
\end{syntax}

\begin{example}{8.8cm}
\hspace{2mm}The rule
\begin{grammar}
RULE NP:
   determiner, [adjectives], noun.
\end{grammar}
\hspace{2mm}is short for something like
\begin{grammar}
RULE NP:
   determiner, Oqcxaaq, noun.

Oqcxaaq:
   adjectives; .
\end{grammar}
\hspace{2mm}It would be more elegant to introduce  expicitly a rule like
\begin{grammar}
optional adjectives:
   adjectives; .
\end{grammar}
\end{example}

\subsection{Terminal symbols}
A \XX{terminal}{symbol} of the grammar (or \X{terminal} for short)
is either a syntax terminal or a lexicon-terminal or a quasi-terminal.
\begin{syntax}
terminal-symbol:
   syntax-terminal;
   lexicon-terminal;
   quasi-terminal.
\end{syntax}

A lexicon-terminal is a (suitably parameterized) call of a 
lexicon-nonterminal imported from one of the lexica used.
\begin{syntax}
lexicon-terminal:
   nonterminal-name, actual-parameter-list-pack-option.
\end{syntax}

A \XX{syntax}{terminal} is a text denotation that is to be matched
against (part of) the input.  Usually the syntax terminals are
defined in a separate lexicon (see \S\ref{se:lexica}), but they may
also be defined in the grammar.  However, matching of
syntax terminals from the lexicon is much faster than matching
syntax terminals from the grammar. Therefore, in a production situation
it is advisible to avoid having syntax terminals in the grammar.
\begin{syntax}
syntax-terminal:
   text-denotation.
\end{syntax}
\begin{example}{1.7cm}
\hspace{2mm}The following rule is a candidate for inclusion as three entries in the lexicon:
\begin{grammar}
RULE perspron (plural): "we" ; "they" ; "them" .
\end{grammar}
\end{example}

A \XX{quasi}{terminal} is a (suitably parameterized) call of a built-in
function.
\begin{syntax}
quasi-terminal:
   special-name, actual-parameter-list-pack-option.
\end{syntax}
A quasi-terminal serves to specify a (possibly open) class
of \XXs{terminal}{symbol}, whose structure is described by a
\XX{regular}{expression} given as (one of) its parameter(s) which must
be matched against (part of) the input.

\begin{example}{5.2cm}
\hspace{2mm}A (very crude) robust rule for recognizing nouns and guessing their number might be
\begin{grammar}
RULE noun (NUMBER):
   lexnoun (NUMBER);
   $PENALTY (4),
      ( $MATCH("[a-z]*s"), \{NUMBER :: plur\};
        $MATCH("[a-z]*[^s]"), \{NUMBER :: sing\}
      ).
\end{grammar}
\hspace{2mm}The Regular Expression matching used here is explained in \S\ref{se:RE}.
\end{example}

\subsection{Guards}\label{guards}
A \X{guard} serves to express the \X{restriction} of the value of one
(or more) affix variables to a certain domain.
\begin{syntax}
guard:
   "\{", restriction-list, "\}".

restriction:
   affix-variable, "::", affix-expression.
\end{syntax}
A restriction specifies that the value of the affix-variable is
restricted by set-intersection to (a subset of) the set specified by
the affix-expression. If this intersection is empty, the guard fails.
An affix-expression occuring at a parameter position denotes an
\XX{implicit}{restriction}.

It is important to realize that restriction is not equality:
a guard like
{\tt \{PERSON :: first\}} does not check whether {\tt PERSON} has the
value {\tt first}, but whether its value {\em includes} {\tt first};
if that is the case, its value is restricted to {\tt first}.

\subsection{Free order parts}
A \XX{free-order}{part} describes in a compact way a construct whose
members may occur in any order.
\begin{syntax}
free-order-part:
   member, "&", member;
   member, "&", free-order-part.
\end{syntax}
A free\-order\-part recognizes all permutations of its members.
\begin{example}{6.9cm}
\hspace{2mm}The following complement orders are possible for a Dutch sentence:
\hspace{7mm}{\tiny hij gaf} {\small [het boek] [aan jou] [op woensdag]}
\hspace{7mm}\emph{{\tiny he gave} {\small [the book] [to you] [on Wednesday]}}
\hspace{7mm}{\tiny hij gaf} {\small [het boek] [op woensdag] [aan jou]}
\hspace{7mm}{\tiny hij gaf} {\small [op woensdag] [het boek] [aan jou]}
\hspace{7mm}{\tiny hij gaf} {\small [aan jou] [het boek] [op woensdag]}
\hspace{7mm}{\tiny hij gaf} {\small [aan jou] [op woensdag] [het boek]}
\hspace{7mm}{\tiny hij gaf} {\small [op woensdag] [aan jou] [het boek]}
\hspace{2mm}These can be described by a syntax rule with a free order part as its right-hand-side:
\begin{grammar}
RULE complements:
   direct object NP & indirect object PP & time PP .
\end{grammar}
\end{example}

\section{The Transduction part}
The default output of an \agfl-parser is a \XX{parse}{tree}, which can
also be obtained in the form of a \XX{labelled}{bracket tree}.
However, any other form 
of output may be specified in the grammar in its stead, by giving a
\X{transduction} for each syntax rule. The \XX{transduction}{part}
of a syntax rule specifies this alternative output (see \S\ref{se:transd}
for examples).
\begin{syntax}
transduction-part:
   "/", transduction-member-list-option.
\end{syntax}

The transduction part rewrites to the concatenation of the
transductions of its transduction members.  A
transduction member may be a text denotation, which rewrites to its
literal value, or a placeholder referring to an element in the preceding
syntax part. The transduction of a placeholder will be its transduction as
specified in the rule specifying this placeholder.
If no transduction part is specified, the \XX{standard}{transduction}
will be used.

\begin{quote}
\large The standard transduction consists of the
concatenation of the transductions of the members in the
syntax part of the alternative, in that order.
\end{quote}

\begin{example}{1.8cm}
\hspace{2mm}The following rules have the same meaning:
\begin{grammar}
A: B, C, D.
A: B, C, D / B, C, D.
\end{grammar}
\end{example}
The \XX{empty}{transduction} may be
specified by means of an empty transduction part.

\begin{example}{2cm}
\begin{grammar}
A: B, C, D / .
\end{grammar}
\hspace{2mm}The transduction results of {\tt B}, {\tt C} and {\tt D} disappear.
\end{example}

~
\begin{syntax}
transduction-member:
   nonterminal-placeholder;
   quasi-terminal-placeholder;
   affix-placeholder;
   group-placeholder;
   option-placeholder;
   text-denotation.
\end{syntax}
The form of the placeholders and their transductions are discussed
below.

\subsubsection{Placeholder indices}
A dot index .$n$ following a nonterminal placeholder is used to refer to
the $n^{th}$ instance of that nonterminal in the production; and
similarly for the other kinds of placeholders.
An index $n$ used as a placeholder refers to the $n^{th}$
member of the production.

\subsubsection{Nonterminal placeholders}
\begin{syntax}
nonterminal-placeholder:
  nonterminal-name, dot-index-option;
  index.

dot-index: ".", number.
\end{syntax}
A nonterminal placeholder consists of the name of the corresponding
nonterminal (without the parameters!). Its transduction consists of
the transduction of the corresponding nonterminal in the syntax-part.
\begin{example}{1.7cm}
\begin{grammar}
RULE sent: subject(N,P), predicate (N,P) /
   "[", subject, ",", predicate "]".
\end{grammar}
\end{example}

\subsubsection{Affix placeholders}
\begin{syntax}
affix-placeholder: affix-variable, dot-index-option.
\end{syntax}
The transduction of an affix-placeholder consists of the
value of the corresponding affix-variable in the syntax-rule.
\begin{example}{3cm}
\begin{grammar}
PART:: none; PREP.
RULE verbform (N, P, PART).
verbform (N, P, none):
   verb (N,P), preposition(PREP)/ verb, preposition.
verbform (N, P, PART):
   verb (N,P)/ verb, PART.
\end{grammar}
\end{example}

\subsubsection{Quasi-terminal placeholders}
\begin{syntax}
quasi-terminal-placeholder:
   special-name, dot-index-option.
\end{syntax}
As is the case with nonterminals, the dot index .$n$ of a special name is used
to refer to the $n^{th}$ instance of this special name in the production. The
transduction of a quasi-terminal-placeholder is the character
sequence in the input matched by that quasi-terminal.
If it ended on a terminator character (see \S\ref{se:term})
the transduction will include a space at that position.
\begin{example}{2cm}
\begin{grammar}
RULE filter:
   word;
   $MATCH(".*") / "SKIPPED: ", $MATCH.
\end{grammar}
\end{example}

\subsubsection{Group placeholders}
The transduction of a group placeholder is the transduction of the
corresponding group.
\begin{syntax}
group-placeholder: "()", dot-index-option.
\end{syntax}
\begin{example}{2.8cm}
\begin{grammar}
sententia:
    subject(NUMB,PERS,GENDER),
       (nominal predicate(NUMB,PERS,GENDER;
        verbal predicate(NUMB,PERS,GENDER)) /
           "SUBJ:", subject, "PRED:", ().
\end{grammar}
\end{example}

\subsubsection{Option placeholders}
The transduction of a option placeholder is the transduction of the
corresponding option.
\begin{syntax}
group-placeholder: "[]", dot-index-option.
\end{syntax}
\begin{example}{2.7cm}
\begin{grammar}
sentence:
    [circumstance], subject(NUMB,PERS),
       VOC clause(NUMB,PERS), [circumstance] /
          "[", subject, ",", VOC clause, [].1, [].2, "]".
\end{grammar}
\end{example}

\subsubsection{Text denotation}
A text denotation is transduced literally (as in the above examples).

\section{Lexica}\label{se:lexica}
Linguistic applications generally require a parser with a large lexical
database.
While developing a grammar it may be perfectly adequate to include in
it a small or even reasonably large lexicon in the form of grammar rules
producing a choice between terminal symbols. But in case of
applications with a larger vocabulary it is
better to create separate modules for grammar and lexicon.
Separating grammar and lexicon has the following advantages:
\begin{itemize}
\item faster parser generation;
\item faster lexical analysis and parsing;
\item separation of concerns by grammar modularization;
\item better maintainability of grammar and lexicon.
\end{itemize}

\subsection{Lexica in \agfl}\label{lexfreq}
In the \agfl\ system a lexicon can be used from a
grammar or another lexicon and will then be automatically included in
the parser generated. The lexicon interface has already been described.

A lexicon file has the following syntax:
\begin{syntax}
lexicon-file:
   entry-or-comment-sequence.

entry-or-comment:
   lexicon-entry;
   comment.
\end{syntax}

A \XX{lexicon}{entry} is rather like a (simple) grammar rule written
backwards.
\begin{syntax}
lexicon-entry:
   terminal-symbol, tab-character, call, frequency-part-option.
\end{syntax}
The tab character indicated serves to separate the two sides of the rule
in a uniform style, suitable for using {\tt grep} and similar {\sc unix}
utilities.
The inverted notation for lexicon rules also makes it easier to maintain
lexica. Some utilities for merging and subtracting lexica are included
with the \agfl\ system.
\begin{example}{5.5cm}
\begin{grammar}
#  auxverb TO BE
"am"    to be(sing,first)
"are"   to be(plur,first|third)
"are"   to be(NUMB,secnd)
"is"    to be(sing,third)
#  auxverb TO HAVE
"have"  to have(sing,first)
"have"  to have(plur,first|third)
"have"  to have(NUMB,secnd)
"has"   to have(sing,third)
\end{grammar}
\hspace{2mm}Notice that the second person is described separately, in order to cope with the fact that
\hspace{2mm}english does not distinguish the singular from the plural in the second person pronoun.
\end{example}

\begin{example}{4cm}
\hspace{2mm}An example with explicit lexical frequencies (in the previous example the default value 1
\hspace{2mm}was assumed) reflects the fact that the words {\sf time} and {\sf times} prevalently occur as nouns
\hspace{2mm}rather than as verbforms.
\begin{grammar}
"time"    NNT1   2814   
"time"    VVI    6   
"times"   NNT2   345
"times"   VVN    1
\end{grammar}
\end{example}

\section{Lexical conventions}\label{lex-conv}
The conventions regarding the form of names, denotations, comments and
pragmats are discussed in this section.

\subsection{Names}\label{naming}
The \Xs{name} are formed
from \Xs{letter} and some additional characters, the \Xs{ornament}.
\agfl\ is case-sensitive, but the distinction between \XX{upper}{case} and
\XX{lower}{case} letters has no particular semantics, apart from the
following:
\begin{itemize}
\item affix-names are in capital letters
\item affix-terminals are in small letters. They may start with a {\tt +} and {\tt -}.
\item nonterminal names start with a letter. They may contain embedded spaces 
\item names of grammars and lexica are also used as file-names, so they
must obey the local restrictions on file names.
\end{itemize}
Names may also contain certain ornaments like {\tt \_}, {\tt @}, {\tt ?},
{\tt -}, and {\tt +}.
\begin{syntax}
nonterminal-name:
   identifier.
\end{syntax}
\begin{syntax}
grammar-name, lexicon-name:
   simple-identifier.
\end{syntax}
\begin{syntax}
affix-name:
   capital-identifier.
\end{syntax}
\begin{syntax}
affix-terminal:
   "+", affix-terminal;
   "-", affix-terminal;
   small-letter, small-letment-sequence-option.
\end{syntax}
\begin{syntax}
special-name:
   "\$", capital-identifier.
\end{syntax}
A {name} has to fit on a single line.
Provided the host operating system supports this, characters
with nonstandard codes (i.e.\ ASCII-codes above 127) will also be
accepted as letters and can be used for non-Latin characters.

\subsubsection{Identifiers}
Identifiers may contain \XXs{embedded}{space}.
Multiple {embedded space}s will be reduced to one space.

\begin{syntax}
identifier:
   simple-identifier;
   simple-identifier, embedded-space, identifier.

simple-identifier:
   letter, letment-sequence-option.

letment:
   letter;
   digit;
   ornament.

small-letment:
   small-letter;
   digit;
   ornament.

letter:
   capital-letter;
   small-letter.

ornament:
   "+", "-", "?", "@", "_".

embedded-space:
   " ", embedded-space-option.

capital-identifier:
   capital-letter;
   capital-identifier, capital-letter;
   capital-identifier, ornament.
\end{syntax}

\subsection{Text denotations}
A \XX{text}{denotation} is a {\em nonempty} text enclosed between
double quotes.
\begin{syntax}
text-denotation:
   double-quote, character-sequence, double-quote.
\end{syntax}

Every character within a character-sequence denotes itself,
except for certain \Xs{operator} and \XXs{escaped}{character}.
Within a text-denotation, a double quote should be denoted as \verb+\"+,
the sequence \verb+\-+ denotes the literal hyphen and a backslash is
written as \verb+\\+.  In addition, the sequence \verb+\n+ denotes the
end-of-line character, and \verb+\t+ denotes the tab character.  Every
other character following a backslash denotes the same character
without the backslash.

\subsection{Numbers}
A \X{number} is denoted by an optional minus sign followed by a sequence of decimal digits.
\begin{syntax}
number: sign option, digit-sequence.

sign option: "-"; "+"; .
\end{syntax}

\subsection{Comments}
A comment is a part of the input which should not be translated by the
system.
A comment starts with the comment-symbol \verb+#+ and comprizes the rest
of the input line.  It can be used at every place where a linebreak
can be inserted.

\subsection{Pragmats}\label{pragmats}
A \X{pragmat} is a comment whose meaning is known to the system and
which influences the system's behaviour (although it does not
alter the input language).
{Pragmat}s may occur at any place among the statements and definitions
in a grammar.
{Pragmat}s in the input of the parser generator influence its output
and therefore to some extent the behaviour of generated parsers.
{Pragmat}s differ from command-line options in that they form part of the
input and therefore do not have to be presented at every execution of the
system.

\begin{syntax}
pragmat:
  "\verb+|+", directive-list, "\verb+|+".

directive:
  "segment mode"; "line mode"
  "word terminators", text-denotation;
  "invisible characters", text-denotation;
  "invisible", nonterminal-spec-list;
  "translate", translate-spec.

nonterminal-spec-list:
  nonterminal name, "/", arity.

arity:
  number.

translate-spec:
  "default";
  "off";
  text-denotation.
\end{syntax}
Below is an example of pragmats from a grammar written for segment
parsing.
Furthermore, two grammar rules will not show in the parse tree:
the syntax rule with the name {\sf xmltags} with $0$ parameters
and the syntax rule {\sf agreement predicate} with $2$ parameters.

\begin{example}{1.8cm}
\begin{grammar}
    | segment mode |
    | invisible xmltags/0, agreement predicate/2 |
\end{grammar}
\end{example}
