Agfl sentence generator
***********************

The generator is written by Doru Arfire <darfire@fastmail.fm>.

This program is made up of two parts: an AGFL grammar parser and loader, 
and the generator itself. I will discuss each of the two.

1. The AGFL grammar parser

The first task of the parser is to eliminate all the unnecessary characters
in the grammar file, so that the DCG parsing would work. This is done using
a simple finite automata transducer (transducer.pl) with null transitions.
This transducer eliminates commentaries, pragmas and spaces (including 
newlines and tabs). In case of an identifier composed of more than one word 
it leaves only one space between 2 words of the same identifier. It handles
correctly string between quotes and escaped characters.

This operation is done on a rule by rule basis, meaning that the transducer
reads one character at a time, interpreting it. When it encounters the end
of a rule (a '.' character), it returns the string obtained.

After eliminating all the extra characters the parser then "feeds" the 
string obtained to a DCG grammar which tries to fit that string into a rule.
If the result of this operation is positive, than the logical representation
of this rule is added to the internal Prolog database. The structure of
a syntax rule is:
synrule(Name, Type, Parameters, Weights, Input, Output, Alternatives), where:
Name - rule name
Type - rule type
Parameters - rule parameters
Weights - the sum of the weights of all alternatives
Input, Output - variables used during generation. These are similar to the 
hidden input and output variables of a DCG rule. In another words, Input
is the list of character codes obtained by appending to A the list Output,
where A is a list of generated codes that match this rule.
Alternatives - a list containing all the alternatives of this rule

Other types of AGFL statements (affix rule, root statement), have similar
representation.

Special AGFL constructs like option and group are transformed to syntax rules
with special names. Groups take the name group_<unique random string> while
options take the name option_<unique random string>. In addition the special
rules generated for options take another empty alternative, that has the 
weight equal to the sum of all the other alternative.

The final operation at this stage is to calculate the affix domains for all
the affix variables used by the grammar.

After loading all grammar files the parser loads lexicon files. For the sake
of efficiency this is done without a transducer, meaning that a lexicon entry
should respect the format: <text denotation><tab character><syntax heading>.

After loading all the lexicon entries, these are grouped into lexicon sets.
A lexicon set contains lexicon entries with the same syntax heading.

At this point the Prolog internal database contains a logical representation
of our grammar.

2. The AGFL grammar generator

The generator involves two random selection moments. The first one happens 
when we call a rule with a number of parameters, and there are several rule
definition with the same name and number of parameters, and for all of them
the parameters match. In this case one is randomly chosen, and in case it fails,
one of the others is randomly chosen, and so on, until one does not fail, or
all of them fail. At every selection, all definitions involved have the same
weight.

The second case when we do a random selection is when we have to choose between
several alternatives in the same rule definition. In this case we perform a
weighted random selection. If the selected alternative fails, one of the other
is chosen, until all fail or one of them is successfull.


TODO:
*Implement regular expressions. At this stage $MATCH and $SKIP are introduced
as they are in the final string. So, in case a verification is needed, they
should be replaced by string that match their regular expression
*Implement predefined affix domain support
*Test with other Prolog systems other than SWI
