Module Token: tokens for grammars

 

type t =
    Tterm of string
  | Tident of string
  | Tlident of string
  | Tuident of string
  | Tint of string
  | Tfloat of string
  | Tstring of string
  | Tchar of char
  | Tquotation of (string * string)
  | Tantiquot of string * string
  | Tlocate of (int * string)
  | Teoi
;;

The type for tokens, generated by lexers. They have canonical names but their meaning depend on the lexer's decision. See the module Plexer, a standard lexer provided.

exception Error of string;;

An lexing error exception to be used by lexers.

type pattern =
    P_TERM of string
  | P_IDENT of string option
  | P_LIDENT of string option
  | P_UIDENT of string option
  | P_INT of string option
  | P_FLOAT
  | P_STRING
  | P_CHAR
  | P_QUOTATION
  | P_ANTIQUOT of string
  | P_LOCATE
  | P_EOI
;;

Token patterns. Generated by the instruction EXTEND in entries rules productions. Each token pattern matches the corresponding token, of type t:
* P_TERM s matches Tterm s
* P_IDENT (Some s) matches Tident s
* P_IDENT None matches Tident s for any s
* P_LIDENT (Some s) matches Tlident s
* P_LIDENT None matches Tlident s for any s
* P_UIDENT (Some s) matches Tuident s
* P_UIDENT None matches Tuident s for any s
* P_INT (Some s) matches Tint s
* P_INT None matches Tint s for any s
* P_FLOAT matches Tfloat s for any s
* P_STRING matches Tstring s for any s
* P_CHAR matches Tchar c for any c
* P_QUOTATION matches Tquotation s1 s2 for any s1 and s2
* P_ANTIQUOT s1 matches Tantiquot s1 s2 for any s2
* P_LOCATE matches Tlocate i s for any i and s
* P_EOI matches Teoi.

type location_function = int -> int * int;;

The type for a function associating a number of a token in a stream (starting from 0) to its source location.

type lexer =
  { func : char Stream.t -> t Stream.t * location_function;
    add_keyword : string -> unit;
    remove_keyword : string -> unit;
    text : pattern -> string }
;;

The type for a lexer used by Camlp4 grammars.
The field func is the lexer function. The character stream is the input stream to be lexed. The result is a pair of a tokens stream and a location function for this tokens stream.
The field add_keyword is a function adding a new keyword in the lexer. The lexer can test whether this keyword respects its rules. Called by the function Grammar.extend, generated by the instruction EXTEND.
The field remove_keyword is a function removing a keyword from the lexer. Called by the function Grammar.delete_rule, generated by the instruction DELETE_RULE.
The field text returns the name of some token pattern, used in error messages.