Recognise a non-alphanumeric, non-whitespace character
Recognise a string literal, including escaped characters.
(Note: doesn't yet handle escape sequences such as \123)
Recognise a whitespace character
Recognise a sequence of at least one sub-lexers
Recognise a character that matches a predicate
Recognise any of the characters in the given string
Recognise a sequence of at zero or more sub-lexers. This is not
guaranteed to consume input
Given a mapping from lexers to token generating functions (the
TokenMap a) and an input string, return a list of recognised tokens,
and the line, column, and remainder of the input at the first point in the
string where there are no recognised tokens.
Recognise anything but the given character
Recognise a specific character
Recognise an integer literal (possibly with a '-' prefix)
Recognise a specific string
Recognise no input (doesn't consume any input)
Recognise a digit 0-9
Recognise a character literal, including escaped characters.
(Note: doesn't yet handle escape sequences such as \123)
Recognise any character
A mapping from lexers to the tokens they produce.
This is a list of pairs (Lexer, String -> tokenType)
For each Lexer in the list, if a substring in the input matches, run
the associated function to produce a token of type tokenType
A token, and the line and column where it was in the input
A language of token recognisers.
The consumes
flag is True is the recogniser is guaranteed to consume
at least one character
A token recogniser. Guaranteed to consume at least one character.
Alternative recognisers. If both consume, the combination is guaranteed
to consumer a character.
Sequence two recognisers. If either consumes a character, the sequence
is guaranteed to consume a character.