TokenHierarchy class with
its static methods that provide its instance for the given input source.See: Description
| Interface | Description |
|---|---|
| TokenHierarchyListener |
Listener for chagnes in the token hierarchy.
|
| TokenId |
Identifier of a token (could also be called a token-type).
|
| Class | Description |
|---|---|
| InputAttributes |
Supplementary information about particular input
that may be used to influence the lexer's operation.
|
| Language<T extends TokenId> |
Language describes a set of token ids
that comprise the given language.
|
| LanguagePath |
Language path describes a complete embedding
of the languages starting from the root (top-level) language
till the most embedded language.
|
| Token<T extends TokenId> |
Token describes a lexical element of input text.
|
| TokenChange<T extends TokenId> |
Token change describes modification on one level of a token hierarchy.
|
| TokenHierarchy<I> |
Token hierarchy represents a given input source as a browsable hierarchy of tokens.
|
| TokenHierarchyEvent |
Description of the changes made in a token hierarchy.
|
| TokenSequence<T extends TokenId> |
Token sequence allows to iterate between tokens
of a token hierarchy.
|
| TokenUtilities |
Various utility methods related to token text.
|
| Enum | Description |
|---|---|
| PartType |
Whether
Token represents a complete token
or just a part of a complete token. |
| TokenHierarchyEventType |
Token hierarchy event type determines the reason
why token hierarchy modification described by
TokenHierarchyEvent
happened. |
TokenHierarchy class with
its static methods that provide its instance for the given input source.
TokenHierarchy.tokenSequence()
allows to walk through the tokens that were created for the given input.
Token models a single token recognized in the input.
It carries a token identification TokenId returned by
Token.id()
and token text represented as CharSequence
returned by Token.text().
Language represents
all the token ids for a single language.