public interface TokenValidator<T extends TokenId>
Token validation is part of fine-tuning of the lexing
and should be considered for all tokens that may have significant length
such as whitespace or comments.
The advantage of validation is that compared to lexing it typically only explores the modified characters and few adjacent characters.
|Modifier and Type||Method and Description|
This method is invoked in mutable environments prior lexer invocation to check whether token in which the text modification occurred was only slightly modified by the performed modification and the lexer's invocation is not necessary.
Token<T> validateToken(Token<T> token, TokenFactory<T> factory, CharSequence tokenText, int modRelOffset, int removedLength, CharSequence removedText, int insertedLength, CharSequence insertedText)
token- non-null token affected by the modification. The token's text is undefined and must not be retrieved from the token at this time.
factory- non-null for producing of the new token to be returned.
tokenText- non-null text of the token already affected by the modification.
modRelOffset- >0 offset of the text removal/insertion inside the token.
insertedLength- >0 length of the inserted text.