org.netbeans.spi.lexer

Interface TokenValidator<T extends TokenId>

public interface TokenValidator<T extends TokenId>

Token validator checks whether an existing token was affected by just performed input source modification so that it needs to be relexed.
If the modification was limited to a single non-flyweight token and the token validator exists for a particular token id then the token validation is attempted.
Token validator can refuse validation by returning null from its only method if the modification affects the token or if the validation is unsure.

Token validation is part of fine-tuning of the lexing and should be considered for all tokens that may have significant length such as whitespace or comments.
The advantage of validation is that compared to lexing it typically only explores the modified characters and few adjacent characters.

Method Summary

Modifier and Type

Method and Description

Token<T>

validateToken(Token<T> token, TokenFactory<T> factory, CharSequence tokenText, int modRelOffset, int removedLength, CharSequence removedText, int insertedLength, CharSequence insertedText) This method is invoked in mutable environments prior lexer invocation to check whether token in which the text modification occurred was only slightly modified by the performed modification and the lexer's invocation is not necessary.

Method Detail

validateToken

Token<T> validateToken(Token<T> token,
                       TokenFactory<T> factory,
                       CharSequence tokenText,
                       int modRelOffset,
                       int removedLength,
                       CharSequence removedText,
                       int insertedLength,
                       CharSequence insertedText)

This method is invoked in mutable environments prior lexer invocation to check whether token in which the text modification occurred was only slightly modified by the performed modification and the lexer's invocation is not necessary.
Typically the token can be validated by returning the token with the same token id (just with different length that can be determined by tokenText.length()).
But the validator can also return a token with different token id (e.g. the identifier can become a keyword after the modification).

Parameters:

token - non-null token affected by the modification. The token's text is undefined and must not be retrieved from the token at this time.

factory - non-null for producing of the new token to be returned.

tokenText - non-null text of the token already affected by the modification.

modRelOffset - >0 offset of the text removal/insertion inside the token.

insertedLength - >0 length of the inserted text.

Returns:

a new token instance produced by the token factory.
Null should be returned if the token must be relexed or if the validator is unsure whether it's able to resolve the situation properly.