org.netbeans.api.lexer

Class TokenHierarchy<I>

java.lang.Object

org.netbeans.api.lexer.TokenHierarchy<I>

public final class TokenHierarchy<I>
extends Object

Token hierarchy represents a given input source as a browsable hierarchy of tokens.
It's is an entry point into the Lexer API.
It allows to create token sequences for hierarchy exploration and watching for token changes by attaching the token hierarchy listeners.
The hierarchy may either be flat or it can be a tree if the corresponding language hierarchy contains language embeddings.

Method Summary

Modifier and Type	Method and Description
void	addTokenHierarchyListener(TokenHierarchyListener listener) Add listener for token changes inside this hierarchy.
static <I extends CharSequence,T extends TokenId> TokenHierarchy<I>	create(I inputText, boolean copyInputText, Language<T> language, Set<T> skipTokenIds, InputAttributes inputAttributes) Create token hierarchy for the given input text.
static <I extends CharSequence> TokenHierarchy<I>	create(I inputText, Language<?> language) Create token hierarchy for the given non-mutating input text (for example java.lang.String).
static <I extends Reader,T extends TokenId> TokenHierarchy<I>	create(I inputReader, Language<T> language, Set<T> skipTokenIds, InputAttributes inputAttributes) Create token hierarchy for the given reader.
List<TokenSequence<?>>	embeddedTokenSequences(int offset, boolean backwardBias) Gets the list of all embedded TokenSequences at the given offset.
static <D extends Document> TokenHierarchy<D>	get(D doc) Get or create mutable token hierarchy for the given swing document.
I	inputSource() Get input source providing text over which this token hierarchy was constructed.
boolean	isActive() Token hierarchy may be set inactive to release resources consumed by tokens.
boolean	isMutable() Whether input text of this token hierarchy is mutable or not.
Set<LanguagePath>	languagePaths() Get a set of language paths used by this token hierarchy.
void	removeTokenHierarchyListener(TokenHierarchyListener listener) Remove listener for token changes inside this hierarchy.
TokenSequence<?>	tokenSequence() Get token sequence of the top level language of the token hierarchy.
<T extends TokenId> TokenSequence<T>	tokenSequence(Language<T> language) Get token sequence of the top level of the language hierarchy only if it's of the given language.
List<TokenSequence<?>>	tokenSequenceList(LanguagePath languagePath, int startOffset, int endOffset) Get immutable list of token sequences with the given language path from this hierarchy.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

get

public static <D extends Document> TokenHierarchy<D> get(D doc)

Get or create mutable token hierarchy for the given swing document.
The document may define a top language by doing doc.putProperty("mimeType", mimeType) (a language defined for the given mime type will be searched and used) or by doing putProperty(Language.class, language). Otherwise the returned hierarchy will be inactive and TokenHierarchy.tokenSequence() will return null.
All the operations with the obtained token hierarchy must be done under document's read lock (or write lock).

Parameters:

doc - non-null swing text document for which the token hiearchy should be obtained.

Returns:

non-null token hierarchy.

create

public static <I extends CharSequence> TokenHierarchy<I> create(I inputText,
                                                                Language<?> language)

Create token hierarchy for the given non-mutating input text (for example java.lang.String).

See Also:

TokenHierarchy.create(CharSequence,boolean,Language,Set,InputAttributes)

create

public static <I extends CharSequence,T extends TokenId> TokenHierarchy<I> create(I inputText,
                                                                                  boolean copyInputText,
                                                                                  Language<T> language,
                                                                                  Set<T> skipTokenIds,
                                                                                  InputAttributes inputAttributes)

Create token hierarchy for the given input text.

Parameters:

inputText - input text containing the characters to tokenize.

copyInputText - true in case the content of the input will not be modified in the future so the created tokens can reference it.
false means that the text can change in the future and the tokens should not directly reference it. Instead copy of the necessary text from the input should be made and the original text should not be referenced.

language - language defining how the input will be tokenized.

skipTokenIds - set containing the token ids for which the tokens should not be created in the created token hierarchy.
null may be passed which means that no tokens will be skipped.
This applies to top level of the token hierarchy only (not to embedded tokens).
The provided set should be efficient enough - ideally created by e.g. Language.tokenCategoryMembers(String) or Language.merge(Collection,Collection).

inputAttributes - additional properties related to the input that may influence token creation or lexer operation for the particular language (such as version of the language to be used).

Returns:

non-null token hierarchy.

create

public static <I extends Reader,T extends TokenId> TokenHierarchy<I> create(I inputReader,
                                                                            Language<T> language,
                                                                            Set<T> skipTokenIds,
                                                                            InputAttributes inputAttributes)

Create token hierarchy for the given reader.

Parameters:

inputReader - input reader containing the characters to tokenize.

language - language defining how the input will be tokenized.

skipTokenIds - set containing the token ids for which the tokens should not be created in the created token hierarchy.
null may be passed which means that no tokens will be skipped.
This applies to top level of the token hierarchy only (not to embedded tokens).
The provided set should be efficient enough - ideally created by e.g. Language.tokenCategoryMembers(String) or Language.merge(Collection,Collection).

inputAttributes - additional properties related to the input that may influence token creation or lexer operation for the particular language (such as version of the language to be used).

Returns:

non-null token hierarchy.

tokenSequence

public TokenSequence<?> tokenSequence()

Get token sequence of the top level language of the token hierarchy.
For token hierarchies over mutable input sources the input source must be read-locked.
The token sequences for inner levels of the token hierarchy can be obtained by calling TokenSequence.embedded().

Returns:

token sequence of the top level of the token hierarchy or null if the token hierarchy is currently inactive (TokenHierarchy.isActive() returns false).

tokenSequence

public <T extends TokenId> TokenSequence<T> tokenSequence(Language<T> language)

Get token sequence of the top level of the language hierarchy only if it's of the given language.

Returns:

non-null token sequence or null if the hierarchy is active and its top level token sequence satisfies the condition (tokenSequence().language() == language).
Null is returned otherwise.

tokenSequenceList

public List<TokenSequence<?>> tokenSequenceList(LanguagePath languagePath,
                                                int startOffset,
                                                int endOffset)

Get immutable list of token sequences with the given language path from this hierarchy.
For mutable token hierarchies the method should only be invoked within read-locked input source. A new list should be obtained after each modification. ConcurrentModificationException may be thrown when iterating over (or retrieving items) from the obsolete list.
For forward exploration of the list the iterator is preferred over index-based iteration because the list contents can be constructed lazily.

Parameters:

languagePath - non-null language path that the obtained token sequences will all have.

startOffset - starting offset of the TSs to get. Use 0 for no limit. If the particular TS ends after this offset then it will be returned.

endOffset - ending offset of the TS to get. Use Integer.MAX_VALUE for no limit. If the particular TS starts before this offset then it will be returned.

Returns:

non-null list of TokenSequences or null if the token hierarchy is inactive (TokenHierarchy.isActive() returns false).

embeddedTokenSequences

public List<TokenSequence<?>> embeddedTokenSequences(int offset,
                                                     boolean backwardBias)

Gets the list of all embedded TokenSequences at the given offset. This method will use the top level TokenSequence in this hierarchy to drill down through the token at the specified offset and all its possible embedded sub-sequences.

If the offset lies at the border between two tokens the backwardBias parameter will be used to choose either the token on the left hand side (backwardBias == true) of the offset or on the right hand side (backwardBias == false).

For token hierarchies over mutable input sources this method must only be invoked within a read-lock over the mutable input source.

Parameters:

offset - The offset to look at.

backwardBias - If true the backward lying token will be used in case that the offset specifies position between two tokens. If false the forward lying token will be used.

Returns:

The list of all sequences embedded at the given offset. The list may be empty if there are no tokens in the top level TokenSequence at the given offset and in the specified direction or if the token hierarchy is inactive (TokenHierarchy.isActive() returns false). The sequences in the list are ordered from the top level sequence to the bottom one.

Since:

1.20

languagePaths

public Set<LanguagePath> languagePaths()

Get a set of language paths used by this token hierarchy.
The set includes "static" paths that are those reachable by traversing token ids of the top language and searching for the default embeddings that could be created by LanguageHierarchy.embedding(Token,LanguagePath,InputAttributes).

For token hierarchies over mutable input sources this method must only be invoked within a read-lock over the mutable input source.

Returns:

non-null set of language paths. The set will be empty if the token hierarchy is inactive (TokenHierarchy.isActive() returns false).

isMutable

public boolean isMutable()

Whether input text of this token hierarchy is mutable or not.

Returns:

true if the input text is mutable or false otherwise.

inputSource

public I inputSource()

Get input source providing text over which this token hierarchy was constructed.
It may be CharSequence or Reader or a mutable input source such as swing text document Document.

Returns:

non-null input source.

isActive

public boolean isActive()

Token hierarchy may be set inactive to release resources consumed by tokens.
Only token hierarchies over a mutable input can become inactive.
When inactive the hierarchy does not hold any tokens and TokenHierarchy.tokenSequence() return null.

For token hierarchies over mutable input sources this method must only be invoked within a read-lock over the mutable input source.

Returns:

true if valid tokens exist for this hierarchy or false if the token hierarchy is inactive and there are currently no active tokens to represent it.

addTokenHierarchyListener

public void addTokenHierarchyListener(TokenHierarchyListener listener)

Add listener for token changes inside this hierarchy.

Parameters:

listener - token change listener to be added.

removeTokenHierarchyListener

public void removeTokenHierarchyListener(TokenHierarchyListener listener)

Remove listener for token changes inside this hierarchy.

Parameters:

listener - token change listener to be removed.

toString

public String toString()

Overrides:

toString in class Object