Token (Lexer)

java.lang.Object
- org.netbeans.api.lexer.Token<T>

```
public abstract class Token<T extends TokenId>
extends Object
```
Token describes a lexical element of input text.
It mainly provides an identification by id() and a textual body (aka token's image) by text().
Only lexers should produce token instances and they should do it solely by using methods of TokenFactory.
Note: Do not create custom extensions of this class - lexers may only return implementations produced by TokenFactory. Creation of any other token implementations will be refused.

Token guarantees stability of the id() and length() methods. The hashCode() and equals(Object) methods use the default implementations from java.lang.Object.
The two tokens are only equal if they are the same object.

Constructor Summary

Constructors
Modifier Constructor and Description

protected Token()
Create token instance.

Constructors
Modifier	Constructor and Description
`protected`	`Token()` Create token instance.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`equals(Object o)` Make sure the default implementation of `equals()` is used and the token can safely be used in maps.
`abstract Object`	`getProperty(Object key)` Get extra property of this token.
`int`	`hashCode()` Make sure the default implementation of `hashCode()` is used and the token can safely be used in maps.
`abstract boolean`	`hasProperties()` Quickly determine whether this token has any extra properties.
`abstract T`	`id()` Get identification of this token.
`abstract boolean`	`isCustomText()` Check whether `text()` returns a custom value that may differ from the original content of the text input.
`abstract boolean`	`isFlyweight()` Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
`boolean`	`isRemoved()` Check whether this token is no longer part of the token hierarchy.
`List<? extends Token<T>>`	`joinedParts()` Get all token parts comprising this token ordered from lowest to highest part's offset.
`Token<T>`	`joinToken()` Get a complete token that is joined from multiple parts (this token is one of those parts).
`abstract int`	`length()` Get number of characters in the original text input that the token spans.
`abstract int`	`offset(TokenHierarchy<?> tokenHierarchy)` Get the offset at which this token is present in the input or `-1` if this token is flyweight (and therefore does not store offset).
`abstract PartType`	`partType()` Check whether this token represents a complete token or whether it's a particular part of a complete token.
`abstract CharSequence`	`text()` Get text of this token (aka token's image) as a character sequence.

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - Token
```
protected Token()
```
    Create token instance.
    
    Throws:
    
    IllegalStateException - if a non-lexer-module-implementation token is attempted to be created.
- Method Detail
  - id
```
public abstract T id()
```
    Get identification of this token.
    
    Returns:
    
    non-null identification of this token.
  - text
```
public abstract CharSequence text()
```
    Get text of this token (aka token's image) as a character sequence.
    This text usually corresponds to the characters present in the lexed text input unless isCustomText() returns true.
    Note for mutable input sources:
    This method should only be called within a readonly (or read-write) transaction over the underlying input source (such as javax.swing.text.Document.render() for Swing documents).
    The result returned by this method is only valid within a readonly (or read-write) transaction over the input source (method must be re-called during the next readonly transaction).
    
    Returns:
    
    non-null, non-empty text of this token. It may be null in case the token was used for a mutable input and it was removed from the token list for the given input (but even in such case the text can be retained in certain cases).
    The behavior of equals() and hashCode() of the returned character sequence is generally undefined.
    The returned character sequence can NOT be compared to another character sequence by using its equals() method.
    TokenUtilities contains utility methods related to token text comparing.
    
    The returned text is just a pointer to the primary source of the data e.g. a swing document. The character data are not duplicated in the tokens.
  - isCustomText
```
public abstract boolean isCustomText()
```
    Check whether text() returns a custom value that may differ from the original content of the text input.
    Using custom text may be useful in case when only certain part of the token is useful for the particular use and the token's text can be shrinked and possibly a flyweight text can be used.
    Also this is useful when using lexers generated by various lexer generators that generally allow to use a custom text in the produced tokens.
    
    Returns:
    
    true if the text of the token does not correspond to the original characters present in the text input being lexed.
  - length
```
public abstract int length()
```
    Get number of characters in the original text input that the token spans.
    Usually this is the same value like text().length() unless isCustomText() returns true.
    Also this method will return valid length in all cases even when the text of the token could become null.
    
    Returns:
    
    >=0 length of the token.
  - offset
```
public abstract int offset(TokenHierarchy<?> tokenHierarchy)
```
    Get the offset at which this token is present in the input or -1 if this token is flyweight (and therefore does not store offset).
    Note: Use of TokenSequence.offset() is usually preferred over this method because it returns actual offset even for the flyweight tokens.
    If necessary the flyweight token may be replaced by regular token by using TokenSequence.offsetToken().
    The complexity of the method should generally be constant regardless of the level of the language embedding.
    
    Parameters:
    
    tokenHierarchy - null should be passed (the parameter is reserved for future use when token hierarchy snapshots will be implemented).
    
    Returns:
    
    >=0 offset of the token in the input or -1 if this token is flyweight.
  - isFlyweight
```
public abstract boolean isFlyweight()
```
    Checks whether this token instance is used for multiple occurrences of this token in this or other inputs.
    For example keywords or operators are typically flyweight tokens while e.g. identifiers are not flyweight as their text generally varies.
    Flyweight tokens may decrease the memory consumption for the tokens considerably for frequently used tokens. For example a single space ' ' may be a useful flyweight token as it's used very often throughout a source. The decision of what tokens are made flyweight is upon the implementor of the particular language.
    If the token is flyweight its offset(TokenHierarchy) returns -1.
    
    Returns:
    
    true if the token is flyweight or false otherwise.
  - isRemoved
```
public boolean isRemoved()
```
    Check whether this token is no longer part of the token hierarchy.
    
    Returns:
    
    true if the token was removed from the token hierarchy or false if it's still present in the hierarchy. Flyweight tokens always return true from this method.
  - partType
```
public abstract PartType partType()
```
    Check whether this token represents a complete token or whether it's a particular part of a complete token.
    Some lexers may also use this information to express an incomplete token. For example an unclosed block comment at the end of java source is represented as a BLOCK_COMMENT token id and PartType.START.
    
    Returns:
    
    PartType.COMPLETE for regular token or other part types for particular token parts.
  - joinToken
```
public Token<T> joinToken()
```
    Get a complete token that is joined from multiple parts (this token is one of those parts).
    
    Returns:
    
    complete token or null if this token is not a part of any token.
  - joinedParts
```
public List<? extends Token<T>> joinedParts()
```
    Get all token parts comprising this token ordered from lowest to highest part's offset.
    It's guaranteed that each token part is continuous in the input text (there are no gaps inside the token part's text).
    On the other hand there may be textual gaps between two adajcent token parts.
    
    Returns:
    
    list of token parts or null if the token is continuous.
  - hasProperties
```
public abstract boolean hasProperties()
```
    Quickly determine whether this token has any extra properties.
  - getProperty
```
public abstract Object getProperty(Object key)
```
    Get extra property of this token.
    The token properties are defined by the lexer upon token creation. The clients of the API cannot set any property of the token.
    
    Parameters:
    
    key - non-null key of the property to get.
    
    Returns:
    
    non-null value of the property or null if the property does not have any value.
    
    See Also:
    
    hasProperties()
  - hashCode
```
public final int hashCode()
```
    Make sure the default implementation of hashCode() is used and the token can safely be used in maps.
    
    Overrides:
    
    hashCode in class Object
  - equals
```
public final boolean equals(Object o)
```
    Make sure the default implementation of equals() is used and the token can safely be used in maps.
    
    Overrides:
    
    equals in class Object

Class Token<T extends TokenId>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

Token

Method Detail

id

text

isCustomText

length

offset

isFlyweight

isRemoved

partType

joinToken

joinedParts

hasProperties

getProperty

hashCode

equals