org.netbeans.api.lexer

Interface TokenId

public interface TokenId

Identifier of a token (could also be called a token-type).
It is not a token, because it does not contain the text (also called image) of the token.

Token ids are typically defined as enums by the following pattern:

 public enum JavaTokenId implements TokenId {

     ERROR(null, "error"),
     IDENTIFIER(null, "identifier"),
     ABSTRACT("abstract", "keyword"),
     ...
     SEMICOLON(";", "separator"),
     ...


     private final String fixedText; // Used by lexer for production of flyweight tokens

     private final String primaryCategory;

     JavaTokenId(String fixedText, String primaryCategory) {
         this.fixedText = fixedText;
         this.primaryCategory = primaryCategory;
     }

     public String fixedText() {
         return fixedText;
     }

     public String primaryCategory() {
         return primaryCategory;
     }

 }
 

Token ids can also be generated (e.g. by lexer generation tools) by using LanguageHierarchy.newId(String,int,String) method.
All token ids of a language must have both unique ordinal and name.
Token name should be all uppercase while token categories should be named in lowercase.

Detailed information and rules for naming can be found in TokenId Naming.

Method Summary

Modifier and Type	Method and Description
String	name() Get name of this tokenId.
int	ordinal() Get integer identification of this tokenId.
String	primaryCategory() Get name of primary token category into which this token belongs.

Method Detail

name

String name()

Get name of this tokenId.

It can serve for several purposes such as finding a possible style information for the given token.

Returns:

non-null unique name of the TokenId. The name must be unique among other TokenId instances of the particular language where it is defined. The name should consist of uppercase alphanumeric letters and underscores only.

ordinal

int ordinal()

Get integer identification of this tokenId.

Returns:

numeric identification of this TokenId.
Ordinal must be a non-negative integer unique among all the tokenIDs inside the language where it is declared.
The ordinals are usually defined and adopted from lexer generator tool that generates the lexer for the given language.
They do not have to be consecutive.
On they other hand there should not be big gaps (e.g. 100 or more) because indexing arrays are constructed based on the ordinal values so the length of the indexing array corresponds to the highest ordinal of all the token ids declared for the particular language.
The intIds allow more efficient use of the tokenIds in switch-case statements.

primaryCategory

String primaryCategory()

Get name of primary token category into which this token belongs.
Other token categories for this id can be defined in the language hierarchy.

Returns:

name of the primary token category into which this token belongs or null if there is no primary category for this token.