AbstractStreamTokenizer (API documentation of hlt.language packages)
Date: 2013-01-30

hlt.language.io

## Class AbstractStreamTokenizer

- java.lang.Object
- - hlt.language.io.AbstractStreamTokenizer

- Direct Known Subclasses:
    LAStreamTokenizer, StreamTokenizer

      public abstract class AbstractStreamTokenizer
      extends java.lang.Object

  There are two concrete subclasses of this abstract class: `hlt.language.io.StreamTokenizer` and `hlt.language.io.LAStreamTokenizer`. They provide the exact same interface as that of the standard Java Core API's `java.io.StreamTokenizer` but with a simpler (and IMHO sounder) implementation. One may use both subclasses instead of `java.io.StreamTokenizer` without change. However, they support a more complete and more general set of methods. In addition, `hlt.language.io.LAStreamTokenizer` provides a method for arbitrary lookaheads:

       public final String lookahead (int n) throws IOException

  which returns the string of `n` characters in the input stream including and starting with the current character.

  The syntax configuration used as the default setting of the tokenizer is such that:

  - the type precedence ordering is set to: `WORD`, `WHITESPACE`, `COMMENT`, `QUOTE`, and `ORDINARY`;
  - control characters and space are `WHITESPACE` characters;
  - letters and numerals are `WORD` characters;
  - `_` and `.` are `WORD` characters;
  - single quote and double quote are quote characters;
  - end-of-line is not significant;
  - white spaces are not significant;
  - all other characters are ordinary;
  - parsing numbers is effective;
  - C-style and C++-style comments are recognized and skipped.
  This default configuration is reinstated with `setDefaultSyntax()`.
  See Also:
    `StreamTokenizer`, `LAStreamTokenizer`

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static byte` | `COMMENT`
    `boolean` | `isInteger` This is set to true whenever an integer number has been read.
    `static byte` | `NUMERIC`
    `double` | `nval` This contains the token value whenever a number has been read.
    `static byte` | `ORDINARY` The following are public constants denoting recognized token types.
    `static byte` | `QUOTE`
    `static byte` | `SPECIAL`
    `java.lang.String` | `sval` This contains the token string when a word, or a quoted word, has been read.
    `static int` | `TT_EOF` This value indicates that the end of file has been read.
    `static int` | `TT_EOL` This value indicates that an end of line has been read.
    `static int` | `TT_NOTHING` This value indicates that no token has been read yet.
    `static int` | `TT_NUMBER` This value indicates that a number has been read.
    `static int` | `TT_SPECIAL` This value indicates that a special token has been returned.
    `static int` | `TT_WORD` This value indicates that a word has been read.
    `int` | `ttype` This always contains the type of token just read.
    `static byte` | `WHITESPACE`
    `static byte` | `WORD`

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `AbstractStreamTokenizer()`

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `abstract void` | `commentChar(int c)` Makes the specified character a comment start character exclusively.
    `int` | `currentChar()` returns the current character.
    `abstract void` | `disableComments()`
    `abstract void` | `enableComments()`
    `void` | `eolIsSignificant(boolean flag)` The end-of-line is recognized as a token iff the argument is `true`.
    `java.lang.String` | `getFile()` Returns the current file if any, or `null`.
    `int` | `getLineNumber()` Returns the current line number.
    `void` | `ignoreNonDecimals()` Specifies that octal and hexadecimal numbers should not be parsed.
    `boolean` | `isNumericChar(int c)` Checks whether this character's type is numeric, among other types.
    `boolean` | `isOrdinaryChar(int c)` Checks whether this character's type is ordinary, among other types.
    `boolean` | `isQuoteChar(int c)` Checks whether this character's type is quote, among other types.
    `boolean` | `isWhitespaceChar(int c)` Checks whether this character's type is whitespace, among other types.
    `boolean` | `isWordChar(int c)` Checks whether this character's type is word, among other types.
    `int` | `lineno()` Returns the current line number.
    `java.lang.String` | `location()`
    `abstract int` | `nextToken()` Reads a token and returns its type, which is also the value of `ttype`.
    `void` | `ordinaryChar(int c)` Makes the specified character ordinary exclusively.
    `void` | `ordinaryChars(int from, int to)` Makes all characters between the first and second arguments inclusive ordinary exclusively.
    `void` | `ordinaryChars(java.lang.String chars)` Sets all characters in this string to be ordinary characters, exclusively.
    `void` | `parseDigitsOnly()` Sets this tokenizer to parse numbers, but only whole numbers made out of maximal sequences of numerical digits, considering as non-numeric characters the `'.'`, `'+'`, `'-'`, as well as letters (i.e., `'e'`, `'E'`, `'f'`, `'F'`, `'l'`, `'L'`, `'d'`, or `'D'`))
    `void` | `parseNonDecimals()` Specifies that octal and hexadecimal numbers should be parsed.
    `void` | `parseNonDecimals(boolean flag)`
    `void` | `parseNumbers()` Specifies that numbers should be parsed.
    `void` | `parseNumbers(boolean flag)`
    `boolean` | `parsingDigitsOnly()`
    `int` | `peek()` Returns the next character to be read.
    `int` | `peekBack()` Returns the previous character that was read.
    `void` | `pushBack()` If called, the next call to `NextToken()` will return the same token again.
    `void` | `quoteChar(int c)` Makes the specified character a quote character exclusively.
    `void` | `quoteChar(int c, int e)` Makes the specified character a quote character exclusively, and specifies its escape character.
    `void` | `quotePair(int left, int right)` Same as above with implicit default escape character.
    `void` | `quotePair(int left, int right, int escape)` Defines a pair of characters as a pair of left and right quotes.
    `void` | `resetSyntax()` Make all characters ordinary, do not parse numbers, and do not recognize comments.
    `void` | `setDefaultSyntax()` Resets the syntax configuration of the tokenizer to the default setting.
    `void` | `setOrdinaryChar(int c)` Sets this character to be an ordinary character, non-exclusively.
    `void` | `setOrdinaryChars(java.lang.String chars)` Sets all characters in this string to be ordinary characters, non-exclusively.
    `void` | `setQuoteChar(int c)` Same as above with implicit default escape character.
    `void` | `setQuoteChar(int c, int e)` Sets this character to be a quote character, non-exclusively, and specifies its escape character.
    `void` | `setQuotePair(int left, int right, int escape)` Same as `quotePair` but `left`'s `QUOTE_TYPE` is non-exclusive.
    `void` | `setType(int c, int type)` Set the type of this character to be the one specified, non-exclusively.
    `void` | `setType(int low, int high, int type)` Set the type of all characters in this range to be the one specified, non-exclusively.
    `void` | `setType(java.lang.String chars, int type)` Set the type of all characters in this string to be the one specified, non-exclusively.
    `void` | `setWhitespaceChar(int c)` Sets this character to be a whitespace character, non-exclusively.
    `void` | `setWhitespaceChars(java.lang.String chars)` Sets all characters in this string to be whitespace characters, non-exclusively.
    `void` | `setWordChar(int c)` Sets this character to be a word character, non-exclusively.
    `void` | `setWordChars(java.lang.String chars)` Sets all characters in this string to be word characters, non-exclusively.
    `void` | `skipChar()` A public method for skipping the current character in the input.
    `void` | `skipChar(boolean flag)` Same as `skipChar()`, but if the specified flag is `false` (resp., `true`) resets the start (resp., end) location of the current token.
    `abstract void` | `slashSlashComments(boolean flag)` C++-style comments will be skipped iff the argument is `true`.
    `abstract void` | `slashStarComments(boolean flag)` C-style comments will be skipped iff the argument is `true`.
    `void` | `spaceIsSignificant(boolean flag)` White spaces are recognized as tokens iff the argument is `true`.
    `Location` | `tokenEnd()` Returns the location of the end of the token at hand.
    `Location` | `tokenStart()` Returns the location of the start of the token at hand.
    `java.lang.String` | `toString()` Returns a printable value of the current state of this `StreamTokenizer`.
    `void` | `unsetType(int c, int type)` Unset the specified type, but no other one, for this character.
    `void` | `unsetType(int low, int high, int type)` Unset the specified type, but no other one, for all characters in this range.
    `void` | `unsetType(java.lang.String chars, int type)` Unset the specified type, but no other one, for all characters in this string.
    `void` | `whitespaceChar(int c)` Sets this character to be a whitespace character, exclusively.
    `void` | `whitespaceChars(int from, int to)` Makes all characters between the first and second arguments inclusive white space characters exclusively.
    `void` | `whitespaceChars(java.lang.String chars)` Sets all characters in this string to be whitespace characters, exclusively.
    `void` | `wordChar(int c)` Sets this character to be a word character, exclusively.
    `void` | `wordChars(int from, int to)` Makes all characters between the first and second arguments inclusive word characters exclusively.
    `void` | `wordChars(java.lang.String chars)` Sets all characters in this string to be word characters, exclusively.

    - ### Methods inherited from class java.lang.Object

      `equals, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### ORDINARY

          public static final byte ORDINARY

      The following are public constants denoting recognized token types.
      See Also:
        Constant Field Values

    - #### WHITESPACE

          public static final byte WHITESPACE

      See Also:
        Constant Field Values

    - #### QUOTE

          public static final byte QUOTE

      See Also:
        Constant Field Values

    - #### WORD

          public static final byte WORD

      See Also:
        Constant Field Values

    - #### NUMERIC

          public static final byte NUMERIC

      See Also:
        Constant Field Values

    - #### COMMENT

          public static final byte COMMENT

      See Also:
        Constant Field Values

    - #### SPECIAL

          public static final byte SPECIAL

      See Also:
        Constant Field Values

    - #### TT_EOF

          public static final int TT_EOF

      This value indicates that the end of file has been read.
      See Also:
        Constant Field Values

    - #### TT_EOL

          public static final int TT_EOL

      This value indicates that an end of line has been read. This happens iff `eolIsSignificant` has been set to `true`.
      See Also:
        Constant Field Values

    - #### TT_WORD

          public static final int TT_WORD

      This value indicates that a word has been read. The field `sval` contains the string. All alphabetical letters are word characters by default. So are the numerals '`0`' to '`9`' and the characters '`_`', and '`.`'.
      See Also:
        Constant Field Values

    - #### TT_NUMBER

          public static final int TT_NUMBER

      This value indicates that a number has been read. The field `sval` contains the number as a `double`.

      This happens whenever `parseNumbers()` (or `parseNumbers(true)`) has been called (default) and `parseNumbers(false)` has not yet been called. Note that the characters '`0`' to '`9`' and the characters '`-`', '`+`', and '`.`' are always considered numeric when parsing numbers is on.

      The format of numbers recognized is that of Java, including octal (`0...`) and hexadecimal (`0x...` or `0X...`) integers, and complete floating point format. Type letter suffixes (i.e., '`l`' or '`L`' for integers, and '`f`', '`F`', '`d`', '`D`' for floating point numbers, are also recognized, but they are ignored since the value returned is always a `double`.``

      See Also:
        Constant Field Values

    - #### TT_NOTHING

          public static final int TT_NOTHING

      This value indicates that no token has been read yet.
      See Also:
        Constant Field Values

    - #### TT_SPECIAL

          public static final int TT_SPECIAL

      This value indicates that a special token has been returned.
      See Also:
        Constant Field Values

    - #### ttype

          public int ttype

      This always contains the type of token just read. It is equal to:
      - `TT_NOTHING`, if nothing has been read yet.
      - `TT_EOF`, if the end of file has been read.
      - `TT_EOL`, if the end of line is significant and has been read.
      - `TT_NUMBER`, if numbers are parsed and a number has been read.
      - `TT_WORD`, is a word has been read. Any character can be declared to be a word character with `wordChar(...)`.
      - a quote character, if a quoted word has been read. The word value is then in `sval`. The single quote character and the double quote character are quote characters by default. Any character can be declared to be a quote character with `quoteChar(...)`. Distinct left/right quotes can also be defined with `quotePair(...)`.
      - An ordinary character, otherwise. Any character can be declared to be an ordinary character with `ordinaryChar(...)`.
      - `TT_SPECIAL`, if nothing else above applies.

    - #### sval

          public java.lang.String sval

      This contains the token string when a word, or a quoted word, has been read.

    - #### nval

          public double nval

      This contains the token value whenever a number has been read.

    - #### isInteger

          public boolean isInteger

      This is set to true whenever an integer number has been read.

  - ### Constructor Detail

    - #### AbstractStreamTokenizer

          public AbstractStreamTokenizer()

  - ### Method Detail

    - #### commentChar

          public abstract void commentChar(int c)

      Makes the specified character a comment start character exclusively.

    - #### slashSlashComments

          public abstract void slashSlashComments(boolean flag)

      C++-style comments will be skipped iff the argument is `true`.

    - #### slashStarComments

          public abstract void slashStarComments(boolean flag)

      C-style comments will be skipped iff the argument is `true`.

    - #### nextToken

          public abstract int nextToken()
                                 throws java.io.IOException

      Reads a token and returns its type, which is also the value of `ttype`.
      Throws:
        `java.io.IOException`

    - #### enableComments

          public abstract void enableComments()

    - #### disableComments

          public abstract void disableComments()

    - #### tokenStart

          public Location tokenStart()

      Returns the location of the start of the token at hand.

    - #### tokenEnd

          public Location tokenEnd()

      Returns the location of the end of the token at hand.

    - #### getLineNumber

          public final int getLineNumber()

      Returns the current line number.

    - #### lineno

          public final int lineno()

      Returns the current line number. For compatibility with `java.io.StreamTokenizer`.

    - #### getFile

          public final java.lang.String getFile()

      Returns the current file if any, or `null`.

    - #### location

          public java.lang.String location()

    - #### ordinaryChars

          public final void ordinaryChars(int from,
                           int to)

      Makes all characters between the first and second arguments inclusive ordinary exclusively.

    - #### wordChars

          public final void wordChars(int from,
                       int to)

      Makes all characters between the first and second arguments inclusive word characters exclusively.

    - #### whitespaceChars

          public final void whitespaceChars(int from,
                             int to)

      Makes all characters between the first and second arguments inclusive white space characters exclusively.

    - #### quoteChar

          public final void quoteChar(int c)

      Makes the specified character a quote character exclusively.

    - #### quoteChar

          public final void quoteChar(int c,
                       int e)

      Makes the specified character a quote character exclusively, and specifies its escape character.

    - #### ordinaryChar

          public final void ordinaryChar(int c)

      Makes the specified character ordinary exclusively.

    - #### eolIsSignificant

          public final void eolIsSignificant(boolean flag)

      The end-of-line is recognized as a token iff the argument is `true`.

    - #### spaceIsSignificant

          public final void spaceIsSignificant(boolean flag)

      White spaces are recognized as tokens iff the argument is `true`.

    - #### parseNumbers

          public final void parseNumbers()

      Specifies that numbers should be parsed.

    - #### parseNonDecimals

          public final void parseNonDecimals()

      Specifies that octal and hexadecimal numbers should be parsed.

    - #### ignoreNonDecimals

          public final void ignoreNonDecimals()

      Specifies that octal and hexadecimal numbers should not be parsed.

    - #### parseDigitsOnly

          public final void parseDigitsOnly()

      Sets this tokenizer to parse numbers, but only whole numbers made out of maximal sequences of numerical digits, considering as non-numeric characters the `'.'`, `'+'`, `'-'`, as well as letters (i.e., `'e'`, `'E'`, `'f'`, `'F'`, `'l'`, `'L'`, `'d'`, or `'D'`))

    - #### parsingDigitsOnly

          public final boolean parsingDigitsOnly()

    - #### parseNumbers

          public final void parseNumbers(boolean flag)

    - #### parseNonDecimals

          public final void parseNonDecimals(boolean flag)

    - #### pushBack

          public final void pushBack()

      If called, the next call to `NextToken()` will return the same token again.

    - #### toString

          public java.lang.String toString()

      Returns a printable value of the current state of this `StreamTokenizer`.
      Overrides:
        `toString` in class `java.lang.Object`

    - #### quotePair

          public final void quotePair(int left,
                       int right,
                       int escape)

      Defines a pair of characters as a pair of left and right quotes. This allows tokenizing words quoted by two different characters. Note that only the left quote is categorized as `QUOTE_TYPE`. The third argument specifies the escape character: if 0, no escape is allowed within these quotes.

    - #### quotePair

          public final void quotePair(int left,
                       int right)

      Same as above with implicit default escape character.

    - #### setQuotePair

          public final void setQuotePair(int left,
                          int right,
                          int escape)

      Same as `quotePair` but `left`'s `QUOTE_TYPE` is non-exclusive.

    - #### setQuoteChar

          public final void setQuoteChar(int c)

      Same as above with implicit default escape character.

    - #### setQuoteChar

          public final void setQuoteChar(int c,
                          int e)

      Sets this character to be a quote character, non-exclusively, and specifies its escape character.

    - #### setOrdinaryChar

          public final void setOrdinaryChar(int c)

      Sets this character to be an ordinary character, non-exclusively.

    - #### wordChar

          public final void wordChar(int c)

      Sets this character to be a word character, exclusively.

    - #### setWordChar

          public final void setWordChar(int c)

      Sets this character to be a word character, non-exclusively.

    - #### wordChars

          public final void wordChars(java.lang.String chars)

      Sets all characters in this string to be word characters, exclusively.

    - #### setWordChars

          public final void setWordChars(java.lang.String chars)

      Sets all characters in this string to be word characters, non-exclusively.

    - #### whitespaceChar

          public final void whitespaceChar(int c)

      Sets this character to be a whitespace character, exclusively.

    - #### setWhitespaceChar

          public final void setWhitespaceChar(int c)

      Sets this character to be a whitespace character, non-exclusively.

    - #### whitespaceChars

          public final void whitespaceChars(java.lang.String chars)

      Sets all characters in this string to be whitespace characters, exclusively.

    - #### setWhitespaceChars

          public final void setWhitespaceChars(java.lang.String chars)

      Sets all characters in this string to be whitespace characters, non-exclusively.

    - #### ordinaryChars

          public final void ordinaryChars(java.lang.String chars)

      Sets all characters in this string to be ordinary characters, exclusively.

    - #### setOrdinaryChars

          public final void setOrdinaryChars(java.lang.String chars)

      Sets all characters in this string to be ordinary characters, non-exclusively.

    - #### isOrdinaryChar

          public final boolean isOrdinaryChar(int c)

      Checks whether this character's type is ordinary, among other types.

    - #### isNumericChar

          public final boolean isNumericChar(int c)

      Checks whether this character's type is numeric, among other types.

    - #### isWordChar

          public final boolean isWordChar(int c)

      Checks whether this character's type is word, among other types.

    - #### isWhitespaceChar

          public final boolean isWhitespaceChar(int c)

      Checks whether this character's type is whitespace, among other types.

    - #### isQuoteChar

          public final boolean isQuoteChar(int c)

      Checks whether this character's type is quote, among other types.

    - #### peek

          public final int peek()

      Returns the next character to be read.

    - #### peekBack

          public final int peekBack()

      Returns the previous character that was read.

    - #### skipChar

          public final void skipChar()
                              throws java.io.IOException

      A public method for skipping the current character in the input.
      Throws:
        `java.io.IOException`

    - #### skipChar

          public final void skipChar(boolean flag)
                              throws java.io.IOException

      Same as `skipChar()`, but if the specified flag is `false` (resp., `true`) resets the start (resp., end) location of the current token.
      Throws:
        `java.io.IOException`

    - #### setType

          public final void setType(int c,
                     int type)

      Set the type of this character to be the one specified, non-exclusively.

    - #### setType

          public final void setType(int low,
                     int high,
                     int type)

      Set the type of all characters in this range to be the one specified, non-exclusively.

    - #### setType

          public final void setType(java.lang.String chars,
                     int type)

      Set the type of all characters in this string to be the one specified, non-exclusively.

    - #### unsetType

          public final void unsetType(int c,
                       int type)

      Unset the specified type, but no other one, for this character.

    - #### unsetType

          public final void unsetType(int low,
                       int high,
                       int type)

      Unset the specified type, but no other one, for all characters in this range.

    - #### unsetType

          public final void unsetType(java.lang.String chars,
                       int type)

      Unset the specified type, but no other one, for all characters in this string.

    - #### currentChar

          public final int currentChar()

      returns the current character.

    - #### resetSyntax

          public final void resetSyntax()

      Make all characters ordinary, do not parse numbers, and do not recognize comments.

    - #### setDefaultSyntax

          public final void setDefaultSyntax()

      Resets the syntax configuration of the tokenizer to the default setting. Namely:
      - set the type precedence ordering to: `WORD`, `WHITESPACE`, `COMMENT`, `QUOTE`, and `ORDINARY`;
      - make all control characters and space `WHITESPACE` characters;
      - make all letters and numerals `WORD` characters;
      - make `_` and `.` `WORD` characters;
      - make `'` and `"` quote characters (`'` for characters and character octal codes and unicode, and `"` for strings);
      - make end-of-line not significant;
      - make white spaces not significant;
      - make all other characters ordinary;
      - make parsing numbers effective;
      - make skipping C-style and C++-style comments effective.