FileTokenizer
, LineTokenizer
, StringTokenizer
, TokenFilter.FileTokenizer
, TokenFilter.StringTokenizer
public interface Tokenizer
Modifier and Type | Method | Description |
---|---|---|
java.lang.String |
getPostToken() |
return the string between tokens, after the
previous token.
|
java.lang.String |
getToken(java.io.Reader in) |
get the next token from the input stream
|
java.lang.String getToken(java.io.Reader in) throws java.io.IOException
in
- the input streamjava.io.IOException
- if an error occursjava.lang.String getPostToken()