org.opensolaris.opengrok.analysis
Class JFlexTokenizer

java.lang.Object
  extended by org.apache.lucene.util.AttributeSource
      extended by org.apache.lucene.analysis.TokenStream
          extended by org.apache.lucene.analysis.Tokenizer
              extended by org.opensolaris.opengrok.analysis.JFlexTokenizer
All Implemented Interfaces:
java.io.Closeable
Direct Known Subclasses:
CSymbolTokenizer, CxxSymbolTokenizer, FortranSymbolTokenizer, JavaSymbolTokenizer, LispSymbolTokenizer, PlainFullTokenizer, PlainSymbolTokenizer, ShSymbolTokenizer, TclSymbolTokenizer, TroffFullTokenizer

public abstract class JFlexTokenizer
extends org.apache.lucene.analysis.Tokenizer

this class was created because of lucene 2.4.1 update which introduced char[] in Tokens instead of String lucene 3.0.0 uses AttributeSource instead of Tokens to make things even easier :-D Generally this is a "template" for all new Tokenizers, so be carefull when changing it, it will impact almost ALL symbol tokenizers in OpenGrok ... Created on August 24, 2009


Nested Class Summary
 
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
org.apache.lucene.util.AttributeSource.AttributeFactory, org.apache.lucene.util.AttributeSource.State
 
Field Summary
protected  org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt
           
protected  org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute posIncrAtt
           
protected  org.apache.lucene.analysis.tokenattributes.TermAttribute termAtt
           
 
Fields inherited from class org.apache.lucene.analysis.Tokenizer
input
 
Constructor Summary
JFlexTokenizer()
           
 
Method Summary
 boolean incrementToken()
          This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ...
protected  void setAttribs(java.lang.String str, int start, int end)
           
abstract  boolean yylex()
           
 
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset, reset
 
Methods inherited from class org.apache.lucene.analysis.TokenStream
end, reset
 
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Field Detail

termAtt

protected org.apache.lucene.analysis.tokenattributes.TermAttribute termAtt

offsetAtt

protected org.apache.lucene.analysis.tokenattributes.OffsetAttribute offsetAtt

posIncrAtt

protected org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute posIncrAtt
Constructor Detail

JFlexTokenizer

public JFlexTokenizer()
Method Detail

yylex

public abstract boolean yylex()
                       throws java.io.IOException
Throws:
java.io.IOException

incrementToken

public boolean incrementToken()
                       throws java.io.IOException
This will reinitalize internal AttributeImpls, or it returns false if end of input Reader ...

Specified by:
incrementToken in class org.apache.lucene.analysis.TokenStream
Returns:
false if no more tokens, otherwise true
Throws:
java.io.IOException

setAttribs

protected void setAttribs(java.lang.String str,
                          int start,
                          int end)