Module org.apache.lucene.analysis.common

Package org.apache.lucene.analysis.synonym

Class SynonymGraphFilter

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.TokenFilter

org.apache.lucene.analysis.synonym.SynonymGraphFilter

All Implemented Interfaces:

Closeable, AutoCloseable, Unwrappable<TokenStream>

public final class SynonymGraphFilter
extends TokenFilter

Applies single- or multi-token synonyms from a SynonymMap to an incoming TokenStream, producing a fully correct graph output. This is a replacement for SynonymFilter, which produces incorrect graphs for multi-token synonyms.

However, if you use this during indexing, you must follow it with FlattenGraphFilter to squash tokens on top of one another like SynonymFilter, because the indexer can't directly consume a graph. To get fully correct positional queries when your synonym replacements are multiple tokens, you should instead apply synonyms using this TokenFilter at query time and translate the resulting graph to a TermAutomatonQuery e.g. using TokenStreamToTermAutomatonQuery.

NOTE: this cannot consume an incoming graph; results will be undefined.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
(package private) static class	SynonymGraphFilter.BufferedInputToken
(package private) static class	SynonymGraphFilter.BufferedOutputToken

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

AttributeSource.State

Field Summary

Fields

Modifier and Type	Field	Description
private final ByteArrayDataInput	bytesReader
private int	captureCount
private boolean	finished
private final FST<BytesRef>	fst
private final FST.BytesReader	fstReader
private final boolean	ignoreCase
private int	lastNodeOut
private boolean	liveToken
private RollingBuffer<SynonymGraphFilter.BufferedInputToken>	lookahead
private int	lookaheadNextRead
private int	lookaheadNextWrite
private int	matchEndOffset
private int	matchStartOffset
private int	maxLookaheadUsed
private int	nextNodeOut
private final OffsetAttribute	offsetAtt
private final LinkedList<SynonymGraphFilter.BufferedOutputToken>	outputBuffer
private final PositionIncrementAttribute	posIncrAtt
private final PositionLengthAttribute	posLenAtt
private final FST.Arc<BytesRef>	scratchArc
private final BytesRef	scratchBytes
private final CharsRefBuilder	scratchChars
private final SynonymMap	synonyms
private final CharTermAttribute	termAtt
static final String	TYPE_SYNONYM
private final TypeAttribute	typeAtt

Fields inherited from class org.apache.lucene.analysis.TokenFilter

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor	Description
SynonymGraphFilter(TokenStream input, SynonymMap synonyms, boolean ignoreCase)	Apply previously built synonyms to incoming tokens.

Method Summary

Modifier and Type	Method	Description
private void	bufferOutputTokens(BytesRef bytes, int matchInputLength)	Expands the output graph into the necessary tokens, adding synonyms as side paths parallel to the input tokens, and buffers them in the output token buffer.
private void	capture()	Buffers the current input token into lookahead buffer.
(package private) int	getCaptureCount()
(package private) int	getMaxLookaheadUsed()
boolean	incrementToken()	Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
private boolean	parse()	Scans the next input token(s) to see if a synonym matches.
private void	releaseBufferedToken()
void	reset()	This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Methods inherited from class org.apache.lucene.analysis.TokenFilter

close, end, unwrap

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

TYPE_SYNONYM

public static final String TYPE_SYNONYM

See Also:

• Constant Field Values

termAtt

private final CharTermAttribute termAtt

posIncrAtt

private final PositionIncrementAttribute posIncrAtt

posLenAtt

private final PositionLengthAttribute posLenAtt

typeAtt

private final TypeAttribute typeAtt

offsetAtt

private final OffsetAttribute offsetAtt

synonyms

private final SynonymMap synonyms

ignoreCase

private final boolean ignoreCase

fst

private final FST<BytesRef> fst

fstReader

private final FST.BytesReader fstReader

scratchArc

private final FST.Arc<BytesRef> scratchArc

bytesReader

private final ByteArrayDataInput bytesReader

scratchBytes

private final BytesRef scratchBytes

scratchChars

private final CharsRefBuilder scratchChars

outputBuffer

private final LinkedList<SynonymGraphFilter.BufferedOutputToken> outputBuffer

nextNodeOut

private int nextNodeOut

lastNodeOut

private int lastNodeOut

maxLookaheadUsed

private int maxLookaheadUsed

captureCount

private int captureCount

liveToken

private boolean liveToken

matchStartOffset

private int matchStartOffset

matchEndOffset

private int matchEndOffset

finished

private boolean finished

lookaheadNextRead

private int lookaheadNextRead

lookaheadNextWrite

private int lookaheadNextWrite

lookahead

private RollingBuffer<SynonymGraphFilter.BufferedInputToken> lookahead

Constructor Details

SynonymGraphFilter

public SynonymGraphFilter(TokenStream input,
 SynonymMap synonyms,
 boolean ignoreCase)

Apply previously built synonyms to incoming tokens.

Parameters:

input - input tokenstream

synonyms - synonym map

ignoreCase - case-folds input for matching with Character.toLowerCase(int). Note, if you set this to true, it's your responsibility to lowercase the input entries when you create the SynonymMap

Method Details

incrementToken

public boolean incrementToken()
                       throws IOException

Description copied from class: TokenStream

Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().

Specified by:

incrementToken in class TokenStream

Returns:

false for end of stream; true otherwise

Throws:

IOException

releaseBufferedToken

private void releaseBufferedToken()
                           throws IOException

Throws:

IOException

parse

private boolean parse()
               throws IOException

Scans the next input token(s) to see if a synonym matches. Returns true if a match was found.

Throws:

IOException

bufferOutputTokens

private void bufferOutputTokens(BytesRef bytes,
 int matchInputLength)

Expands the output graph into the necessary tokens, adding synonyms as side paths parallel to the input tokens, and buffers them in the output token buffer.

capture

private void capture()

Buffers the current input token into lookahead buffer.

reset

public void reset()
           throws IOException

Description copied from class: TokenFilter

This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

NOTE: The default implementation chains the call to the input TokenStream, so be sure to call super.reset() when overriding this method.

Overrides:

reset in class TokenFilter

Throws:

IOException

getCaptureCount

int getCaptureCount()

getMaxLookaheadUsed

int getMaxLookaheadUsed()