Module org.apache.lucene.analysis.common

Package org.apache.lucene.analysis.custom

Class CustomAnalyzer

java.lang.Object

org.apache.lucene.analysis.Analyzer

org.apache.lucene.analysis.custom.CustomAnalyzer

All Implemented Interfaces:

Closeable, AutoCloseable

public final class CustomAnalyzer
extends Analyzer

A general-purpose Analyzer that can be created with a builder-style API. Under the hood it uses the factory classes TokenizerFactory, TokenFilterFactory, and CharFilterFactory.

You can create an instance of this Analyzer using the builder by passing the SPI names (as defined by ServiceLoader interface) to it:

 Analyzer ana = CustomAnalyzer.builder(Paths.get("/path/to/config/dir"))
   .withTokenizer(StandardTokenizerFactory.NAME)
   .addTokenFilter(LowerCaseFilterFactory.NAME)
   .addTokenFilter(StopFilterFactory.NAME, "ignoreCase", "false", "words", "stopwords.txt", "format", "wordset")
   .build();
 

The parameters passed to components are also used by Apache Solr and are documented on their corresponding factory classes. Refer to documentation of subclasses of TokenizerFactory, TokenFilterFactory, and CharFilterFactory.

This is the same as the above:

 Analyzer ana = CustomAnalyzer.builder(Paths.get("/path/to/config/dir"))
   .withTokenizer("standard")
   .addTokenFilter("lowercase")
   .addTokenFilter("stop", "ignoreCase", "false", "words", "stopwords.txt", "format", "wordset")
   .build();
 

The list of names to be used for components can be looked up through: TokenizerFactory.availableTokenizers(), TokenFilterFactory.availableTokenFilters(), and CharFilterFactory.availableCharFilters().

You can create conditional branches in the analyzer by using CustomAnalyzer.Builder.when(String, String...) and CustomAnalyzer.Builder.whenTerm(Predicate):

 Analyzer ana = CustomAnalyzer.builder()
    .withTokenizer("standard")
    .addTokenFilter("lowercase")
    .whenTerm(t -> t.length() > 10)
      .addTokenFilter("reversestring")
    .endwhen()
    .build();
 

Since:

5.0.0

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	CustomAnalyzer.Builder	Builder for CustomAnalyzer.
static class	CustomAnalyzer.ConditionBuilder	Factory class for a ConditionalTokenFilter

Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer

Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents

Field Summary

Fields

Modifier and Type	Field	Description
private final CharFilterFactory[]	charFilters
private final Integer	offsetGap
private final Integer	posIncGap
private final TokenFilterFactory[]	tokenFilters
private final TokenizerFactory	tokenizer

Fields inherited from class org.apache.lucene.analysis.Analyzer

GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY

Constructor Summary

Constructors

Constructor	Description
CustomAnalyzer(CharFilterFactory[] charFilters, TokenizerFactory tokenizer, TokenFilterFactory[] tokenFilters, Integer posIncGap, Integer offsetGap)

Method Summary

Modifier and Type	Method	Description
static CustomAnalyzer.Builder	builder()	Returns a builder for custom analyzers that loads all resources from Lucene's classloader.
static CustomAnalyzer.Builder	builder(Path configDir)	Returns a builder for custom analyzers that loads all resources from the given file system base directory.
static CustomAnalyzer.Builder	builder(ResourceLoader loader)	Returns a builder for custom analyzers that loads all resources using the given ResourceLoader.
protected Analyzer.TokenStreamComponents	createComponents(String fieldName)	Creates a new Analyzer.TokenStreamComponents instance for this analyzer.
List<CharFilterFactory>	getCharFilterFactories()	Returns the list of char filters that are used in this analyzer.
int	getOffsetGap(String fieldName)	Just like Analyzer.getPositionIncrementGap(java.lang.String), except for Token offsets instead.
int	getPositionIncrementGap(String fieldName)	Invoked before indexing a IndexableField instance if terms have already been added to that field.
List<TokenFilterFactory>	getTokenFilterFactories()	Returns the list of token filters that are used in this analyzer.
TokenizerFactory	getTokenizerFactory()	Returns the tokenizer that is used in this analyzer.
protected Reader	initReader(String fieldName, Reader reader)	Override this if you want to add a CharFilter chain.
protected Reader	initReaderForNormalization(String fieldName, Reader reader)	Wrap the given Reader with CharFilters that make sense for normalization.
protected TokenStream	normalize(String fieldName, TokenStream in)	Wrap the given TokenStream in order to apply normalization filters.
String	toString()

Methods inherited from class org.apache.lucene.analysis.Analyzer

attributeFactory, close, getReuseStrategy, normalize, tokenStream, tokenStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

charFilters

private final CharFilterFactory[] charFilters

tokenizer

private final TokenizerFactory tokenizer

tokenFilters

private final TokenFilterFactory[] tokenFilters

posIncGap

private final Integer posIncGap

offsetGap

private final Integer offsetGap

Constructor Details

CustomAnalyzer

CustomAnalyzer(CharFilterFactory[] charFilters,
 TokenizerFactory tokenizer,
 TokenFilterFactory[] tokenFilters,
 Integer posIncGap,
 Integer offsetGap)

Method Details

builder

public static CustomAnalyzer.Builder builder()

Returns a builder for custom analyzers that loads all resources from Lucene's classloader. All path names given must be absolute with package prefixes.

builder

public static CustomAnalyzer.Builder builder(Path configDir)

Returns a builder for custom analyzers that loads all resources from the given file system base directory. Place, e.g., stop word files there. Files that are not in the given directory are loaded from Lucene's classloader.

builder

public static CustomAnalyzer.Builder builder(ResourceLoader loader)

Returns a builder for custom analyzers that loads all resources using the given ResourceLoader.

initReader

protected Reader initReader(String fieldName,
 Reader reader)

Description copied from class: Analyzer

Override this if you want to add a CharFilter chain.

The default implementation returns reader unchanged.

Overrides:

initReader in class Analyzer

Parameters:

fieldName - IndexableField name being indexed

reader - original Reader

Returns:

reader, optionally decorated with CharFilter(s)

initReaderForNormalization

protected Reader initReaderForNormalization(String fieldName,
 Reader reader)

Description copied from class: Analyzer

Wrap the given Reader with CharFilters that make sense for normalization. This is typically a subset of the CharFilters that are applied in Analyzer.initReader(String, Reader). This is used by Analyzer.normalize(String, String).

Overrides:

initReaderForNormalization in class Analyzer

createComponents

protected Analyzer.TokenStreamComponents createComponents(String fieldName)

Description copied from class: Analyzer

Creates a new Analyzer.TokenStreamComponents instance for this analyzer.

Specified by:

createComponents in class Analyzer

Parameters:

fieldName - the name of the fields content passed to the Analyzer.TokenStreamComponents sink as a reader

Returns:

the Analyzer.TokenStreamComponents for this analyzer.

normalize

protected TokenStream normalize(String fieldName,
 TokenStream in)

Description copied from class: Analyzer

Wrap the given TokenStream in order to apply normalization filters. The default implementation returns the TokenStream as-is. This is used by Analyzer.normalize(String, String).

Overrides:

normalize in class Analyzer

getPositionIncrementGap

public int getPositionIncrementGap(String fieldName)

Description copied from class: Analyzer

Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.

Overrides:

getPositionIncrementGap in class Analyzer

Parameters:

fieldName - IndexableField name being indexed.

Returns:

position increment gap, added to the next token emitted from Analyzer.tokenStream(String,Reader). This value must be >= 0.

getOffsetGap

public int getOffsetGap(String fieldName)

Description copied from class: Analyzer

Just like Analyzer.getPositionIncrementGap(java.lang.String), except for Token offsets instead. By default this returns 1. This method is only called if the field produced at least one token for indexing.

Overrides:

getOffsetGap in class Analyzer

Parameters:

fieldName - the field just indexed

Returns:

offset gap, added to the next token emitted from Analyzer.tokenStream(String,Reader). This value must be >= 0.

getCharFilterFactories

public List<CharFilterFactory> getCharFilterFactories()

Returns the list of char filters that are used in this analyzer.

getTokenizerFactory

public TokenizerFactory getTokenizerFactory()

Returns the tokenizer that is used in this analyzer.

getTokenFilterFactories

public List<TokenFilterFactory> getTokenFilterFactories()

Returns the list of token filters that are used in this analyzer.

toString

public String toString()

Overrides:

toString in class Object