Module org.apache.lucene.analysis.nori

Package org.apache.lucene.analysis.ko

Class KoreanTokenizerFactory

java.lang.Object

org.apache.lucene.analysis.AbstractAnalysisFactory

org.apache.lucene.analysis.TokenizerFactory

org.apache.lucene.analysis.ko.KoreanTokenizerFactory

All Implemented Interfaces:

ResourceLoaderAware

public class KoreanTokenizerFactory
extends TokenizerFactory
implements ResourceLoaderAware

Factory for KoreanTokenizer.

 <fieldType name="text_ko" class="solr.TextField">
   <analyzer>
     <tokenizer class="solr.KoreanTokenizerFactory"
                decompoundMode="discard"
                userDictionary="user.txt"
                userDictionaryEncoding="UTF-8"
                outputUnknownUnigrams="false"
                discardPunctuation="true"
     />
  </analyzer>
 </fieldType>
 

Supports the following attributes:

• userDictionary: User dictionary path.
• userDictionaryEncoding: User dictionary encoding.
• decompoundMode: Decompound mode. Either 'none', 'discard', 'mixed'. Default is discard. See KoreanTokenizer.DecompoundMode
• outputUnknownUnigrams: If true outputs unigrams for unknown words.
• discardPunctuation: true if punctuation tokens should be dropped from the output.

Since:

7.4.0

Field Summary

Fields

Modifier and Type	Field	Description
private static final String	DECOMPOUND_MODE
private static final String	DISCARD_PUNCTUATION
private final boolean	discardPunctuation
private final KoreanTokenizer.DecompoundMode	mode
static final String	NAME	SPI name
private static final String	OUTPUT_UNKNOWN_UNIGRAMS
private final boolean	outputUnknownUnigrams
private static final String	USER_DICT_ENCODING
private static final String	USER_DICT_PATH
private UserDictionary	userDictionary
private final String	userDictionaryEncoding
private final String	userDictionaryPath

Fields inherited from class org.apache.lucene.analysis.AbstractAnalysisFactory

LUCENE_MATCH_VERSION_PARAM, luceneMatchVersion

Constructor Summary

Constructors

Constructor	Description
KoreanTokenizerFactory()	Default ctor for compatibility with SPI
KoreanTokenizerFactory(Map<String,String> args)	Creates a new KoreanTokenizerFactory

Method Summary

Modifier and Type	Method	Description
KoreanTokenizer	create(AttributeFactory factory)	Creates a TokenStream of the specified input using the given AttributeFactory
void	inform(ResourceLoader loader)	Initializes this component with the provided ResourceLoader (used for loading classes, files, etc).

Methods inherited from class org.apache.lucene.analysis.TokenizerFactory

availableTokenizers, create, findSPIName, forName, lookupClass, reloadTokenizers

Methods inherited from class org.apache.lucene.analysis.AbstractAnalysisFactory

defaultCtorException, get, get, get, get, get, getBoolean, getChar, getClassArg, getFloat, getInt, getLines, getLuceneMatchVersion, getOriginalArgs, getPattern, getSet, getSnowballWordSet, getWordSet, isExplicitLuceneMatchVersion, require, require, require, requireBoolean, requireChar, requireFloat, requireInt, setExplicitLuceneMatchVersion, splitAt, splitFileNames

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

NAME

public static final String NAME

SPI name

See Also:

• Constant Field Values

USER_DICT_PATH

private static final String USER_DICT_PATH

See Also:

• Constant Field Values

USER_DICT_ENCODING

private static final String USER_DICT_ENCODING

See Also:

• Constant Field Values

DECOMPOUND_MODE

private static final String DECOMPOUND_MODE

See Also:

• Constant Field Values

OUTPUT_UNKNOWN_UNIGRAMS

private static final String OUTPUT_UNKNOWN_UNIGRAMS

See Also:

• Constant Field Values

DISCARD_PUNCTUATION

private static final String DISCARD_PUNCTUATION

See Also:

• Constant Field Values

userDictionaryPath

private final String userDictionaryPath

userDictionaryEncoding

private final String userDictionaryEncoding

userDictionary

private UserDictionary userDictionary

mode

private final KoreanTokenizer.DecompoundMode mode

outputUnknownUnigrams

private final boolean outputUnknownUnigrams

discardPunctuation

private final boolean discardPunctuation

Constructor Details

KoreanTokenizerFactory

public KoreanTokenizerFactory(Map<String,String> args)

Creates a new KoreanTokenizerFactory

KoreanTokenizerFactory

public KoreanTokenizerFactory()

Default ctor for compatibility with SPI

Method Details

inform

public void inform(ResourceLoader loader)
            throws IOException

Description copied from interface: ResourceLoaderAware

Initializes this component with the provided ResourceLoader (used for loading classes, files, etc).

Specified by:

inform in interface ResourceLoaderAware

Throws:

IOException

create

public KoreanTokenizer create(AttributeFactory factory)

Description copied from class: TokenizerFactory

Creates a TokenStream of the specified input using the given AttributeFactory

Specified by:

create in class TokenizerFactory