Module org.apache.lucene.classification

Package org.apache.lucene.classification

Class BM25NBClassifier

java.lang.Object

org.apache.lucene.classification.BM25NBClassifier

All Implemented Interfaces:

Classifier<BytesRef>

public class BM25NBClassifier
extends Object
implements Classifier<BytesRef>

A classifier approximating naive bayes classifier by using pure queries on BM25.

Field Summary

Fields

Modifier and Type	Field	Description
private final Analyzer	analyzer	Analyzer to be used for tokenizing unseen input text
private final String	classFieldName	name of the field to be used as a class / category output
private final IndexReader	indexReader	IndexReader used to access the Classifier's index
private final IndexSearcher	indexSearcher	IndexSearcher to run searches on the index for retrieving frequencies
private final Query	query	Query used to eventually filter the document set to be used to classify
private final String[]	textFieldNames	names of the fields to be used as input text

Constructor Summary

Constructors

Constructor	Description
BM25NBClassifier(IndexReader indexReader, Analyzer analyzer, Query query, String classFieldName, String... textFieldNames)	Creates a new NaiveBayes classifier.

Method Summary

Modifier and Type	Method	Description
ClassificationResult<BytesRef>	assignClass(String inputDocument)	Assign a class (with score) to the given text String
private List<ClassificationResult<BytesRef>>	assignClassNormalizedList(String inputDocument)	Calculate probabilities for all classes for a given input text
private double	calculateLogLikelihood(String[] tokens, Term term)
private double	calculateLogPrior(Term term)
List<ClassificationResult<BytesRef>>	getClasses(String text)	Get all the classes (sorted by score, descending) assigned to the given text String.
List<ClassificationResult<BytesRef>>	getClasses(String text, int max)	Get the first max classes (sorted by score, descending) assigned to the given text String.
private double	getTermProbForClass(Term classTerm, String... words)
private ArrayList<ClassificationResult<BytesRef>>	normClassificationResults(List<ClassificationResult<BytesRef>> assignedClasses)	Normalize the classification results based on the max score available
private String[]	tokenize(String text)	tokenize a String on this classifier's text fields and analyzer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

indexReader

private final IndexReader indexReader

IndexReader used to access the Classifier's index

textFieldNames

private final String[] textFieldNames

names of the fields to be used as input text

classFieldName

private final String classFieldName

name of the field to be used as a class / category output

analyzer

private final Analyzer analyzer

Analyzer to be used for tokenizing unseen input text

indexSearcher

private final IndexSearcher indexSearcher

IndexSearcher to run searches on the index for retrieving frequencies

query

private final Query query

Query used to eventually filter the document set to be used to classify

Constructor Details

BM25NBClassifier

public BM25NBClassifier(IndexReader indexReader,
 Analyzer analyzer,
 Query query,
 String classFieldName,
 String... textFieldNames)

Creates a new NaiveBayes classifier.

Parameters:

indexReader - the reader on the index to be used for classification

analyzer - an Analyzer used to analyze unseen text

query - a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used

classFieldName - the name of the field used as the output for the classifier NOTE: must not be heavely analyzed as the returned class will be a token indexed for this field

textFieldNames - the name of the fields used as the inputs for the classifier, NO boosting supported per field

Method Details

assignClass

public ClassificationResult<BytesRef> assignClass(String inputDocument)
                                           throws IOException

Description copied from interface: Classifier

Assign a class (with score) to the given text String

Specified by:

assignClass in interface Classifier<BytesRef>

Parameters:

inputDocument - a String containing text to be classified

Returns:

a ClassificationResult holding assigned class of type T and score

Throws:

IOException - If there is a low-level I/O error.

getClasses

public List<ClassificationResult<BytesRef>> getClasses(String text)
                                                throws IOException

Description copied from interface: Classifier

Get all the classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

Returns:

the whole list of ClassificationResult, the classes and scores. Returns null if the classifier can't make lists.

Throws:

IOException - If there is a low-level I/O error.

getClasses

public List<ClassificationResult<BytesRef>> getClasses(String text,
 int max)
                                                throws IOException

Description copied from interface: Classifier

Get the first max classes (sorted by score, descending) assigned to the given text String.

Specified by:

getClasses in interface Classifier<BytesRef>

Parameters:

text - a String containing text to be classified

max - the number of return list elements

Returns:

the whole list of ClassificationResult, the classes and scores. Cut for "max" number of elements. Returns null if the classifier can't make lists.

Throws:

IOException - If there is a low-level I/O error.

assignClassNormalizedList

private List<ClassificationResult<BytesRef>> assignClassNormalizedList(String inputDocument)
                                                                throws IOException

Calculate probabilities for all classes for a given input text

Parameters:

inputDocument - the input text as a String

Returns:

a List of ClassificationResult, one for each existing class

Throws:

IOException - if assigning probabilities fails

normClassificationResults

private ArrayList<ClassificationResult<BytesRef>> normClassificationResults(List<ClassificationResult<BytesRef>> assignedClasses)

Normalize the classification results based on the max score available

Parameters:

assignedClasses - the list of assigned classes

Returns:

the normalized results

tokenize

private String[] tokenize(String text)
                   throws IOException

tokenize a String on this classifier's text fields and analyzer

Parameters:

text - the String representing an input text (to be classified)

Returns:

a String array of the resulting tokens

Throws:

IOException - if tokenization fails

calculateLogLikelihood

private double calculateLogLikelihood(String[] tokens,
 Term term)
                               throws IOException

Throws:

IOException

getTermProbForClass

private double getTermProbForClass(Term classTerm,
 String... words)
                            throws IOException

Throws:

IOException

calculateLogPrior

private double calculateLogPrior(Term term)
                          throws IOException

Throws:

IOException