Module org.apache.lucene.classification
Package org.apache.lucene.classification.document

Class SimpleNaiveBayesDocumentClassifier

java.lang.Object
org.apache.lucene.classification.SimpleNaiveBayesClassifier
org.apache.lucene.classification.document.SimpleNaiveBayesDocumentClassifier
All Implemented Interfaces:
Classifier<BytesRef>, DocumentClassifier<BytesRef>
public class SimpleNaiveBayesDocumentClassifier extends SimpleNaiveBayesClassifier implements DocumentClassifier<BytesRef>
A simplistic Lucene based NaiveBayes classifier, see http://en.wikipedia.org/wiki/Naive_Bayes_classifier

  • Field Details

  • Constructor Details

    • SimpleNaiveBayesDocumentClassifier

      public SimpleNaiveBayesDocumentClassifier(IndexReader indexReader, Query query, String classFieldName, Map<String,Analyzer> field2analyzer, String... textFieldNames)
      Creates a new NaiveBayes classifier.
      Parameters:
      indexReader - the reader on the index to be used for classification
      query - a Query to eventually filter the docs used for training the classifier, or null if all the indexed docs should be used
      classFieldName - the name of the field used as the output for the classifier NOTE: must not be heavely analyzed as the returned class will be a token indexed for this field
      textFieldNames - the name of the fields used as the inputs for the classifier, they can contain boosting indication e.g. title^10

  • Method Details

    • assignClass

      public ClassificationResult<BytesRef> assignClass(Document document) throws IOException
      Description copied from interface: DocumentClassifier
      Assign a class (with score) to the given Document
      Specified by:
      assignClass in interface DocumentClassifier<BytesRef>
      Parameters:
      document - a Document to be classified. Fields are considered features for the classification.
      Returns:
      a ClassificationResult holding assigned class of type T and score
      Throws:
      IOException - If there is a low-level I/O error.

    • getClasses

      public List<ClassificationResult<BytesRef>> getClasses(Document document) throws IOException
      Description copied from interface: DocumentClassifier
      Get all the classes (sorted by score, descending) assigned to the given Document.
      Specified by:
      getClasses in interface DocumentClassifier<BytesRef>
      Parameters:
      document - a Document to be classified. Fields are considered features for the classification.
      Returns:
      the whole list of ClassificationResult, the classes and scores. Returns null if the classifier can't make lists.
      Throws:
      IOException - If there is a low-level I/O error.

    • getClasses

      public List<ClassificationResult<BytesRef>> getClasses(Document document, int max) throws IOException
      Description copied from interface: DocumentClassifier
      Get the first max classes (sorted by score, descending) assigned to the given text String.
      Specified by:
      getClasses in interface DocumentClassifier<BytesRef>
      Parameters:
      document - a Document to be classified. Fields are considered features for the classification.
      max - the number of return list elements
      Returns:
      the whole list of ClassificationResult, the classes and scores. Cut for "max" number of elements. Returns null if the classifier can't make lists.
      Throws:
      IOException - If there is a low-level I/O error.

    • assignNormClasses

      private List<ClassificationResult<BytesRef>> assignNormClasses(Document inputDocument) throws IOException
      Throws:
      IOException

    • analyzeSeedDocument

      private void analyzeSeedDocument(Document inputDocument, Map<String,List<String[]>> fieldName2tokensArray, Map<String,Float> fieldName2boost) throws IOException
      This methods performs the analysis for the seed document and extract the boosts if present. This is done only one time for the Seed Document.
      Parameters:
      inputDocument - the seed unseen document
      fieldName2tokensArray - a map that associated to a field name the list of token arrays for all its values
      fieldName2boost - a map that associates the boost to the field
      Throws:
      IOException - If there is a low-level I/O error

    • getTokenArray

      protected String[] getTokenArray(TokenStream tokenizedText) throws IOException
      Returns a token array from the TokenStream in input
      Parameters:
      tokenizedText - the tokenized content of a field
      Returns:
      a String array of the resulting tokens
      Throws:
      IOException - If tokenization fails because there is a low-level I/O error

    • calculateLogLikelihood

      private double calculateLogLikelihood(String[] tokenizedText, String fieldName, Term term, int docsWithClass) throws IOException
      Parameters:
      tokenizedText - the tokenized content of a field
      fieldName - the input field name
      term - the Term referring to the class to calculate the score of
      docsWithClass - the total number of docs that have a class
      Returns:
      a normalized score for the class
      Throws:
      IOException - If there is a low-level I/O error

    • getTextTermFreqForClass

      private double getTextTermFreqForClass(Term term, String fieldName) throws IOException
      Returns the average number of unique terms times the number of docs belonging to the input class
      Parameters:
      term - the class term
      Returns:
      the average number of unique terms
      Throws:
      IOException - If there is a low-level I/O error

    • getWordFreqForClass

      private int getWordFreqForClass(String word, String fieldName, Term term) throws IOException
      Returns the number of documents of the input class ( from the whole index or from a subset) that contains the word ( in a specific field or in all the fields if no one selected)
      Parameters:
      word - the token produced by the analyzer
      fieldName - the field the word is coming from
      term - the class term
      Returns:
      number of documents of the input class
      Throws:
      IOException - If there is a low-level I/O error

    • calculateLogPrior

      private double calculateLogPrior(Term term, int docsWithClassSize) throws IOException
      Throws:
      IOException

    • docCount

      private int docCount(Term term) throws IOException
      Throws:
      IOException