Module org.apache.lucene.core

Package org.apache.lucene.codecs.lucene90.compressing

Class Lucene90CompressingTermVectorsWriter

java.lang.Object

org.apache.lucene.codecs.TermVectorsWriter

org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingTermVectorsWriter

All Implemented Interfaces:

Closeable, AutoCloseable, Accountable

public final class Lucene90CompressingTermVectorsWriter
extends TermVectorsWriter

TermVectorsWriter for Lucene90CompressingTermVectorsFormat.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	Lucene90CompressingTermVectorsWriter.CompressingTermVectorsSub
private class	Lucene90CompressingTermVectorsWriter.DocData	a pending doc
private class	Lucene90CompressingTermVectorsWriter.FieldData	a pending field

Field Summary

Fields

Modifier and Type	Field	Description
(package private) static final boolean	BULK_MERGE_ENABLED
(package private) static final String	BULK_MERGE_ENABLED_SYSPROP
private final int	chunkSize
private final CompressionMode	compressionMode
private final Compressor	compressor
private Lucene90CompressingTermVectorsWriter.DocData	curDoc
private Lucene90CompressingTermVectorsWriter.FieldData	curField
(package private) static final int	FLAGS_BITS
private FieldsIndexWriter	indexWriter
private final BytesRef	lastTerm
private int[]	lengthsBuf
private final int	maxDocsPerChunk
(package private) static final int	META_VERSION_START
private IndexOutput	metaStream
private long	numChunks
private long	numDirtyChunks
private long	numDirtyDocs
private int	numDocs
(package private) static final int	OFFSETS
(package private) static final int	PACKED_BLOCK_SIZE
private final ByteBuffersDataOutput	payloadBytes
private int[]	payloadLengthsBuf
(package private) static final int	PAYLOADS
private final Deque<Lucene90CompressingTermVectorsWriter.DocData>	pendingDocs
(package private) static final int	POSITIONS
private int[]	positionsBuf
private final ByteBuffersDataOutput	scratchBuffer
private final String	segment
private int[]	startOffsetsBuf
private final ByteBuffersDataOutput	termSuffixes
(package private) static final String	VECTORS_EXTENSION
(package private) static final String	VECTORS_INDEX_CODEC_NAME
(package private) static final String	VECTORS_INDEX_EXTENSION
(package private) static final String	VECTORS_META_EXTENSION
private IndexOutput	vectorsStream
(package private) static final int	VERSION_CURRENT
(package private) static final int	VERSION_START
private final BlockPackedWriter	writer

Fields inherited from interface org.apache.lucene.util.Accountable

NULL_ACCOUNTABLE

Constructor Summary

Constructors

Constructor	Description
Lucene90CompressingTermVectorsWriter(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, int chunkSize, int maxDocsPerChunk, int blockShift)	Sole constructor.

Method Summary

Modifier and Type	Method	Description
private Lucene90CompressingTermVectorsWriter.DocData	addDocData(int numVectorFields)
void	addPosition(int position, int startOffset, int endOffset, BytesRef payload)	Adds a term position and offsets
void	addProx(int numProx, DataInput positions, DataInput offsets)	Called by IndexWriter when writing new segments.
private boolean	canPerformBulkMerge(MergeState mergeState, MatchingReaders matchingReaders, int readerIndex)
void	close()
private void	copyChunks(MergeState mergeState, Lucene90CompressingTermVectorsWriter.CompressingTermVectorsSub sub, int fromDocID, int toDocID)
void	finish(int numDocs)	Called before TermVectorsWriter.close(), passing in the number of documents that were written.
void	finishDocument()	Called after a doc and all its fields have been added.
void	finishField()	Called after a field and all its terms have been added.
private void	flush(boolean force)
private int[]	flushFieldNums()	Returns a sorted array containing unique field numbers
private void	flushFields(int totalFields, int[] fieldNums)
private void	flushFlags(int totalFields, int[] fieldNums)
private int	flushNumFields(int chunkDocs)
private void	flushNumTerms(int totalFields)
private void	flushOffsets(int[] fieldNums)
private void	flushPayloadLengths()
private void	flushPositions()
private void	flushTermFreqs()
private void	flushTermLengths()
Collection<Accountable>	getChildResources()	Returns nested resources of this class.
int	merge(MergeState mergeState)	Merges in the term vectors from the readers in mergeState.
long	ramBytesUsed()	Return the memory usage of this object in bytes.
void	startDocument(int numVectorFields)	Called before writing the term vectors of the document.
void	startField(FieldInfo info, int numTerms, boolean positions, boolean offsets, boolean payloads)	Called before writing the terms of the field.
void	startTerm(BytesRef term, int freq)	Adds a term and its term frequency freq.
(package private) boolean	tooDirty(Lucene90CompressingTermVectorsReader candidate)	Returns true if we should recompress this reader, even though we could bulk merge compressed data
private boolean	triggerFlush()

Methods inherited from class org.apache.lucene.codecs.TermVectorsWriter

addAllDocVectors, finishTerm

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

VECTORS_EXTENSION

static final String VECTORS_EXTENSION

See Also:

• Constant Field Values

VECTORS_INDEX_EXTENSION

static final String VECTORS_INDEX_EXTENSION

See Also:

• Constant Field Values

VECTORS_META_EXTENSION

static final String VECTORS_META_EXTENSION

See Also:

• Constant Field Values

VECTORS_INDEX_CODEC_NAME

static final String VECTORS_INDEX_CODEC_NAME

See Also:

• Constant Field Values

VERSION_START

static final int VERSION_START

See Also:

• Constant Field Values

VERSION_CURRENT

static final int VERSION_CURRENT

See Also:

• Constant Field Values

META_VERSION_START

static final int META_VERSION_START

See Also:

• Constant Field Values

PACKED_BLOCK_SIZE

static final int PACKED_BLOCK_SIZE

See Also:

• Constant Field Values

POSITIONS

static final int POSITIONS

See Also:

• Constant Field Values

OFFSETS

static final int OFFSETS

See Also:

• Constant Field Values

PAYLOADS

static final int PAYLOADS

See Also:

• Constant Field Values

FLAGS_BITS

static final int FLAGS_BITS

segment

private final String segment

indexWriter

private FieldsIndexWriter indexWriter

metaStream

private IndexOutput metaStream

vectorsStream

private IndexOutput vectorsStream

compressionMode

private final CompressionMode compressionMode

compressor

private final Compressor compressor

chunkSize

private final int chunkSize

numChunks

private long numChunks

numDirtyChunks

private long numDirtyChunks

numDirtyDocs

private long numDirtyDocs

numDocs

private int numDocs

pendingDocs

private final Deque<Lucene90CompressingTermVectorsWriter.DocData> pendingDocs

curDoc

private Lucene90CompressingTermVectorsWriter.DocData curDoc

curField

private Lucene90CompressingTermVectorsWriter.FieldData curField

lastTerm

private final BytesRef lastTerm

positionsBuf

private int[] positionsBuf

startOffsetsBuf

private int[] startOffsetsBuf

lengthsBuf

private int[] lengthsBuf

payloadLengthsBuf

private int[] payloadLengthsBuf

termSuffixes

private final ByteBuffersDataOutput termSuffixes

payloadBytes

private final ByteBuffersDataOutput payloadBytes

writer

private final BlockPackedWriter writer

maxDocsPerChunk

private final int maxDocsPerChunk

scratchBuffer

private final ByteBuffersDataOutput scratchBuffer

BULK_MERGE_ENABLED_SYSPROP

static final String BULK_MERGE_ENABLED_SYSPROP

BULK_MERGE_ENABLED

static final boolean BULK_MERGE_ENABLED

Constructor Details

Lucene90CompressingTermVectorsWriter

Lucene90CompressingTermVectorsWriter(Directory directory,
 SegmentInfo si,
 String segmentSuffix,
 IOContext context,
 String formatName,
 CompressionMode compressionMode,
 int chunkSize,
 int maxDocsPerChunk,
 int blockShift)
                              throws IOException

Sole constructor.

Throws:

IOException

Method Details

addDocData

private Lucene90CompressingTermVectorsWriter.DocData addDocData(int numVectorFields)

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Specified by:

close in class TermVectorsWriter

Throws:

IOException

startDocument

public void startDocument(int numVectorFields)
                   throws IOException

Description copied from class: TermVectorsWriter

Called before writing the term vectors of the document. TermVectorsWriter.startField(FieldInfo, int, boolean, boolean, boolean) will be called numVectorFields times. Note that if term vectors are enabled, this is called even if the document has no vector fields, in this case numVectorFields will be zero.

Specified by:

startDocument in class TermVectorsWriter

Throws:

IOException

finishDocument

public void finishDocument()
                    throws IOException

Description copied from class: TermVectorsWriter

Called after a doc and all its fields have been added.

Overrides:

finishDocument in class TermVectorsWriter

Throws:

IOException

startField

public void startField(FieldInfo info,
 int numTerms,
 boolean positions,
 boolean offsets,
 boolean payloads)
                throws IOException

Description copied from class: TermVectorsWriter

Called before writing the terms of the field. TermVectorsWriter.startTerm(BytesRef, int) will be called numTerms times.

Specified by:

startField in class TermVectorsWriter

Throws:

IOException

finishField

public void finishField()
                 throws IOException

Description copied from class: TermVectorsWriter

Called after a field and all its terms have been added.

Overrides:

finishField in class TermVectorsWriter

Throws:

IOException

startTerm

public void startTerm(BytesRef term,
 int freq)
               throws IOException

Description copied from class: TermVectorsWriter

Adds a term and its term frequency freq. If this field has positions and/or offsets enabled, then TermVectorsWriter.addPosition(int, int, int, BytesRef) will be called freq times respectively.

Specified by:

startTerm in class TermVectorsWriter

Throws:

IOException

addPosition

public void addPosition(int position,
 int startOffset,
 int endOffset,
 BytesRef payload)
                 throws IOException

Description copied from class: TermVectorsWriter

Adds a term position and offsets

Specified by:

addPosition in class TermVectorsWriter

Throws:

IOException

triggerFlush

private boolean triggerFlush()

flush

private void flush(boolean force)
            throws IOException

Throws:

IOException

flushNumFields

private int flushNumFields(int chunkDocs)
                    throws IOException

Throws:

IOException

flushFieldNums

private int[] flushFieldNums()
                      throws IOException

Returns a sorted array containing unique field numbers

Throws:

IOException

flushFields

private void flushFields(int totalFields,
 int[] fieldNums)
                  throws IOException

Throws:

IOException

flushFlags

private void flushFlags(int totalFields,
 int[] fieldNums)
                 throws IOException

Throws:

IOException

flushNumTerms

private void flushNumTerms(int totalFields)
                    throws IOException

Throws:

IOException

flushTermLengths

private void flushTermLengths()
                       throws IOException

Throws:

IOException

flushTermFreqs

private void flushTermFreqs()
                     throws IOException

Throws:

IOException

flushPositions

private void flushPositions()
                     throws IOException

Throws:

IOException

flushOffsets

private void flushOffsets(int[] fieldNums)
                   throws IOException

Throws:

IOException

flushPayloadLengths

private void flushPayloadLengths()
                          throws IOException

Throws:

IOException

finish

public void finish(int numDocs)
            throws IOException

Description copied from class: TermVectorsWriter

Called before TermVectorsWriter.close(), passing in the number of documents that were written. Note that this is intentionally redundant (equivalent to the number of calls to TermVectorsWriter.startDocument(int), but a Codec should check that this is the case to detect the JRE bug described in LUCENE-1282.

Specified by:

finish in class TermVectorsWriter

Throws:

IOException

addProx

public void addProx(int numProx,
 DataInput positions,
 DataInput offsets)
             throws IOException

Description copied from class: TermVectorsWriter

Called by IndexWriter when writing new segments.

This is an expert API that allows the codec to consume positions and offsets directly from the indexer.

The default implementation calls TermVectorsWriter.addPosition(int, int, int, BytesRef), but subclasses can override this if they want to efficiently write all the positions, then all the offsets, for example.

NOTE: This API is extremely expert and subject to change or removal!!!

Overrides:

addProx in class TermVectorsWriter

Throws:

IOException

copyChunks

private void copyChunks(MergeState mergeState,
 Lucene90CompressingTermVectorsWriter.CompressingTermVectorsSub sub,
 int fromDocID,
 int toDocID)
                 throws IOException

Throws:

IOException

merge

public int merge(MergeState mergeState)
          throws IOException

Description copied from class: TermVectorsWriter

Merges in the term vectors from the readers in mergeState. The default implementation skips over deleted documents, and uses TermVectorsWriter.startDocument(int), TermVectorsWriter.startField(FieldInfo, int, boolean, boolean, boolean), TermVectorsWriter.startTerm(BytesRef, int), TermVectorsWriter.addPosition(int, int, int, BytesRef), and TermVectorsWriter.finish(int), returning the number of documents that were written. Implementations can override this method for more sophisticated merging (bulk-byte copying, etc).

Overrides:

merge in class TermVectorsWriter

Throws:

IOException

tooDirty

boolean tooDirty(Lucene90CompressingTermVectorsReader candidate)

Returns true if we should recompress this reader, even though we could bulk merge compressed data

The last chunk written for a segment is typically incomplete, so without recompressing, in some worst-case situations (e.g. frequent reopen with tiny flushes), over time the compression ratio can degrade. This is a safety switch.

canPerformBulkMerge

private boolean canPerformBulkMerge(MergeState mergeState,
 MatchingReaders matchingReaders,
 int readerIndex)

ramBytesUsed

public long ramBytesUsed()

Description copied from interface: Accountable

Return the memory usage of this object in bytes. Negative values are illegal.

getChildResources

public Collection<Accountable> getChildResources()

Description copied from interface: Accountable

Returns nested resources of this class. The result should be a point-in-time snapshot (to avoid race conditions).

See Also:

• Accountables