Embedded search engines often persist each immutable index segment as one compact file. That file must support very different access paths: reading stored fields by document number, mapping field names to integer identifiers, traversing a term dictionary, jumping from a term to its posting list, and optionally decoding frequencies, norms, positions, and column-like document values.
The Bluge Ice segment format is a useful example because it explains the layout directly. The file is written in the reverse order of normal query access. Data sections are emitted first, later indexes point back to offsets already known, and a fixed-position footer at the end records the offsets a reader needs to bootstrap navigation.
The broader training pattern is not specific to one Go library. Any search segment format must choose between sequential write simplicity and random-read efficiency. A footer-driven, offset-indexed layout lets the writer make a single pass while allowing the reader to jump directly to the small part of the segment needed for a query.
Writers often cannot emit a top-level index until they know the byte offsets of lower-level structures. Ice solves this by writing payload sections first, then writing indexes that point to those payloads, and finally writing a footer that points to the indexes. Query access runs in the opposite direction: read footer, locate indexes, then jump to payloads.
The footer is fixed enough that the reader can find it without scanning the file. It stores document count, offsets for stored fields, fields index, and document values, plus a chunk factor, version, and CRC. Versioning matters because everything after the fixed footer can evolve as long as the parser checks the version before interpreting offsets.
Search query execution follows a stable chain: field name to field id, field id to term dictionary, term dictionary entry to posting-list offset, posting bitmap to candidate documents, then optional detail streams for frequency, norm, and location data.
Once a segment is immutable, byte offsets become stable identifiers. The format can compress payloads aggressively while keeping tiny offset tables available for direct seeking.
Stored fields are encoded per document. A preparation step builds metadata bytes and data bytes in field-id order; metadata records field id, start offset, and value length, while the data bytes are compressed. The stored-fields index then records one 64-bit offset per document so a reader can jump to document N without decoding earlier documents.
// Pseudocode for the write shape of stored fields.
for doc in segment.documents:
start = writer.offset()
meta = varints([])
data = bytes()
for field in fields_sorted_by_id(doc):
value_start = len(data)
data += encode(field.value)
meta += varint(field.id)
meta += varint(value_start)
meta += varint(len(field.value))
compressed = snappy(data)
writer.varint(len(meta))
writer.varint(len(compressed))
writer.bytes(meta)
writer.bytes(compressed)
stored_index.append(start)
for offset in stored_index:
writer.u64be(offset)Posting details are chunked by document number. For each posting list, the writer records where each detail chunk starts and then writes varint streams for term frequency, scoring norm, or location tuples. Given a document number and chunk factor, the reader can jump to the relevant chunk instead of scanning all details.
Field dictionaries are encoded as finite state transducers whose values point to posting-list offsets. The footer then stores the offsets of the field index and document-value region, so opening the segment only requires reading a small fixed tail before navigating the rest of the file.
An application that embeds a Go search library can ship a single segment file per flush or merge. Reads become memory-map friendly, and writes avoid maintaining many separate sidecar files for dictionaries, postings, and stored fields.
Immutable segment files make merge planning simpler. A merger reads old segments through their public iterators, remaps document numbers after deletes, and writes a new segment with fresh offsets and a fresh footer.
Footer offsets and explicit section boundaries make corruption diagnosis easier. Engineers can verify the checksum, inspect the footer, seek to each section, and determine whether a failure is in the dictionary, postings, stored fields, or document values.
The footer should contain only the values required to find richer metadata. Keep it fixed-size or versioned so the reader can locate it from the end of the file. Put large or evolving structures behind offsets.
type Footer struct {
NumDocs uint64
StoredIndexOffset uint64
FieldsIndexOffset uint64
DocValueOffset uint64
ChunkFactor uint32
Version uint32
CRC32 uint32
}
func OpenSegment(r ReaderAt, size int64) (*Segment, error) {
footer, err := readFooterAtEnd(r, size)
if err != nil { return nil, err }
if !supported(footer.Version) { return nil, ErrUnsupportedVersion }
if !crcMatches(r, size, footer.CRC32) { return nil, ErrCorruptSegment }
return &Segment{reader: r, footer: footer}, nil
}A posting bitmap answers which documents match a term. Frequency, norm, and location streams answer how they should score or highlight. Keeping them separate lets simple existence queries avoid detail decoding.
Field names, field ids, and dictionary offsets are small compared with posting data. Loading that metadata once when the segment opens avoids repeated disk seeks for every query term.