#include <OnDiskInvertedLists.h>

Inheritance diagram for faiss::OnDiskInvertedLists:

Classes
struct	List

struct	OngoingPrefetch

struct	Slot

Public Member Functions
	OnDiskInvertedLists (size_t nlist, size_t code_size, const char *filename)
	are inverted lists mapped read-only

size_t	list_size (size_t list_no) const override
	get the size of a list

const uint8_t *	get_codes (size_t list_no) const override

const idx_t *	get_ids (size_t list_no) const override

size_t	add_entries (size_t list_no, size_t n_entry, const idx_t ids, const uint8_t code) override

void	update_entries (size_t list_no, size_t offset, size_t n_entry, const idx_t ids, const uint8_t code) override

void	resize (size_t list_no, size_t new_size) override

size_t	merge_from (const InvertedLists **ils, int n_il)

void	prefetch_lists (const long *list_nos, int nlist) const override

void	do_mmap ()

void	update_totsize (size_t new_totsize)

void	resize_locked (size_t list_no, size_t new_size)

size_t	allocate_slot (size_t capacity)

void	free_slot (size_t offset, size_t capacity)

Public Member Functions inherited from faiss::InvertedLists
	InvertedLists (size_t nlist, size_t code_size)

virtual void	release_codes (const uint8_t *codes) const
	release codes returned by get_codes (default implementation is nop

virtual void	release_ids (const idx_t *ids) const
	release ids returned by get_ids

virtual idx_t	get_single_id (size_t list_no, size_t offset) const

virtual const uint8_t *	get_single_code (size_t list_no, size_t offset) const

virtual size_t	add_entry (size_t list_no, idx_t theid, const uint8_t *code)
	add one entry to an inverted list

virtual void	update_entry (size_t list_no, size_t offset, idx_t id, const uint8_t *code)

virtual void	reset ()

void	merge_from (InvertedLists *oivf, size_t add_id)
	move all entries from oivf (empty on output)

Public Attributes
std::vector< List >	lists

std::list< Slot >	slots

std::string	filename

size_t	totsize

uint8_t *	ptr

bool	read_only

LockLevels *	locks

OngoingPrefetch *	pf

Public Attributes inherited from faiss::InvertedLists
size_t	nlist
	number of possible key values

size_t	code_size
	code size per vector in bytes

Additional Inherited Members
Public Types inherited from faiss::InvertedLists
typedef Index::idx_t	idx_t

Detailed Description

On-disk storage of inverted lists.

The data is stored in a mmapped chunk of memory (base ptointer ptr, size totsize). Each list is a range of memory that contains (object List) that contains:

uint8_t codes[capacity * code_size]
followed by idx_t ids[capacity]

in each of the arrays, the size <= capacity first elements are used, the rest is not initialized.

Addition and resize are supported by:

roundind up the capacity of the lists to a power of two
maintaining a list of empty slots, sorted by size.
resizing the mmapped block is adjusted as needed.

An OnDiskInvertedLists is compact if the size == capacity for all lists and there are no available slots.

Addition to the invlists is slow. For incremental add it is better to use a default ArrayInvertedLists object and convert it to an OnDisk with merge_from.

When it is known that a set of lists will be accessed, it is useful to call prefetch_lists, that launches a set of threads to read the lists in parallel.

Definition at line 52 of file OnDiskInvertedLists.h.