Support portable bulk bootstrap of a new content generation from an immutable data snapshot

[lyang24]

Is your feature request related to a problem? Please describe.

Users operating large Vespa corpora sometimes need to build an entirely new,
isolated content generation from an authoritative external snapshot, such as a
data lake or warehouse export.

Common reasons include:

• Adding fields that cannot be reconstructed from documents currently stored in
  Vespa.
• Changing document transformations, tokenization, schemas, or field types in a
  way that is unsafe to expose as a mixed-state in-place update.
• Rebuilding or healing the complete corpus.
• Building a new application or content cluster for shadow testing, canary
  rollout, migration, or rollback.

Today, the generally available approach is to feed every document through the
normal document API into an empty target cluster. This is functionally correct,
but at hundreds of millions of documents it can take days and consume substantial
CPU, network, transaction-log, indexing, flush, and fusion capacity.

The normal feed path is optimized for durable online mutations. A full bootstrap
of an isolated and initially non-serving cluster has different requirements and
may be able to avoid some of this per-document overhead.

Existing Vespa mechanisms do not appear to cover this use case completely:

• Native reindexing revisits documents already stored in Vespa and recomputes
  derived fields. It cannot reconstruct fields that exist only in an external
  source, and it does not build an isolated corpus generation.
• Visiting can export and re-feed documents, but visiting does not provide
  snapshot isolation and the restore still uses the regular feed path.
• Node-local Proton state consists of several coordinated components, including
  the document store, document metadata, disk indexes, attributes, configuration
  snapshots, flushed serial numbers, and transaction-log state. Copying these
  directories is not a supported, topology-independent import mechanism.
• Backup or application cloning may reproduce an existing Vespa generation, but
  does not build a transformed generation from an authoritative external
  snapshot.

As a result, users have to build substantial orchestration around a potentially
multi-day feed operation while also handling retries, forward changes, deletion
semantics, validation, and cutover.

Describe the solution you'd like

Provide a supported bulk-bootstrap mechanism for creating a new, initially
non-serving Vespa content generation from an immutable snapshot.

The user-facing capability is more important than prescribing a particular file
format. Possible implementations could include:

1. Importing a portable Vespa-native generation artifact built offline; or
2. An engine-managed bulk-load mode for an empty target that builds native state
   more efficiently than the regular online feed path.

A useful interface would have the following properties:

• Immutable manifest: The import is bound to an exact Vespa version,
  application/schema digest, document types, indexing configuration, source
  identity, and source watermark.
• Complete query-visible state: The result covers the document store,
  inverted indexes, attributes, tensors, document metadata, and delete/tombstone
  semantics required to produce the same query-visible result as ordinary feed.
• Topology independence: The artifact is logical or partitioned in a way that
  Vespa can safely redistribute to a target cluster whose node layout differs
  from the system that produced it.
• Integrity validation: Vespa rejects incompatible, incomplete, or corrupted
  input using checksums, format compatibility, schema checks, and completeness
  metadata.
• Checkpoint and resume: A large import reports durable progress and can
  resume after process, node, or controller failure without rebuilding successful
  partitions.
• Failure isolation: An incomplete or failed generation is never exposed as
  query-ready.
• Forward-write compatibility: The bootstrap establishes a base watermark
  and can safely converge with mutations that occurred after that watermark.
  Older snapshot state must not overwrite newer accepted writes.
• Destination completion evidence: Completion means the target has applied
  and made the imported state searchable, rather than only confirming that an
  external producer finished uploading data.
• Operational controls: Progress, effective rate, resource consumption,
  pause/resume, throttling, and terminal errors are observable through a
  supported API.
• Independent activation: Import completion should not automatically change
  serving traffic. Applications can validate, shadow, canary, and activate the
  new target separately.

The exact method of combining the snapshot with forward mutations could be a
separate API or follow-up feature. At minimum, the bootstrap should expose a
precise base watermark so an external indexing system can perform a safe,
versioned catch-up.

An initial, smaller version of the feature could support only:

• Empty target content clusters.
• A fixed application/schema version.
• Full-document puts and deletes.
• No topology changes during import.
• Explicit finalization before the target becomes ready.

This would still provide substantial value while leaving portable prebuilt
indexes and more advanced incremental restore for later work.

Describe alternatives you've considered

1. Regular document feed into an isolated target

   This is the current safe fallback and supports arbitrary external
   transformations. However, every document takes the online mutation path, and
   large rebuilds may take days and create substantial indexing, transaction-log,
   compaction, and network load.

2. Vespa native reindexing

   Native reindexing is useful when new values can be derived from document
   fields already stored in Vespa. It does not handle externally sourced fields,
   incompatible whole-corpus changes, or isolated generation construction.

3. In-place field backfill

   This may work for additive fields that can remain unused until coverage is
   complete. It is unsafe for destructive changes, cross-field invariants, or
   query behavior that cannot tolerate mixed old and new documents. It can also
   contend with production serving and feed SLOs.

4. Visit/export followed by re-feed

   This is useful for cloning logical documents but does not provide a
   point-in-time snapshot during concurrent writes. It also retains the cost of
   feeding every document through the normal destination write path.

5. Copying content-node data directories or storage volumes

   Proton can load its own local persisted state during node startup, but that
   state is coupled to configuration, serial numbers, bucket ownership, local
   sub-databases, attributes, indexes, document metadata, and transaction-log
   replay. Treating local directories as a portable public artifact would be
   fragile and unsafe.

6. Backup or application/data cloning

   These mechanisms are valuable for recovery or reproducing an existing
   generation. They do not address rebuilding transformed documents from an
   external authoritative snapshot under a new schema or indexing contract.

7. Maintaining application-specific offline index builders

   Users could attempt to invoke internal Vespa indexing components themselves,
   but this would couple them to unsupported storage formats and internal
   invariants that may change between Vespa versions.

Additional context

The desired workflow is conceptually:

1. Freeze an immutable external dataset and record its source watermark.
2. Build or prepare the bulk-bootstrap artifact in parallel.
3. Start or allocate an isolated target Vespa application/content cluster.
4. Import the snapshot into the target using the proposed capability.
5. Apply or reconcile forward changes after the snapshot watermark.
6. Validate document and field parity, query behavior, freshness, capacity, and
   serving latency.
7. Shadow or canary the new target before switching traffic.
8. Retain the previous target for rollback.

This request is not asking Vespa to:

• Read directly from a particular warehouse such as BigQuery.
• Replace application-level source-cut, validation, routing, or rollback logic.
• Expose raw content-node directories as a stable public storage format.
• Automatically activate an imported generation for serving.
• Provide a general disaster-recovery backup system as part of the same feature.

The primary success criterion would be that, for a fixed logical input and
application package, bulk bootstrap produces the same query-visible state as the
ordinary feed path while materially reducing end-to-end rebuild time and resource
cost.

Additional acceptance properties would include:

• A corrupted or schema-incompatible artifact fails before activation.
• Restarting an interrupted import resumes completed work.
• A newer forward mutation is never overwritten by older snapshot state.
• Target topology differences are either supported or explicitly rejected during
  preflight.
• Import completion is based on searchable destination state.
• The target remains unavailable for normal queries until explicitly finalized.

Relevant source observations from the current Vespa implementation:

• DocumentDB initializes coordinated local state and then replays the
  transaction log:
  

  vespa/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp

  Lines 202 to 219 in c2324d1

  	LOG(debug, "DocumentDB(%s): Creating database in directory '%s'", _docTypeName.toString().c_str(),
  	_baseDir.c_str());
  	_feedHandler->init(_config_store->getOldestSerialNum());
  	_feedHandler->setBucketDBHandler(&_subDBs.getBucketDBHandler());
  	saveInitialConfig(configSnapshot);
  	resumeSaveConfig();
  	SerialNum configSerial = _config_store->getPrevValidSerial(_feedHandler->getPrunedSerialNum() + 1);
  	assert(configSerial > 0);
  	DocumentDBConfig::SP loaded_config;
  	_config_store->loadConfig(*configSnapshot, configSerial, loaded_config);
  	// Grab relevant parts from pending config
  	loaded_config = DocumentDBConfigScout::scout(loaded_config, *_pendingConfigSnapshot.get());
  	// Ignore configs that are not relevant during replay of transaction log
  	loaded_config = DocumentDBConfig::makeReplayConfig(loaded_config);
  	_initConfigSnapshot = loaded_config;
  	_initConfigSerialNum = configSerial;

  
  

  vespa/searchcore/src/vespa/searchcore/proton/server/documentdb.cpp

  Lines 640 to 651 in c2324d1

  	void DocumentDB::startTransactionLogReplay() {
  	// This configSnapshot is only used to reuse DocumentTypeRepo
  	// and TuneFile when loading configs during replay.
  	DocumentDBConfig::SP configSnapshot = getActiveConfig();
  	IDocumentSubDB* readySubDB = _subDBs.getReadySubDB();
  	SerialNum oldestFlushedSerial = getOldestFlushedSerial();
  	SerialNum newestFlushedSerial = getNewestFlushedSerial();
  	(void)_state->enterReplayTransactionLogState();
  	_feedHandler->replayTransactionLog(readySubDB->getIndexManager()->getFlushedSerialNum(),
  	readySubDB->getSummaryManager()->getBackingStore().lastSyncToken(),
  	oldestFlushedSerial, newestFlushedSerial, *_config_store,
  	_owner.shared_replay_throttler());

• IndexManager::loadDiskIndex loads an internal node-local disk-index
  directory, but this is only one part of complete searchable state:
  

  vespa/searchcore/src/vespa/searchcore/proton/index/indexmanager.cpp

  Lines 53 to 58 in c2324d1

  	IDiskIndex::SP IndexManager::MaintainerOperations::loadDiskIndex(const std::string& indexDir) {
  	return std::make_shared<DiskIndexWrapper>(indexDir, _tuneFileSearch, _posting_list_cache);
  	}
  	IDiskIndex::SP IndexManager::MaintainerOperations::reloadDiskIndex(const IDiskIndex& oldIndex) {
  	return std::make_shared<DiskIndexWrapper>(dynamic_cast<const DiskIndexWrapper&>(oldIndex), _tuneFileSearch);

• Persistence iteration explicitly does not guarantee a snapshot view:
  

  vespa/persistence/src/vespa/persistence/spi/persistenceprovider.h

  Lines 290 to 297 in c2324d1

  	* The caller shall not make any assumptions on whether or not documents
  	* that arrive to--or are removed from--the bucket in the time between
  	* separate invocations of iterate for the same iterator id will show up
  	* in the results, assuming that these documents do not violate the
  	* selection criteria. This means that there is no requirement for
  	* maintaining a "snapshot" view of the bucket's state as it existed upon
  	* the initial createIterator call. Neither shall the caller make any
  	* assumptions on the ordering of the returned documents.

We would appreciate guidance on whether an existing Vespa Cloud or self-managed
capability is intended to cover this workflow, and whether the maintainers would
prefer to approach it first as an optimized empty-cluster bulk-load mode or as a
portable native generation artifact.

[joanna-lgmt]

I was wondering if the initial implementation could focus on the engine-managed bulk-load mode for an empty cluster, since this seems like a simpler starting point before introducing a portable native generation artifact. Would this approach also make it easier to handle the base watermark and forward mutations safely?

[lyang24]

I was wondering if the initial implementation could focus on the engine-managed bulk-load mode for an empty cluster, since this seems like a simpler starting point before introducing a portable native generation artifact. Would this approach also make it easier to handle the base watermark and forward mutations safely?

Yes, an engine-managed bulk-load mode for an empty target is the preferred MVP.

It avoids making Proton’s node-local persisted state a portable public format,
while still allowing Vespa to optimize index construction internally.

It would simplify watermark handling, but not eliminate the need for an explicit
versioning protocol. The snapshot should be bound to a source watermark W, and
mutations after W must be replayed with ordering/idempotency guarantees so that
older snapshot operations cannot overwrite newer accepted writes.

[bratseth]

Thanks for the detailed writeup. There are many wishes here and there are inherent tradeoffs where not all of them are physically possible to achieve simultaneously, so I'll try to narrow this down a bit.

First, the most important constraints on solution space:

• Indexes are node-spanning physical structures - that's what make them efficient, and so are inherently tied to a given system topology.
• Cluster-wide consistent snapshots are impossible in combination with continuous writes (Vespa only stores the latest version).
• Node-local consistent snapshots of all data structures can only be taken when a node is offline.

Given this, what solutions make sense?

1. Dumping document data only using visiting, and stand up the cluster by streaming this data into it.

This can be extended to implement versioning schemes by adding a version field and conditional writes. (There are many variants of this, global batch, incremental generations, per-source/feed generations etc., so we believe it is better to keep the platform agnostic to which one is used.)

Pros:

• Fully portable between Vespa versions, application revisions that preserve the fields in question, and system topologies.
• Any subset of data can be captured.
• Restore can be done on a live and serving system with existing data.
• The data dump is legible and can be filtered, modified, split and joined etc.
• Minimal size when compressed.

Cons:

• Restore involves indexing, which is slow when there are indexed tensor fields.

2. Saving and restoring the full indexed node state

This involves an orchestrated process where each node is taken offline and flushed in sequence.

Pros:

• Faster restore, especially with indexed tensors.

Cons:

• Taking a cluster-wide state dump may take a long time.
• The stored state is opaque binary data.
• The stored state can only be restored on a system with the same topology, platform versions within a 3 month age window, and with the same application package.
• The stored state is larger than the raw data (impacts storage and network).

For these reasons, this solution has somewhat limited operational relevance. We know of these use cases:

• Fast node replacements (important for fast OS patching, which is becoming critical).
• Disaster backups with maximally fast time to recovery.
• Fast group-level scaleup.
• Hibernate dev applications.
• Fast creation of a node-downscaled dev clone of a product system.

The two first are supported on Vespa Cloud, the rest are in progress.
We're also working on enabling these with the Vespa Kubernetes operator (it is running the Vespa Cloud code, but some additional work is needed for it to be functional in any environment).

Lastly a more general comment regarding where this is relevant: You almost certainly want to take advantage of the realtime writes supported in Vespa, both the ensure UGC is always current, and to continuously update metadata signals tracking behavior etc., and making snapshot/restore a part of regular operational procedures in production is fundamentally incompatible with this. It is also at least twice as expensive since clusters can't both serve and restore. Vespa is written from the ground up to support continuous evolution of production systems including data, schemas, and topology without ever disrupting queries and writes, and it is a good idea to take advantage of it.