State and storage
Understand the state a tapstate data path must retain for current reads, recovery, and audit
A pipeline needs more than its latest output. Tapstate separates three kinds of state: working state used while records are processed, recovery state needed after an interruption, and materialized state read by downstream consumers.
Choose concrete storage engines, retention, replication, and high-availability behavior for the deployment you operate.
Operational state and analytical history
Operational consumers usually need the latest valid state of an account, order, inventory item, or entitlement. Analytical systems usually retain a longer history optimized for scans, aggregation, and model training.
| Concern | Operational state | Analytical history |
|---|---|---|
| Primary question | What is true now? | What happened over time? |
| Common access | Key lookup, current view, event reaction | Scan, aggregate, compare periods |
| Update pattern | Continuous changes and deletes | Batch or append-oriented ingestion |
| Failure sensitivity | Stale state can affect an action | Delayed data usually affects analysis timing |
The two are complementary. A production design can maintain current state for applications while retaining approved history in a warehouse or lake.
Working state
The data plane needs short-lived state for active processing, buffering, routing, and lookups. Working state improves throughput, but it must not be the only copy of information required after restart.
Recovery state
Durable recovery state includes:
- connector positions, offsets, or checkpoints;
- pipeline identity and canonical configuration;
- retry and failure context;
- schema and mapping state;
- ownership information needed to prevent conflicting execution.
The metadata store retains pipeline definitions, connector configuration, recovery positions, and the durable shared change log. Compatible pipelines can share one capture while retaining independent consumer positions and initial-load boundaries. A restart can resume from retained positions, but records may be replayed. Snapshot-only reads have no CDC continuation position.
For shared capture, a source acknowledgement means the change is durable in tapstate's log; target acknowledgement is recorded separately for each consumer. Direct capture acknowledges progress after target confirmation. See Initial loads and recovery.
Keep source logs long enough for recovery. Delivery remains at-least-once for changes read by the connector; the snapshot-to-stream positioning window can still miss changes. See v0.7 limits.
For self-hosted deployments, the default operator-state database is tapstate_nest, configured with
tapstate.store.mongo.operator-state-database. A Nest can override it with
state.database on the same MongoDB connection. Changing either setting does
not copy existing state. Follow Move Nest state
when moving a running workload. Cloud derives its internal operator-state and
View database names from the platform metadata database; these are separate
from the Atlas connection you configure as an output destination.
Back up configuration and recovery together
In v0.7, stored Source connector configurations are encrypted with a keyring in the metadata database. Preserve the encrypted records and their original keyring together. A display-only Source read omits connector configuration and is not a restorable configuration export.
Stop all writers before taking a consistent backup. Include metadata, operator-state databases, and managed View data, plus the deployment configuration and required working volumes. Account for external targets separately. Restoring older metadata does not restore changes that have expired from source logs or undo writes already delivered to a target. See Upgrade to v0.7.
Materialized state
Materialized state is the current consumer-facing result. An inline view
writes to the deployment's managed MongoDB store. A serve.sync block writes
to its configured target connection.
In Cloud, State Store configures a user-supplied MongoDB Atlas connection
named atlas-store. The server tests the connection before saving it, and
pipelines use its selected output collection. This connection is separate from
the platform's internal metadata and operator-state databases. The self-hosted
Quickstart instead creates a managed views connection automatically.
The Quickstart runs its managed store without database authentication and uses the same MongoDB deployment for control-plane data. That disposable topology is not a tenant or user security boundary. Use sample data only; a production deployment needs independently designed authentication, network exposure, and data isolation.
Its correctness depends on:
- stable keys for updates and deletes;
- defined append or update behavior;
- schema-change policy;
- ordering and duplicate handling;
- reconciliation with the system of record;
- retention and deletion requirements.
Audit history
An audit or replay store answers different questions from a current view. Before retaining change history, define the legal basis, sensitive-field handling, retention period, access policy, and deletion process.
Choose retention, replay, and recovery settings for the workload and deployment you operate. A current-state destination does not automatically provide long-term history.