Capture modes

A capture mode defines where a source starts and whether it keeps reading after the initial data has been copied.

Choose the result you need

You need	Connector guide label	Data included	Keeps reading?	Prepare
One copy of the data that exists now	Full load (`Snapshot`)	Existing rows, documents, or files selected for the run	No	Read permissions and a stable source view
Existing data plus later changes	Full load + CDC	The initial copy, followed by supported inserts, updates, and deletes	Yes	Full-load access plus database log, WAL, Oplog, or changelog access
Messages from a broker or queue	Stream	Messages from the configured starting position	Yes	Topic or queue permissions, consumer identity, and retention
Records exposed by a SaaS or business API	API	Objects returned by the service API	Connector-specific	Application credentials, scopes, rate limits, and pagination

Use only catalog read modes shown in the connector directory, then verify the matching runtime artifact and current product path. Database log support or a preparation guide alone does not mean a current tapstate runtime supports CDC.

Which mode should I choose?

Choose Full load for a one-time migration, test dataset, or periodic replacement where later changes are not required.
Choose Full load + CDC when the destination must begin complete and then remain current.
Choose Stream for a message broker or another source that already produces an ordered stream of records.
Choose API for SaaS and business systems where the connector reads through a vendor API.

Full load (Snapshot)

Full load reads the selected data once and completes. In tapstate resources, this is represented by mode: snapshot.

Before running it, decide:

which tables, collections, or files to include;
whether the source can provide a consistent view while the copy runs;
whether a rerun replaces, appends, or reconciles destination data;
how you will compare record counts and representative values.

Full load + CDC

Full load + CDC first copies the selected existing data, catches up with changes made during that copy, and then continues reading new changes. In tapstate resources, this is represented by mode: cdc.

Initial copy  →  Catch up  →  Continue with new changes

Prepare and verify:

The account can complete the initial read.
The source emits the change records required by the connector.
The change log is retained long enough for planned outages and restarts.
Each source table written with upsert has a primary key in its discovered schema. A unique index is not enough. Use append only for insert-only delivery.
A staging run covers insert, update, delete, stop, and resume behavior.

Initial loads and recovery

For a CDC source, settings.read_mode controls a pipeline run: snapshot_only, cdc_only, or snapshot_and_cdc. Use the modes and prerequisites supported by that connector; a source's mode and a pipeline's read mode describe different choices.

Pipeline read mode	First run	Recovery with retained state
`snapshot_and_cdc` (default)	Copy existing rows, then follow changes.	Use retained initial-load completion and resume the consumer's change tail. Unconfirmed work can be replayed.
`cdc_only`	Skip the initial copy and follow changes from the selected starting position.	Resume from retained progress; do not use another pipeline's later position.
`snapshot_only`	Read the selected data once without following later changes.	No CDC continuation position is created. A replacement run can reread the initial load.

Compatible pipelines share a CDC capture by default while keeping independent consumer positions and initial-load completion. Only selected tables belong to a pipeline's capture contract. Changing read_mode does not clear existing state or create a fresh initial load. Review the lifecycle choices below before resetting a run.

Shared and direct capture

With shared capture, changes are durably written to tapstate's change log before pipelines consume them. The connector can then acknowledge those changes to the source. Each pipeline separately records what its target has confirmed. A slower consumer catches up from the retained change log.

A source read with srs: false in the pipeline uses direct capture instead. Its recovery progress and source acknowledgements follow that pipeline's target confirmations. It depends on the source retaining the changes it has not confirmed.

Keep both the metadata store and the source logs required by the chosen path. Do not delete or restore live metadata to reset a pipeline: the source may already have released changes that remain only in tapstate's log. See State and storage.

Resume or rerun

To stop temporarily, use stop <pipeline-id> --keep-state, then start <pipeline-id>. A plain restart <pipeline-id> also retains state.
restart <pipeline-id> --rerun clears that pipeline's retained state before starting again. Review its reset and destination write strategy first; a rerun can repeat writes.
A stop without --keep-state clears the pipeline's recovery and operator state. Review the confirmation before proceeding.

Clearing one consumer does not clear the shared chain while another consumer still depends on it, including the chain's completed-table history. It does not guarantee a full source reload. Coordinate a complete reset with every dependent pipeline. State clearing does not delete rows already in the target.

If retained history is missing, capture.recovery-log-gap refuses recovery. If an older checkpoint does not prove a safe continuation, capture.recovery-progress-unproven preserves the state and refuses to guess. Keep a backup and inspect the failure before choosing a rerun. Upgrading does not repair previously unsafe recovery metadata automatically.

Full load plus CDC does not promise exactly-once output or a lossless handoff for every connector. Verify the v0.7 positioning limits and test representative changes before relying on continuity.

Stream and API sources

Stream sources start from an offset or broker-defined position. Retention controls how far back a consumer can start, and recovery can redeliver records.

API sources follow the remote service's authorization, pagination, and rate-limit rules. Check the connector guide to learn whether it performs one read, polls for updates, or uses another incremental mechanism.

Validate the chosen path

Use a staging dataset with representative data types, nulls, large values, and—when applicable—inserts, updates, and deletes. For a continuous source, stop and resume the pipeline, then compare the destination with a trusted source query.

The connector directory lists connector maturity, roles, and read modes. Continue with Resources and pipelines to see how a selected mode appears in .tap.yml.

On this page