Apply and run a pipeline

Use this guide for CLI authoring and automation. For the web workflow, see Build a pipeline.

apply validates resource files and stores them on the server. Lifecycle commands then request the state that the runtime converges toward. Applying a changed pipeline does not restart its active run.

Before you begin

• Validate the workspace locally.
• Register the required connector JARs.
• Apply and test source and target connections from the server environment.
• Use a non-production data set for the preview.

See Register and test connectors for the preceding sequence.

Apply the workspace

Start an authenticated session in the workspace:

tapstate -w work

connect http://127.0.0.1:8080
login admin
apply

With no operand, apply sends the workspace as one batch. To apply one file or directory:

apply source/orders_source.tap.yml

The server reparses and validates each draft. The result reports:

• CREATED for a new artifact;
• UPDATED when canonical content changed;
• UNCHANGED when the stored content hash already matches.

Some checks require the source's discovered schema and therefore run only on the server. If apply returns dsl.upsert-needs-key, every selected source table written with the default upsert mode needs a discovered primary key. Discover the source again and select keyed tables, or use append only when the delivery is insert-only. A unique index or tables[].pk does not satisfy this runtime check.

Confirm the server-held resources:

ls
get orders_source
get orders_to_operational_state

Local files remain authoring input; connected get and ls show the stored resources. Source configuration is omitted from generic artifact output to protect credentials; this output is not a complete configuration backup.

Start the pipeline

start orders_to_operational_state
status orders_to_operational_state --watch

start writes a RUNNING intent at the latest applied pipeline revision. status --watch reports observed runtime state, which can lag the command briefly.

Verify data at the destination. A lifecycle command succeeding proves that the intent was accepted, not that rows arrived.

Lifecycle transitions

An illegal transition returns lifecycle.illegal-transition. If edits cannot reuse a paused pipeline's state, resume returns lifecycle.incompatible-revision. Review the change and decide whether the next run can retain state or needs a full reload.

A lifecycle request can be accepted before the runtime finishes the transition. Before starting again, wait for the status API's observedState to reach STOPPED.

Plain stop clears state. Use --keep-state when you intend to continue from retained progress. To accept clearing in a script, use stop <id> -y.

Clearing leaves this pipeline without a recovery position. Shared capture state is cleared only when its last dependent pipeline clears state, so stopping one consumer does not guarantee a full source reload. Neither choice deletes existing destination rows; reconcile them with your chosen write strategy. See State and storage.

Recover a failed pipeline deliberately:

logs orders_to_operational_state
stop orders_to_operational_state --keep-state
start orders_to_operational_state

Do not start directly from FAILED. Keeping state does not repair an invalid recovery position; investigate recovery diagnostics before retrying or clearing progress.

Verify the outcome

Use all three evidence types:

1. status reaches the expected state;
2. metrics and logs show active work without unresolved errors;
3. source and destination queries show the expected snapshot and later insert, update, and delete behavior.

The server retains recovery positions and durable operator state. A restart can resume from a retained position and replay records; it does not provide exactly-once delivery. Snapshot-only reads may restart their load. Retaining a checkpoint is safe only while its corresponding operator state is also retained. See State and storage and Configure a cluster.

Next steps

• Observe a pipeline
• Troubleshooting
• Current preview boundaries