MongoDB

Before you begin

Connector availability

The MongoDB connector is registered automatically by the Quickstart script. On standalone or fresh server container deployments, the connector catalog starts empty. Confirm that MongoDB is registered before authoring connections:

connectors
register mongodb

See When registration is required.

Target

Grant write access only to the destination database. The current target path does not require the clusterMonitor role.

use admin
db.createUser({
  user: "tapstate",
  pwd: "your_password",
  roles: [
    { role: "readWrite", db: "your_database" }
  ]
})

Create a connection

version: tapstate/v1
kind: source
id: operational_state
connector: mongodb
config:
  isUri: true
  uri: "mongodb://${MONGO_WRITER_USER}:${MONGO_WRITER_PASSWORD}@state-rs0-1:27017,state-rs0-2:27017/operational_state?replicaSet=state-rs0&authSource=admin"

The target deliberately omits mode and tables. Create the pipeline after the MySQL source and MongoDB target both exist:

version: tapstate/v1
kind: pipeline
id: orders_to_operational_state
source: orders_source
serve:
  from: /.*/
  sync:
    - id: write_operational_state
      source: operational_state
      write_mode: upsert
      ddl: fail

upsert writes a document by its key; append only inserts. Keep the DDL policy at fail unless the destination workflow has been staging-tested for the schema events it receives.

Validate the configuration

tapstate validate --workdir tapstate-work

Validation does not require all connection fields, reject unknown connector keys, resolve the URI, or contact MongoDB. A syntactically valid resource that uses mongo.invalid can therefore pass validation.

To check the target, register the MongoDB connector JAR, apply the workspace, and run the test from an authenticated REPL:

tapstate -w tapstate-work

connect http://127.0.0.1:8080
login admin
register mongodb -o json
apply
test operational_state -o json

Only outcome: PASSED establishes that the connector accepted the supplied endpoint and credentials. A placeholder or unreachable host should produce outcome: FAILED or a coded connector test error; the exact checks and timeout come from the registered connector. See Register and test connectors for the full sequence.

Limitations

• Confirm that each materialized collection has a stable key for upsert.
• Test source inserts, updates, deletes, and key changes against representative nested documents before consumers rely on the target.
• Verify target write concern, index creation, schema evolution, and document size limits in the MongoDB version and topology you operate.

Reference

Connection settings

Source read settings

When reading change-data-capture events from MongoDB as a pipeline source:

Tune decodeThreads upward on multi-core runtime hosts when reading from high-throughput MongoDB replica sets with heavy change volume. Increasing decodeQueueSize helps absorb bursty change events but increases heap usage.

Data compatibility

In the current materialization paths:

• Relational TIMESTAMP(3) and DATETIME map to BSON Date preserving the UTC instant at millisecond precision; source time zone or offset formatting is not stored.
• BSON Timestamp values are decoded using seconds since the Unix epoch. The intra-second counter used in internal change streams is not carried into standard date-time values.
• Portable values serialize complex types into JSON-compatible shapes (ObjectId as hex, Binary as Base64, and RegEx as /pattern/flags).