Troubleshooting Postgres Sources
Connector Limitations
General Limitations
- The Postgres source connector currently does not handle schemas larger than 4MB.
- The Postgres source connector does not alter the schema present in your database. Depending on the destination connected to this source, however, the schema may be altered. See the destination's documentation for more details.
- The following two schema evolution actions are currently supported:
- Adding/removing tables without resetting the entire connection at the destination Caveat: In the CDC mode, adding a new table to a connection may become a temporary bottleneck. When a new table is added, the next sync job takes a full snapshot of the new table before it proceeds to handle any changes.
- Resetting a single table within the connection without resetting the rest of the destination tables in that connection
- Changing a column data type or removing a column might break connections.
Version Requirements
- For Airbyte Open Source users, upgrade your Airbyte platform to version
v0.58.0
or newer - Use Postgres v9.3.x or above for non-CDC workflows and Postgres v10 or above for CDC workflows
- For Airbyte Cloud (and optionally for Airbyte Open Source), ensure SSL is enabled in your environment
CDC Requirements
- Incremental sync is only supported for tables with primary keys. For tables without primary keys, use Full Refresh sync.
- Data must be in tables and not views. If you require data synchronization from a view, you would need to create a new connection with
Standard
asReplication Method
. - The modifications you want to capture must be made using
DELETE
/INSERT
/UPDATE
. For example, changes made usingTRUNCATE
/ALTER
will not appear in logs and therefore in your destination. - Schema changes are not supported automatically for CDC sources. Reset and resync data if you make a schema change.
- The records produced by
DELETE
statements only contain primary keys. All other data fields are unset. - Log-based replication only works for master instances of Postgres. CDC cannot be run from a read-replica of your primary database.
- An Airbyte database source using CDC replication can only be used with a single Airbyte destination. This is due to how Postgres CDC is implemented - each destination would recieve only part of the data available in the replication slot.
- Using logical replication increases disk space used on the database server. The additional data is stored until it is consumed.
- Set frequent syncs for CDC to ensure that the data doesn't fill up your disk space.
- If you stop syncing a CDC-configured Postgres instance with Airbyte, delete the replication slot. Otherwise, it may fill up your disk space.
Supported cursors
TIMESTAMP
TIMESTAMP_WITH_TIMEZONE
TIME
TIME_WITH_TIMEZONE
DATE
BIT
BOOLEAN
TINYINT/SMALLINT
INTEGER
BIGINT
FLOAT/DOUBLE
REAL
NUMERIC/DECIMAL
CHAR/NCHAR/NVARCHAR/VARCHAR/LONGVARCHAR
BINARY/BLOB
Vendor-Specific Connector Limitations
warning
Not all implementations or deployments of a database will be the same. This section lists specific limitations and known issues with the connector based on how or where it is deployed.