Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ the [**Configuration Parameters**](#configuration-parameters). Then click **Save
- **`table`**: string (required); the name of the input table in the Table storage
- **`output`**: string (required); the name of the output CSV file
- **`maxTries`**: integer (optional); the max number of retries if an error occurs; the default is `5`
- **`incremental`**: boolean (optional); enables [Incremental Loading](https://help.keboola.com/storage/tables/#incremental-loading); the default is `false`
- **`incrementalFetchingKey`**: string (optional); the name of the key for [incremental fetching](https://help.keboola.com/components/extractors/database/#incremental-fetching)
- **`incremental`**: boolean (optional); enables [Incremental Loading](/storage/tables/#incremental-loading); the default is `false`
- **`incrementalFetchingKey`**: string (optional); the name of the key for [incremental fetching](/components/extractors/database/#incremental-fetching)
- **`mode`**: enum (optional)
- `mapping` (default)
- Row is exported using specified `mapping`.
Expand Down
4 changes: 2 additions & 2 deletions src/content/docs/components/extractors/database/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Several variants of connectors may exist for each database type, depending on th
These connectors work on a relational level, performing queries against the source database to synchronize data.
This straightforward approach suits most use cases and supports Timestamp-based CDC replication.

All are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).
All are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).

Their basic configuration is also part of the [Tutorial - Loading Data from Database](/tutorial/load/database/).

Expand All @@ -55,7 +55,7 @@ database, which is quite similar to SQL databases and also supports the [advance

- [MongoDB connector](/components/extractors/database/mongodb/) for the [MongoDB](https://www.mongodb.com/) database and the [CosmosDB for MongoDB API](https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction).
- [CosmosDB connector](/components/extractors/database/cosmosdb/) for the [CosmosDB SQL API](https://docs.microsoft.com/en-us/azure/cosmos-db/tutorial-query-sql-api).
- [Azure Storage Table connector](/components/extractors/database/cosmosdb/) for the [Azure Table storage](https://azure.microsoft.com/en-us/services/storage/tables) and the [Cosmos DB Table API](https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction).
- [Azure Storage Table connector](/components/extractors/database/azure-storage-table/) for the [Azure Table storage](https://azure.microsoft.com/en-us/services/storage/tables) and the [Cosmos DB Table API](https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction).

## Connecting to Database

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This connector supports the most recent versions of both SQL Server and Azure SQ
This [standard SQL database connector](/components/extractors/database/sqldb) performs queries against the source database to synchronize data.
It offers a straightforward approach suitable for most use cases, enabling [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

All SQL database connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).
All SQL database connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/).

For guidance on basic configuration, please refer to our tutorial: [Loading Data with Database data source connector](/tutorial/load/database/).

Expand All @@ -35,7 +35,7 @@ Upon selecting a CDC-enabled table, the `CDC Mode` option will become visible in

### Functionality

Once you activate the `CDC mode`, the component will synchronize new increments using the system function [`cdc_get_net_changes](https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-net-changes-capture-instance-transact-sql?view=sql-server-ver16).
Once you activate the `CDC mode`, the component will synchronize new increments using the system function [`cdc_get_net_changes`](https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-net-changes-capture-instance-transact-sql?view=sql-server-ver16).
This utilizes dynamic boundaries based on the component state, ensuring that only the most recent changes are retrieved. The primary advantage of this method is its ability to swiftly capture all modifications, **including deletions**.

The table's initial load will undergo a full sync, similar to the standard [incremental fetching](/components/extractors/database/#incremental-fetching) mode. After the initial load, CDC tables come to play.
Expand Down
12 changes: 6 additions & 6 deletions src/content/docs/components/extractors/database/mysql/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ slug: 'components/extractors/database/mysql'

Our connectors support the latest versions of MySQL and AWS Aurora. You can choose among different strategies to synchronize your data:

- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration)
- [Log-based CDC](/components/extractors/database/mysql#log-based-binlog-cdc)
- [Query-based connector](/components/extractors/database/sqldb/#initial-setup)
- [Log-based CDC](/components/extractors/database/mysql/#mysql-log-based-cdc)

## Query-Based Connector

This [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) performs queries against the source database to sync data.
It is a straightforward approach suitable for most use cases, allowing for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

All connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and
All connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and
offer an [advanced mode](/components/extractors/database/sqldb/).

Basic configuration is covered in the [Tutorial - Loading Data from Database](/tutorial/load/database/).
Expand Down Expand Up @@ -124,7 +124,7 @@ underlying [Debezium Schema Change Topic](https://debezium.io/documentation/refe

### Data Type Mapping

MySQL data types are mapped to [Keboola Base Types](https://help.keboola.com/storage/tables/data-types/#base-types) as follows:
MySQL data types are mapped to [Keboola Base Types](/storage/tables/data-types/#base-types) as follows:

| Source Type | Base Type | Note |
|-------------|-----------|----------------------------------------------------------------------|
Expand Down Expand Up @@ -558,7 +558,7 @@ There are two types of masks available:
The connector masks the length of string columns in the output data.
The string is replaced with the specified number of `*` characters.

See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-column-mask-with-length-chars).
See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-column-mask-with-length-chars).

#### Hash Mask

Expand All @@ -570,7 +570,7 @@ Supported hash functions are described in the [MessageDigest section](https://do

***Note:** The hashing strategy version 2 is used to ensure fidelity across job runs and configurations.*

See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-column-mask-hash).
See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-column-mask-hash).

### Sync Options

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ slug: 'components/extractors/database/oracle'

## Query-Based connector

This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data.
This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-oracle) that performs queries against the source database to sync data.
It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and
They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/).

Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/).
Expand Down
55 changes: 27 additions & 28 deletions src/content/docs/components/extractors/database/postgresql/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ slug: 'components/extractors/database/postgresql'

Our connectors support the most recent versions of PostgreSQL. You may choose different strategies to sync your data:

- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration)
- [Log-based CDC](/components/extractors/database/postgresql/#log-based-cdc)
- [Query-based connector](/components/extractors/database/sqldb/#initial-setup)
- [Log-based CDC](/components/extractors/database/postgresql/#postgresql-log-based-cdc)


## Query-Based Connector

This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data.
This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-pgsql) that performs queries against the source database to sync data.
It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication.

They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and
They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and
have an [advanced mode](/components/extractors/database/sqldb/).

Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/).
Expand Down Expand Up @@ -100,7 +100,7 @@ Each result table includes the following system columns:

### Data Type Mapping

MySQL datatypes are mapped to [Keboola Base Types](https://help.keboola.com/storage/tables/data-types/#base-types) as follows:
PostgreSQL datatypes are mapped to [Keboola Base Types](/storage/tables/data-types/#base-types) as follows:

Based on the selected JSON file, the `base_type` column in the table will be updated accordingly:

Expand Down Expand Up @@ -386,20 +386,20 @@ to perform the replication.
##### PostgreSQL on Amazon RDS

It is possible to capture changes in a PostgreSQL database that is running in
link: [Amazon RDS](https://aws.amazon.com/rds/). To do this:
[Amazon RDS](https://aws.amazon.com/rds/). To do this:

* Set the instance parameter `rds.logical_replication` to `1`.
* Verify that the `wal_level` parameter is set to `logical` by running the query `SHOW wal_level` as the database RDS
master user.
This might not be the case in multi-zone replication setups.
You cannot set this option manually.
It is the
link: [automatically changed](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html)
[automatically changed](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html)
when the `rds.logical_replication` parameter is set to `1`.
If the `wal_level` is not set to `logical` after you make the preceding change, it is probably because the instance
has to be restarted after the parameter group change.
Restarts occur during your maintenance window, or you can initiate a restart manually.
* Set the {prodname} `plugin.name` parameter to `pgoutput`.
* Set the Debezium `plugin.name` parameter to `pgoutput`.
* Initiate logical replication from an AWS account that has the `rds_replication` role.
The role grants permissions to manage logical slots and to stream data using logical slots.
By default, only the master user account on AWS has the `rds_replication` role on Amazon RDS.
Expand All @@ -410,18 +410,18 @@ link: [Amazon RDS](https://aws.amazon.com/rds/). To do this:
To enable accounts other than the master account to create an initial snapshot, you must grant `SELECT` permission to
the accounts on the tables to be captured.
For more information about security for PostgreSQL logical replication, see the
link: [PostgreSQL documentation](https://www.postgresql.org/docs/current/logical-replication-security.html).
[PostgreSQL documentation](https://www.postgresql.org/docs/current/logical-replication-security.html).

##### PostgreSQL on Azure

It is possible to use {prodname} with [Azure Database for PostgreSQL](https://docs.microsoft.com/azure/postgresql/),
It is possible to use Debezium with [Azure Database for PostgreSQL](https://docs.microsoft.com/azure/postgresql/),
which has support for the `pgoutput` logical decoding.

Set the Azure replication support to `logical`. You can use the
link: [Azure CLI](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-cli) or
[Azure CLI](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-cli) or
the [Azure Portal](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-portal) to configure
this. For example, to use the Azure CLI, here are
the: [`az postgres server`](https://docs.microsoft.com/cli/azure/postgres/server?view#azure-cli-latest) commands that
the [`az postgres server`](https://docs.microsoft.com/cli/azure/postgres/server?view#azure-cli-latest) commands that
you need to execute:

```
Expand All @@ -432,7 +432,7 @@ az postgres server restart --resource-group mygroup --name myserver

##### PostgreSQL on CrunchyBridge

It is possible to use {prodname} with [CrunchyBridge](https://crunchybridge.com/); logical replication is already
It is possible to use Debezium with [CrunchyBridge](https://crunchybridge.com/); logical replication is already
turned on. The `pgoutput` plugin is available. You will have to create a replication user and provide the correct
privileges.

Expand All @@ -447,13 +447,13 @@ wal_level = logical // Instructs the server to use logical decoding
```

Depending on your requirements, you may have to set other PostgreSQL streaming replication parameters when using
{prodname}.
Debezium.
Examples include `max_wal_senders` and `max_replication_slots` for increasing the number of connectors that can access
the sending server concurrently and `wal_keep_size` for limiting the maximum WAL size which a replication slot will
retain.
For more information about configuring streaming replication, see the
link:https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER[PostgreSQL
documentation].
[PostgreSQL
documentation](https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER).

Debezium uses PostgreSQL's logical decoding, which uses replication slots.
Replication slots are guaranteed to retain all WAL segments required for Debezium even during Debezium outages. For this
Expand All @@ -470,7 +470,6 @@ Otherwise, its default value is applied, which adds a latency of about 200 milli
**TIP:** Reading and
understanding [PostgreSQL documentation about the mechanics and configuration of the PostgreSQL write-ahead log](https://www.postgresql.org/docs/current/static/wal-configuration.html)
is strongly recommended.
endif::community[]

### Setting Up Permissions

Expand Down Expand Up @@ -507,8 +506,8 @@ Keboola(Debezium) streams change events for PostgreSQL source tables from _publi
tables.
Publications contain a filtered set of change events that are generated from one or more tables.
The data in each publication is filtered based on the publication specification.
The specification can be created by the PostgreSQL database administrator or by the {prodname} connector.
To permit the {prodname} PostgreSQL connector to create publications and specify the data to replicate to them, the
The specification can be created by the PostgreSQL database administrator or by the Debezium connector.
To permit the Debezium PostgreSQL connector to create publications and specify the data to replicate to them, the
connector must operate with specific privileges in the database.

There are several options for determining how publications are created.
Expand Down Expand Up @@ -543,7 +542,7 @@ CREATE ROLE _<replication_group>_;
GRANT REPLICATION_GROUP TO __<original_owner>__;
```

3. Add the {prodname} replication user to the group.
3. Add the Debezium replication user to the group.

```sql
GRANT REPLICATION_GROUP TO __<replication_user>__;
Expand All @@ -561,14 +560,14 @@ ALTER TABLE __<table_name>__ OWNER TO REPLICATION_GROUP;

![img.png](/components/extractors/database/postgresql/img.png)

- **Host**: The hostname of the MySQL server.
- **Port**: The port number of the MySQL server.
- **User**: The username to be used to connect to the MySQL server.
- **Password**: The password to be used to connect to the MySQL server.
- **Host**: The hostname of the PostgreSQL server.
- **Port**: The port number of the PostgreSQL server.
- **User**: The username to be used to connect to the PostgreSQL server.
- **Password**: The password to be used to connect to the PostgreSQL server.

#### SSH tunnel

You may opt to use an SSH Tunnel to secure your connection. The [developer documentation](https://developers.keboola.com/integrate/database/ provides detailed instructions for setting up an SSH tunnel.
You may opt to use an SSH Tunnel to secure your connection. The [developer documentation](https://developers.keboola.com/integrate/database/) provides detailed instructions for setting up an SSH tunnel.
While setting up an SSH tunnel requires some work, it is the most reliable and secure option for connecting to your database server.

### Data Source
Expand Down Expand Up @@ -633,8 +632,8 @@ For more details, refer to the [Debezium documentation](https://debezium.io/docu

- **Replication Mode**: The replication mode to be used. The following options are available:
- `Standard`: The connector performs an initial *consistent snapshot* of each of your databases and reads
the binlog from the point at which the snapshot was made.
- `Changes only`: The connector reads the changes from the binlog immediately, skipping the initial load.
the transaction log (WAL) from the point at which the snapshot was made.
- `Changes only`: The connector reads the changes from the transaction log (WAL) immediately, skipping the initial load.
- **Binary data handler**: Specifies how binary columns, for example, blob, binary, and varbinary, should be represented in
change events. The following options are available:
- `Base64`: represents binary data as a base64-encoded String.
Expand Down Expand Up @@ -665,7 +664,7 @@ These parameters control whether the connector creates a publication and how it

More information about the publication creation process can be found in the [Publication Creation](#publication-creation) section.

![img_2.png](/components/extractors/database/postgresql/img_4.png)
![img_2.png](/components/extractors/database/postgresql/img_2.png)

- **Publication Auto Create Mode**: The mode specifying how the connector creates publications. The following options
are available:
Expand Down
Loading