From 9e8f993a362dc1a7e00fba2a7d4e85a1018eccc0 Mon Sep 17 00:00:00 2001 From: "nikita.zverev2" Date: Thu, 18 Jun 2026 13:03:35 +0000 Subject: [PATCH] docs(db-extractors): fix inaccurate links, anchors, and copy-paste errors Audited the 12 database data-source connector pages against the connector source code. Fixes: - correct query-based component IDs for Oracle (keboola.ex-db-oracle) and PostgreSQL (keboola.ex-db-pgsql) - fix broken in-page anchors (#initial-setup, #mysql-log-based-cdc, #postgresql-log-based-cdc) - correct PostgreSQL page copy-paste errors (MySQL->PostgreSQL connection settings, datatype mapping intro, binlog->WAL) - point MySQL column-mask links to Debezium mysql connector docs - repair malformed/AsciiDoc-leftover links and {prodname} placeholders - fix wrong Azure Storage Table link on overview page - convert absolute help.keboola.com links to internal relative links Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .../database/azure-storage-table/index.md | 4 +- .../components/extractors/database/index.md | 4 +- .../extractors/database/ms-sql/index.md | 4 +- .../extractors/database/mysql/index.md | 12 ++-- .../extractors/database/oracle/index.md | 4 +- .../extractors/database/postgresql/index.md | 55 +++++++++---------- 6 files changed, 41 insertions(+), 42 deletions(-) diff --git a/src/content/docs/components/extractors/database/azure-storage-table/index.md b/src/content/docs/components/extractors/database/azure-storage-table/index.md index 9b08e275c..5b8443912 100644 --- a/src/content/docs/components/extractors/database/azure-storage-table/index.md +++ b/src/content/docs/components/extractors/database/azure-storage-table/index.md @@ -38,8 +38,8 @@ the [**Configuration Parameters**](#configuration-parameters). Then click **Save - **`table`**: string (required); the name of the input table in the Table storage - **`output`**: string (required); the name of the output CSV file - **`maxTries`**: integer (optional); the max number of retries if an error occurs; the default is `5` -- **`incremental`**: boolean (optional); enables [Incremental Loading](https://help.keboola.com/storage/tables/#incremental-loading); the default is `false` -- **`incrementalFetchingKey`**: string (optional); the name of the key for [incremental fetching](https://help.keboola.com/components/extractors/database/#incremental-fetching) +- **`incremental`**: boolean (optional); enables [Incremental Loading](/storage/tables/#incremental-loading); the default is `false` +- **`incrementalFetchingKey`**: string (optional); the name of the key for [incremental fetching](/components/extractors/database/#incremental-fetching) - **`mode`**: enum (optional) - `mapping` (default) - Row is exported using specified `mapping`. diff --git a/src/content/docs/components/extractors/database/index.md b/src/content/docs/components/extractors/database/index.md index 7c873c636..3c87e5bee 100644 --- a/src/content/docs/components/extractors/database/index.md +++ b/src/content/docs/components/extractors/database/index.md @@ -32,7 +32,7 @@ Several variants of connectors may exist for each database type, depending on th These connectors work on a relational level, performing queries against the source database to synchronize data. This straightforward approach suits most use cases and supports Timestamp-based CDC replication. -All are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/). +All are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/). Their basic configuration is also part of the [Tutorial - Loading Data from Database](/tutorial/load/database/). @@ -55,7 +55,7 @@ database, which is quite similar to SQL databases and also supports the [advance - [MongoDB connector](/components/extractors/database/mongodb/) for the [MongoDB](https://www.mongodb.com/) database and the [CosmosDB for MongoDB API](https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction). - [CosmosDB connector](/components/extractors/database/cosmosdb/) for the [CosmosDB SQL API](https://docs.microsoft.com/en-us/azure/cosmos-db/tutorial-query-sql-api). -- [Azure Storage Table connector](/components/extractors/database/cosmosdb/) for the [Azure Table storage](https://azure.microsoft.com/en-us/services/storage/tables) and the [Cosmos DB Table API](https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction). +- [Azure Storage Table connector](/components/extractors/database/azure-storage-table/) for the [Azure Table storage](https://azure.microsoft.com/en-us/services/storage/tables) and the [Cosmos DB Table API](https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction). ## Connecting to Database diff --git a/src/content/docs/components/extractors/database/ms-sql/index.md b/src/content/docs/components/extractors/database/ms-sql/index.md index 993ac4f87..436413d2b 100644 --- a/src/content/docs/components/extractors/database/ms-sql/index.md +++ b/src/content/docs/components/extractors/database/ms-sql/index.md @@ -14,7 +14,7 @@ This connector supports the most recent versions of both SQL Server and Azure SQ This [standard SQL database connector](/components/extractors/database/sqldb) performs queries against the source database to synchronize data. It offers a straightforward approach suitable for most use cases, enabling [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication. -All SQL database connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and offer an [advanced mode](/components/extractors/database/sqldb/). +All SQL database connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/). For guidance on basic configuration, please refer to our tutorial: [Loading Data with Database data source connector](/tutorial/load/database/). @@ -35,7 +35,7 @@ Upon selecting a CDC-enabled table, the `CDC Mode` option will become visible in ### Functionality -Once you activate the `CDC mode`, the component will synchronize new increments using the system function [`cdc_get_net_changes](https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-net-changes-capture-instance-transact-sql?view=sql-server-ver16). +Once you activate the `CDC mode`, the component will synchronize new increments using the system function [`cdc_get_net_changes`](https://learn.microsoft.com/en-us/sql/relational-databases/system-functions/cdc-fn-cdc-get-net-changes-capture-instance-transact-sql?view=sql-server-ver16). This utilizes dynamic boundaries based on the component state, ensuring that only the most recent changes are retrieved. The primary advantage of this method is its ability to swiftly capture all modifications, **including deletions**. The table's initial load will undergo a full sync, similar to the standard [incremental fetching](/components/extractors/database/#incremental-fetching) mode. After the initial load, CDC tables come to play. diff --git a/src/content/docs/components/extractors/database/mysql/index.md b/src/content/docs/components/extractors/database/mysql/index.md index 353a0cf94..0532b0177 100644 --- a/src/content/docs/components/extractors/database/mysql/index.md +++ b/src/content/docs/components/extractors/database/mysql/index.md @@ -9,15 +9,15 @@ slug: 'components/extractors/database/mysql' Our connectors support the latest versions of MySQL and AWS Aurora. You can choose among different strategies to synchronize your data: -- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration) -- [Log-based CDC](/components/extractors/database/mysql#log-based-binlog-cdc) +- [Query-based connector](/components/extractors/database/sqldb/#initial-setup) +- [Log-based CDC](/components/extractors/database/mysql/#mysql-log-based-cdc) ## Query-Based Connector This [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) performs queries against the source database to sync data. It is a straightforward approach suitable for most use cases, allowing for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication. -All connectors are [configured](/components/extractors/database/sqldb/#create-new-configuration) similarly and +All connectors are [configured](/components/extractors/database/sqldb/#initial-setup) similarly and offer an [advanced mode](/components/extractors/database/sqldb/). Basic configuration is covered in the [Tutorial - Loading Data from Database](/tutorial/load/database/). @@ -124,7 +124,7 @@ underlying [Debezium Schema Change Topic](https://debezium.io/documentation/refe ### Data Type Mapping -MySQL data types are mapped to [Keboola Base Types](https://help.keboola.com/storage/tables/data-types/#base-types) as follows: +MySQL data types are mapped to [Keboola Base Types](/storage/tables/data-types/#base-types) as follows: | Source Type | Base Type | Note | |-------------|-----------|----------------------------------------------------------------------| @@ -558,7 +558,7 @@ There are two types of masks available: The connector masks the length of string columns in the output data. The string is replaced with the specified number of `*` characters. -See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-column-mask-with-length-chars). +See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-column-mask-with-length-chars). #### Hash Mask @@ -570,7 +570,7 @@ Supported hash functions are described in the [MessageDigest section](https://do ***Note:** The hashing strategy version 2 is used to ensure fidelity across job runs and configurations.* -See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-column-mask-hash). +See the original [Debezium docs](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-column-mask-hash). ### Sync Options diff --git a/src/content/docs/components/extractors/database/oracle/index.md b/src/content/docs/components/extractors/database/oracle/index.md index 1af49a5ed..e7c589f4b 100644 --- a/src/content/docs/components/extractors/database/oracle/index.md +++ b/src/content/docs/components/extractors/database/oracle/index.md @@ -7,10 +7,10 @@ slug: 'components/extractors/database/oracle' ## Query-Based connector -This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data. +This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-oracle) that performs queries against the source database to sync data. It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication. -They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and +They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and have an [advanced mode](/components/extractors/database/sqldb/). Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/). diff --git a/src/content/docs/components/extractors/database/postgresql/index.md b/src/content/docs/components/extractors/database/postgresql/index.md index 9ccd43c39..8ff05336d 100644 --- a/src/content/docs/components/extractors/database/postgresql/index.md +++ b/src/content/docs/components/extractors/database/postgresql/index.md @@ -9,16 +9,16 @@ slug: 'components/extractors/database/postgresql' Our connectors support the most recent versions of PostgreSQL. You may choose different strategies to sync your data: -- [Query-based connector](/components/extractors/database/sqldb/#create-new-configuration) -- [Log-based CDC](/components/extractors/database/postgresql/#log-based-cdc) +- [Query-based connector](/components/extractors/database/sqldb/#initial-setup) +- [Log-based CDC](/components/extractors/database/postgresql/#postgresql-log-based-cdc) ## Query-Based Connector -This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-mysql) that performs queries against the source database to sync data. +This is a [standard connector](https://components.keboola.com/components/keboola.ex-db-pgsql) that performs queries against the source database to sync data. It is the simplest approach suitable for most use cases and allows for [time-stamp based](/components/extractors/database/#incremental-fetching) CDC replication. -They are all [configured](/components/extractors/database/sqldb/#create-new-configuration) in the same way and +They are all [configured](/components/extractors/database/sqldb/#initial-setup) in the same way and have an [advanced mode](/components/extractors/database/sqldb/). Their basic configuration is also part of the [Tutorial - Loading Data with Database Extractor](/tutorial/load/database/). @@ -100,7 +100,7 @@ Each result table includes the following system columns: ### Data Type Mapping -MySQL datatypes are mapped to [Keboola Base Types](https://help.keboola.com/storage/tables/data-types/#base-types) as follows: +PostgreSQL datatypes are mapped to [Keboola Base Types](/storage/tables/data-types/#base-types) as follows: Based on the selected JSON file, the `base_type` column in the table will be updated accordingly: @@ -386,7 +386,7 @@ to perform the replication. ##### PostgreSQL on Amazon RDS It is possible to capture changes in a PostgreSQL database that is running in -link: [Amazon RDS](https://aws.amazon.com/rds/). To do this: +[Amazon RDS](https://aws.amazon.com/rds/). To do this: * Set the instance parameter `rds.logical_replication` to `1`. * Verify that the `wal_level` parameter is set to `logical` by running the query `SHOW wal_level` as the database RDS @@ -394,12 +394,12 @@ link: [Amazon RDS](https://aws.amazon.com/rds/). To do this: This might not be the case in multi-zone replication setups. You cannot set this option manually. It is the - link: [automatically changed](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) + [automatically changed](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) when the `rds.logical_replication` parameter is set to `1`. If the `wal_level` is not set to `logical` after you make the preceding change, it is probably because the instance has to be restarted after the parameter group change. Restarts occur during your maintenance window, or you can initiate a restart manually. -* Set the {prodname} `plugin.name` parameter to `pgoutput`. +* Set the Debezium `plugin.name` parameter to `pgoutput`. * Initiate logical replication from an AWS account that has the `rds_replication` role. The role grants permissions to manage logical slots and to stream data using logical slots. By default, only the master user account on AWS has the `rds_replication` role on Amazon RDS. @@ -410,18 +410,18 @@ link: [Amazon RDS](https://aws.amazon.com/rds/). To do this: To enable accounts other than the master account to create an initial snapshot, you must grant `SELECT` permission to the accounts on the tables to be captured. For more information about security for PostgreSQL logical replication, see the - link: [PostgreSQL documentation](https://www.postgresql.org/docs/current/logical-replication-security.html). + [PostgreSQL documentation](https://www.postgresql.org/docs/current/logical-replication-security.html). ##### PostgreSQL on Azure -It is possible to use {prodname} with [Azure Database for PostgreSQL](https://docs.microsoft.com/azure/postgresql/), +It is possible to use Debezium with [Azure Database for PostgreSQL](https://docs.microsoft.com/azure/postgresql/), which has support for the `pgoutput` logical decoding. Set the Azure replication support to `logical`. You can use the -link: [Azure CLI](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-cli) or +[Azure CLI](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-cli) or the [Azure Portal](https://docs.microsoft.com/en-us/azure/postgresql/concepts-logical#using-azure-portal) to configure this. For example, to use the Azure CLI, here are -the: [`az postgres server`](https://docs.microsoft.com/cli/azure/postgres/server?view#azure-cli-latest) commands that +the [`az postgres server`](https://docs.microsoft.com/cli/azure/postgres/server?view#azure-cli-latest) commands that you need to execute: ``` @@ -432,7 +432,7 @@ az postgres server restart --resource-group mygroup --name myserver ##### PostgreSQL on CrunchyBridge -It is possible to use {prodname} with [CrunchyBridge](https://crunchybridge.com/); logical replication is already +It is possible to use Debezium with [CrunchyBridge](https://crunchybridge.com/); logical replication is already turned on. The `pgoutput` plugin is available. You will have to create a replication user and provide the correct privileges. @@ -447,13 +447,13 @@ wal_level = logical // Instructs the server to use logical decoding ``` Depending on your requirements, you may have to set other PostgreSQL streaming replication parameters when using -{prodname}. +Debezium. Examples include `max_wal_senders` and `max_replication_slots` for increasing the number of connectors that can access the sending server concurrently and `wal_keep_size` for limiting the maximum WAL size which a replication slot will retain. For more information about configuring streaming replication, see the -link:https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER[PostgreSQL -documentation]. +[PostgreSQL +documentation](https://www.postgresql.org/docs/current/runtime-config-replication.html#RUNTIME-CONFIG-REPLICATION-SENDER). Debezium uses PostgreSQL's logical decoding, which uses replication slots. Replication slots are guaranteed to retain all WAL segments required for Debezium even during Debezium outages. For this @@ -470,7 +470,6 @@ Otherwise, its default value is applied, which adds a latency of about 200 milli **TIP:** Reading and understanding [PostgreSQL documentation about the mechanics and configuration of the PostgreSQL write-ahead log](https://www.postgresql.org/docs/current/static/wal-configuration.html) is strongly recommended. -endif::community[] ### Setting Up Permissions @@ -507,8 +506,8 @@ Keboola(Debezium) streams change events for PostgreSQL source tables from _publi tables. Publications contain a filtered set of change events that are generated from one or more tables. The data in each publication is filtered based on the publication specification. -The specification can be created by the PostgreSQL database administrator or by the {prodname} connector. -To permit the {prodname} PostgreSQL connector to create publications and specify the data to replicate to them, the +The specification can be created by the PostgreSQL database administrator or by the Debezium connector. +To permit the Debezium PostgreSQL connector to create publications and specify the data to replicate to them, the connector must operate with specific privileges in the database. There are several options for determining how publications are created. @@ -543,7 +542,7 @@ CREATE ROLE __; GRANT REPLICATION_GROUP TO ____; ``` -3. Add the {prodname} replication user to the group. +3. Add the Debezium replication user to the group. ```sql GRANT REPLICATION_GROUP TO ____; @@ -561,14 +560,14 @@ ALTER TABLE ____ OWNER TO REPLICATION_GROUP; ![img.png](/components/extractors/database/postgresql/img.png) -- **Host**: The hostname of the MySQL server. -- **Port**: The port number of the MySQL server. -- **User**: The username to be used to connect to the MySQL server. -- **Password**: The password to be used to connect to the MySQL server. +- **Host**: The hostname of the PostgreSQL server. +- **Port**: The port number of the PostgreSQL server. +- **User**: The username to be used to connect to the PostgreSQL server. +- **Password**: The password to be used to connect to the PostgreSQL server. #### SSH tunnel -You may opt to use an SSH Tunnel to secure your connection. The [developer documentation](https://developers.keboola.com/integrate/database/ provides detailed instructions for setting up an SSH tunnel. +You may opt to use an SSH Tunnel to secure your connection. The [developer documentation](https://developers.keboola.com/integrate/database/) provides detailed instructions for setting up an SSH tunnel. While setting up an SSH tunnel requires some work, it is the most reliable and secure option for connecting to your database server. ### Data Source @@ -633,8 +632,8 @@ For more details, refer to the [Debezium documentation](https://debezium.io/docu - **Replication Mode**: The replication mode to be used. The following options are available: - `Standard`: The connector performs an initial *consistent snapshot* of each of your databases and reads - the binlog from the point at which the snapshot was made. - - `Changes only`: The connector reads the changes from the binlog immediately, skipping the initial load. + the transaction log (WAL) from the point at which the snapshot was made. + - `Changes only`: The connector reads the changes from the transaction log (WAL) immediately, skipping the initial load. - **Binary data handler**: Specifies how binary columns, for example, blob, binary, and varbinary, should be represented in change events. The following options are available: - `Base64`: represents binary data as a base64-encoded String. @@ -665,7 +664,7 @@ These parameters control whether the connector creates a publication and how it More information about the publication creation process can be found in the [Publication Creation](#publication-creation) section. -![img_2.png](/components/extractors/database/postgresql/img_4.png) +![img_2.png](/components/extractors/database/postgresql/img_2.png) - **Publication Auto Create Mode**: The mode specifying how the connector creates publications. The following options are available: