Skip to content

docs(storage): retire the dev Storage API pages, redirect to help (PRDCT-551) #416

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Iamfle4ka merged 4 commits into from
Aug 26, 2026
Merged

docs(storage): retire the dev Storage API pages, redirect to help (PRDCT-551) #416

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
597d71f
docs(storage): retire the dev Storage API pages, redirect to help (PR…
Aug 25, 2026
f04edaf
Merge branch 'main' into docs/retire-storage-api
Iamfle4ka Aug 26, 2026
9812eab
Restore CRLF in overview/index.md and overview/api/index.md
claude Aug 26, 2026
a998275
Merge branch 'main' into docs/retire-storage-api
Iamfle4ka Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 0 additions & 30 deletions _data/navigation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,36 +17,6 @@ items:
- url: /integrate/
title: Integration
items:
- url: /integrate/storage/
title: Storage
items:
- url: /integrate/storage/php-client/
title: PHP client library

- url: /integrate/storage/r-client/
title: R client library

- url: /integrate/storage/python-client/
title: Python client library

- url: /integrate/storage/docker-cli-client/
title: Docker CLI client

- url: /integrate/storage/api/
title: Using API
items:
- url: /integrate/storage/api/configurations/
title: Configurations API

- url: /integrate/storage/api/importer/
title: Storage API Importer

- url: /integrate/storage/api/import-export/
title: Manually importing and exporting data

- url: /integrate/storage/api/tde-exporter/
title: TDE Exporter

- url: /integrate/jobs/
title: Component Jobs

Expand Down
2 changes: 1 addition & 1 deletion index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ such as Extractors, Storage or Writers, that are orchestrated together through (
In this documentation, we will show you how to

- [**Integrate Keboola with other systems**](/integrate/).
- Use Keboola just to exchange data (using the [Storage API](/integrate/storage/)).
- Use Keboola just to exchange data (using the [Storage API](https://help.keboola.com/storage/api/)).
- Use Keboola as a [data-handling backbone](/overview/api/) for your product.
- Wrap Keboola in your own UI for your customers.
- Control whole data processing pipeline within Keboola from the [outside](/integrate/).
Expand Down
4 changes: 2 additions & 2 deletions integrate/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ We do not cheat or have any advantage over other developers; our UI and other co
As a result, it is very easy to, for example, write custom scripts to bootstrap a project, or do something that our UI does not offer.
Let's have a look into this!

One of the very important components is [Storage](/integrate/storage/), which not only stores all data in a
One of the very important components is [Storage](https://help.keboola.com/storage/api/), which not only stores all data in a
project, but also provides additional functions such as managing other components and their configurations.
When you are integrating your systems with Keboola, **chances are that you want to start with [Storage](/integrate/storage/)**.
When you are integrating your systems with Keboola, **chances are that you want to start with [Storage](https://help.keboola.com/storage/api/)**.


{% comment %}
Expand Down
1 change: 0 additions & 1 deletion integrate/storage/api/async-import-handling.svg
View file
Open in desktop

This file was deleted.

530 changes: 5 additions & 525 deletions integrate/storage/api/configurations.md
View file
Open in desktop

Large diffs are not rendered by default.

275 changes: 5 additions & 270 deletions integrate/storage/api/import-export.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,270 +1,5 @@
---
title: Manually Importing and Exporting Data
permalink: /integrate/storage/api/import-export/
---

* TOC
{:toc}

## Working with Data
Keboola Table Storage (Tables) and Keboola File Storage (File Uploads) are heavily connected together.
Keboola File Storage is technically a layer on top of the Amazon S3 service, and Keboola Table
Storage is a layer on top of a [database backend](https://help.keboola.com/storage/#backends).

To upload a table, take the following steps:

- Request a [file upload](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/files/prepare) from
Keboola File Storage. You will be given a destination for the uploaded file on an S3 server.
- Upload the file there. When the upload is finished, the data file will be available in the *File Uploads* section.
- Initiate an [asynchronous table import](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/tables/-id-/import-async)
from the uploaded file (use it as the `dataFileId` parameter) into the destination table.
The import is asynchronous, so the request only creates a job and you need to poll for its results.
The imported files must conform to the [RFC4180 Specification](https://tools.ietf.org/html/rfc4180).

{: .image-popup}
![Schema of file upload process](/integrate/storage/api/async-import-handling.svg)

Exporting a table from Storage is analogous to its importing. First, data is [asynchronously
exported](https://keboola.docs.apiary.io/#reference/tables/unload-data-asynchronously/asynchronous-export) from
Table Storage into File Uploads. Then you can request to [download
the file](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/files/-fileId-), which will give you
access to an S3 server for the actual file download.

### Manually Uploading a File
To upload a file to Keboola File Storage, follow the instructions outlined in the
[API documentation](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/files/prepare).
First create a file resource; to create a new file called
[`new-file.csv`](/integrate/storage/new-table.csv) with `52` bytes, call:

{% highlight bash %}
curl --request POST --header "Content-Type: application/json" --header "X-StorageApi-Token:storage-token" --data-binary "{ \"name\": \"new-file.csv\", \"sizeBytes\": 52, \"federationToken\": 1 }" https://connection.keboola.com/v2/storage/files/prepare
{% endhighlight %}

Which will return a response similar to this:

{% highlight json %}
{
"id": 192726698,
"created": "2016-06-22T10:44:35+0200",
"isPublic": false,
"isSliced": false,
"isEncrypted": false,
"name": "new_file2.csv",
"url": "https://s3.amazonaws.com/kbc-sapi-files/exp-15/1134/files/2016/06/22/192726697.new_file2?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ2N244XSWYVVYVLQ%2F20160622%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160622T084435Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature=86136cced74cdf919953cde9e2a0b837bd0b8f147aa6b7b30c2febde3b92d83d",
"region": "us-east-1",
"sizeBytes": 52,
"tags": [],
"maxAgeDays": 15,
"runId": null,
"runIds": [],
"creatorToken": {
"id": 53044,
"description": "ondrej.popelka@keboola.com"
},
"uploadParams": {
"key": "exp-15/1134/files/2016/06/22/192726697.new_file2.csv",
"bucket": "kbc-sapi-files",
"acl": "private",
"credentials": {
"AccessKeyId": "ASI...H7Q",
"SecretAccessKey": "QbO...7qu",
"SessionToken": "Ago...bsF",
"Expiration": "2016-06-22T20:44:35+00:00"
}
}
}
{% endhighlight %}

The important parts are: `id` of the file, which will be needed later, the `uploadParams.credentials` node,
which gives you credentials to AWS S3 to upload your file, and
the `key` and `bucket` nodes, which define the target S3 destination as *s3://`bucket`/`key`*.
To upload the files to S3, you need an S3 client. There are a large number of clients available:
for example, use the
[S3 AWS command line client](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).
Before using it, [pass the credentials](https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials)
by executing, for instance, the following commands

on *nix systems:
{% highlight bash %}
export AWS_ACCESS_KEY_ID=ASI...H7Q
export AWS_SECRET_ACCESS_KEY=QbO...7qu
export AWS_SESSION_TOKEN=Ago...wU=
{% endhighlight %}

or on Windows:
{% highlight bash %}
SET AWS_ACCESS_KEY_ID=ASI...H7Q
SET AWS_SECRET_ACCESS_KEY=QbO...7qu
SET AWS_SESSION_TOKEN=Ago...bsF
{% endhighlight %}

Then you can actually upload the `new-table.csv` file by executing the AWS S3 CLI [cp command](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html):
{% highlight bash %}
aws s3 cp new-table.csv s3://kbc-sapi-files/exp-15/1134/files/2016/06/22/192726697.new_file2.csv
{% endhighlight %}

After that, import the file into Table Storage, by calling either
[Create Table API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/buckets/-id-/tables-async)
(for a new table) or
[Load Data API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/tables/-id-/import-async)
(for an existing table).

{% highlight bash %}
curl --request POST --header "Content-Type: application/json" --header "X-StorageApi-Token:storage-token" --data-binary "{ \"dataFileId\": 192726698, \"name\": \"new-table\" }" https://connection.keboola.com/v2/storage/buckets/in.c-main/tables-async
{% endhighlight %}

This will create an asynchronous job, importing data from the `192726698` file into the `new-table` destination table in the `in.c-main` bucket.
Then [poll for the job results](/integrate/jobs/#job-polling), or review its status in the UI.

#### Python Example
The above process is implemented in the following example script in Python. This script uses the
[Requests](https://2.python-requests.org/en/master/) library for sending HTTP requests and
the [Boto 3](https://github.com/boto/boto3) library for working with Amazon S3. Both libraries can be
installed using pip:

{% highlight bash %}
pip install boto3
pip install requests
{% endhighlight %}

{% highlight python %}
{% include async-create.py %}
{% endhighlight %}

#### Upload Files Using Storage API Importer
For production setup, we recommend using the approach [outlined above](#manually-uploading-a-file)
with direct upload to S3 as it is more reliable and universal.
In case you need to avoid using an S3 client, it is also possible to upload the
file by a simple HTTP request to [Storage API Importer Service](/integrate/storage/api/importer/).

{% highlight bash %}
curl --request POST --header "X-StorageApi-Token:storage-token" --form "data=@new-file.csv" https://import.keboola.com/upload-file
{% endhighlight %}

The above will return a response similar to this:

{% highlight json %}
{
"id": 418137780,
"created": "2018-07-17T13:48:57+0200",
"isPublic": false,
"isSliced": false,
"isEncrypted": true,
"name": "404.md",
"url": "https:\/\/kbc-sapi-files.s3.amazonaws.com\/exp-15\/4088\/files\/2018\/07\/17\/418137779.new-file.csv...truncated",
"region": "us-east-1",
"sizeBytes": 1765,
"tags": [],
"maxAgeDays": 15,
"runId": null,
"runIds": [],
"creatorToken": {
"id": 144880,
"description": "file upload"
}
}
{% endhighlight %}

After that, import the file into Table Storage by calling either
[Create Table API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/buckets/-id-/tables-async)
(for a new table) or
[Load Data API call](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/tables/-id-/import-async)
(for an existing table).

### Working with Sliced Files
Depending on the backend and table size, the data file may be sliced into chunks.
Requirements for uploading sliced files are described in the respective part of the
[API documentation](https://api.keboola.com/?service=storage#post-/v2/storage/branch/-branchId-/files/prepare).

When you attempt to download a sliced file, you will instead obtain its manifest
listing the individual parts. Download the parts individually and join them
together. For a reference implementation of this process, see
our [TableExporter class](https://github.com/keboola/storage-api-php-client/blob/master/src/Keboola/StorageApi/TableExporter.php).

**Important:** When exporting a table through the *Table* --- *Export* UI, the file will
be already merged and listed in the *File Uploads* section with the `storage-merged-export` tag.

If you want to download a sliced file, [get credentials](https://api.keboola.com/?service=storage#get-/v2/storage/branch/-branchId-/files/-fileId-)
to download the file from AWS S3. Assuming that the file ID is 192611596, for example, call

{% highlight bash %}
curl --header "X-StorageAPI-Token: storage-token" https://connection.keboola.com/v2/storage/files/192611596?federationToken=1
{% endhighlight %}

which will return a response similar to this:

{% highlight json %}
{
"id": 192611596,
"created": "2016-06-21T15:25:35+0200",
"name": "in.c-redshift.blog-data.csv",
"url": "https://s3.amazonaws.com/kbc-sapi-files/exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csvmanifest?X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ2N244XSWYVVYVLQ%2F20160621%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20160621T135137Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3600&X-Amz-Signature=ee69d94f0af06bcf924df0f710dcd92e6503a13c8a11a86be2606552bf9a8b26",
"region": "us-east-1",
"sizeBytes": 24541,
"tags": [
"table-export"
],
...
"s3Path": {
"bucket": "kbc-sapi-files",
"key": "exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csv"
},
"credentials": {
"AccessKeyId": "ASI...UQQ",
"SecretAccessKey": "LHU...HAp",
"SessionToken": "Ago...uwU=",
"Expiration": "2016-06-22T01:51:37+00:00"
}
}
{% endhighlight %}

The field `url` contains the URL to the file manifest. Upon downloading it, you will get a JSON file with contents
similar to this:

{% highlight json %}
{
"entries": [
{"url":"s3://kbc-sapi-files/exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csv0000_part_00"},
{"url":"s3://kbc-sapi-files/exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csv0001_part_00"}
]
}
{% endhighlight %}

Now you can download the actual data file slices. URLs are provided in the manifest file, and credentials to them
are returned as part of the previous file info call. To download the files from S3, you need an S3 client. There
are a wide number of clients available; for example, use the
[S3 AWS command line client](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html). Before
using it, [pass the credentials](https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#credentials)
by executing , for instance, the following commands

on *nix systems:
{% highlight bash %}
export AWS_ACCESS_KEY_ID=ASI...UQQ
export AWS_SECRET_ACCESS_KEY=LHU...HAp
export AWS_SESSION_TOKEN=Ago...wU=
{% endhighlight %}

or on Windows:
{% highlight bash %}
SET AWS_ACCESS_KEY_ID=ASI...UQQ
SET AWS_SECRET_ACCESS_KEY=LHU...HAp
SET AWS_SESSION_TOKEN=Ago...wU=
{% endhighlight %}

Then you can actually download the files by executing the AWS S3 CLI [cp command](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html):
{% highlight bash %}
aws s3 cp s3://kbc-sapi-files/exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csv0000_part_00 192611594.csv0000_part_00
aws s3 cp s3://kbc-sapi-files/exp-2/578/table-exports/in/c-redshift/blog-data/192611594.csv0001_part_00 192611594.csv0001_part_00
{% endhighlight %}

After that, merge the files together by executing the following commands

on *nix systems:
{% highlight bash %}
cat 192611594.csv0000_part_00 192611594.csv0001_part_00 > merged.csv
{% endhighlight %}

or on Windows:
{% highlight bash %}
copy 192611594.csv0000_part_00 /B +192611594.csv0001_part_00 /B merged2.csv
{% endhighlight %}
---
title: Manually Importing and Exporting Data
permalink: /integrate/storage/api/import-export/
redirect_to: https://help.keboola.com/storage/api/import-export/
---
55 changes: 5 additions & 50 deletions integrate/storage/api/importer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,50 +1,5 @@
---
title: Storage API Importer
permalink: /integrate/storage/api/importer/
---

* TOC
{:toc}

The [whole process of importing](/integrate/storage/api/) a table into Storage can be simplified with the
Storage API Importer Service.
The Storage API Importer allows you to make an HTTP POST request and import a file directly into an existing Storage table.

The HTTP request must contain the `tableId` and `data` form fields. The specified table must already exist in [Storage](https://help.keboola.com/storage/).
Therefore to upload the `my-table.csv` CSV file (and replace the contents) into the `my-table` table in the `in.c-main` bucket,
call:

{% highlight bash %}
curl --request POST --header "X-StorageApi-Token:storage-token" --form "tableId=in.c-main.my-table" --form "data=@my-table.csv" "https://import.keboola.com/write-table"
{% endhighlight %}

Using the Storage API Importer is the easiest way to upload data into Storage (except for
using one of the [API clients](/integrate/storage/#clients)). However, the disadvantage is that the whole data file
has to be posted in a single HTTP request. **The maximum limit for a file size is 2GB and the transfer time is 45 minutes**.
This means that for substantially large files (usually more than hundreds of MB)
you may experience timeouts. If that happens, use the above outlined approach and upload the
files [directly to S3](/integrate/storage/api/import-export/#manually-uploading-a-file).

## Parameters

- `tableId` (required) Storage Table ID, example: in.c-main.users
- `data` (required) Uploaded CSV file. Raw file or compressed by [gzip](http://www.gzip.org/)
- `delimiter` (optional) Field delimiter used in a CSV file. The default value is ' , '. Use '\t' or type the tab char for tabulator.
- `enclosure` (optional) Field enclosure used in a CSV file. The default value is '"'.
- `escapedBy` (optional) CSV escape character; empty by default.
- `incremental` (optional) If incremental is set to 0 (its default), the target table is truncated before each import.

Full list of avaialable parameters is available in the [API documentation](https://api.keboola.com/?service=import#import).

## Examples
To load data incrementally (append new data to existing contents):

{% highlight bash %}
curl --request POST --header "X-StorageApi-Token:storage-token" --form "incremental=1" --form "tableId=in.c-main.my-table" --form "data=@my-table.csv" "https://import.keboola.com/write-table"
{% endhighlight %}

To load data with a non-default delimiter (tabulator) and enclosure (empty):

{% highlight bash %}
curl --request POST --header "X-StorageApi-Token:storage-token" --form "delimiter=\t" --form "enclosure=" --form "tableId=in.c-main.my-table" --form "data=@my-table.csv" "https://import.keboola.com/write-table"
{% endhighlight %}
---
title: Storage API Importer
permalink: /integrate/storage/api/importer/
redirect_to: https://help.keboola.com/storage/api/importer/
---
Loading
Loading