diff --git a/_data/navigation.yml b/_data/navigation.yml index 7505562e..5a1b5162 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -14,120 +14,6 @@ items: - url: /extend/ title: Extending Keboola items: - - url: /extend/generic-extractor/ - title: Generic Extractor - items: - - url: /extend/generic-extractor/tutorial/ - title: Generic Extractor Tutorial - items: - - url: /extend/generic-extractor/tutorial/rest/ - title: REST HTTP API Introduction - - - url: /extend/generic-extractor/tutorial/json/ - title: JSON Introduction - - - url: /extend/generic-extractor/tutorial/basic/ - title: Basic Configuration - - - url: /extend/generic-extractor/tutorial/pagination/ - title: Pagination Tutorial - - - url: /extend/generic-extractor/tutorial/jobs/ - title: Jobs Tutorial - - - url: /extend/generic-extractor/tutorial/mapping/ - title: Mapping Tutorial - - - url: /extend/generic-extractor/configuration/ - title: Configuration - items: - - url: /extend/generic-extractor/configuration/api/ - title: API Configuration - items: - - url: /extend/generic-extractor/configuration/api/pagination/ - title: Pagination - items: - - url: /extend/generic-extractor/configuration/api/pagination/response-url/ - title: Response URL Scroller - - - url: /extend/generic-extractor/configuration/api/pagination/response-param/ - title: Response Parameter Scroller - - - url: /extend/generic-extractor/configuration/api/pagination/offset/ - title: Offset Scroller - - - url: /extend/generic-extractor/configuration/api/pagination/pagenum/ - title: Page Number Scroller - - - url: /extend/generic-extractor/configuration/api/pagination/cursor/ - title: Cursor Scroller - - - url: /extend/generic-extractor/configuration/api/pagination/multiple/ - title: Multiple Scrollers - - - url: /extend/generic-extractor/configuration/api/authentication/ - title: Authentication - items: - - url: /extend/generic-extractor/configuration/api/authentication/query/ - title: Query - - - url: /extend/generic-extractor/configuration/api/authentication/basic/ - title: Basic - - - url: /extend/generic-extractor/configuration/api/authentication/bearer_token/ - title: Bearer Token - - - url: /extend/generic-extractor/configuration/api/authentication/api_key/ - title: API Key - - - url: /extend/generic-extractor/configuration/api/authentication/login/ - title: Login - - - url: /extend/generic-extractor/configuration/api/authentication/oauth_cc/ - title: OAuth 2.0 Client Credentials - - - url: /extend/generic-extractor/configuration/api/authentication/oauth10/ - title: OAuth 1.0 - - - url: /extend/generic-extractor/configuration/api/authentication/oauth20/ - title: OAuth 2.0 - - - url: /extend/generic-extractor/configuration/api/authentication/oauth20-login/ - title: Login using OAuth 2.0 - - - url: /extend/generic-extractor/configuration/config/ - title: Extraction Configuration - items: - - url: /extend/generic-extractor/configuration/config/jobs/ - title: Jobs - items: - - url: /extend/generic-extractor/configuration/config/jobs/children/ - title: Child Jobs - - - url: /extend/generic-extractor/configuration/config/mappings/ - title: Mappings - - - url: /extend/generic-extractor/configuration/iterations/ - title: Iterations - - - url: /extend/generic-extractor/configuration/ssh-proxy/ - title: SSH Proxy Configuration - - - url: /extend/generic-extractor/map/ - title: Configuration Map - - - url: /extend/generic-extractor/functions/ - title: Functions - - - url: /extend/generic-extractor/incremental/ - title: Incremental Extraction - - - url: /extend/generic-extractor/running/ - title: Running Generic Extractor - - - url: /extend/generic-extractor/publish/ - title: Publishing Component - - url: /extend/generic-writer/ title: Generic Writer items: diff --git a/cli/getting-started/index.md b/cli/getting-started/index.md index 29bacfd7..6d631b8f 100644 --- a/cli/getting-started/index.md +++ b/cli/getting-started/index.md @@ -108,7 +108,7 @@ Created new config "main/extractor/ex-generic-v2/wiki" ``` Edit file `main/extractor/ex-generic-v2/wiki/config.json` as -a [Generic Extractor](/extend/generic-extractor/) configuration. A super basic +a [Generic Extractor](https://help.keboola.com/components/extractors/generic-extractor/) configuration. A super basic configuration could look like this: ```json diff --git a/extend/generic-extractor/configuration.png b/extend/generic-extractor/configuration.png deleted file mode 100644 index a2b61352..00000000 Binary files a/extend/generic-extractor/configuration.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/api_key.md b/extend/generic-extractor/configuration/api/authentication/api_key.md index 0d41446f..4658789c 100644 --- a/extend/generic-extractor/configuration/api/authentication/api_key.md +++ b/extend/generic-extractor/configuration/api/authentication/api_key.md @@ -1,74 +1,5 @@ --- title: API Key Authentication permalink: /extend/generic-extractor/configuration/api/authentication/api_key/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/api_key/ --- - -API Key token authentication sends a token in either a header or query parameter of each API request. - -E.g., Headers: `X-StorageApi-Token:your_token` - -This method is available through UI. You can select the `Api Key Auth` method and fill in the token. - -{: .image-popup} -![Api Key](/extend/generic-extractor/configuration/api/authentication/api_key.png) - -### Configuration parameters - -- `Key` - arbitrary name of the header or query parameter key, e.g., `X-StorageApi-Token` -- `Token` - the actual token value -- `Add to` - where to add the token, either to the headers or query parameters - - -### JSON - -In the underlying JSON, the API Key is implemented as follows: - -Place your token into the `config.#__AUTH_TOKEN` parameter. The `Authorization` header is then constructed using the `concat` function. - -**Header section** - -{% highlight json %} -{ - "api": { - ..., - "http": { - "http": { - "headers": { - "X-StorageApi-Token": { - "attr": "#__AUTH_TOKEN" - } - } - } - } - }, - "config": { - "#__AUTH_TOKEN": "secret", - "jobs": [...] - } -} -{% endhighlight %} - - -**Query section** - -{% highlight json %} -{ - "api": { - ..., - "http": { - "http": { - "params": { - "X-StorageApi-Token": { - "attr": "#__AUTH_TOKEN" - } - } - } - } - }, - "config": { - "#__AUTH_TOKEN": "secret", - "jobs": [...] - } -} -{% endhighlight %} - diff --git a/extend/generic-extractor/configuration/api/authentication/api_key.png b/extend/generic-extractor/configuration/api/authentication/api_key.png deleted file mode 100644 index e3b86952..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/api_key.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/auth_ui.png b/extend/generic-extractor/configuration/api/authentication/auth_ui.png deleted file mode 100644 index 0a85af67..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/auth_ui.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/basic.md b/extend/generic-extractor/configuration/api/authentication/basic.md index 27da35b1..21a7f3c7 100644 --- a/extend/generic-extractor/configuration/api/authentication/basic.md +++ b/extend/generic-extractor/configuration/api/authentication/basic.md @@ -1,80 +1,5 @@ --- title: Basic Authentication permalink: /extend/generic-extractor/configuration/api/authentication/basic/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/basic/ --- - -Basic Authentication provides the [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) -method. It requires entering a username and password in the configuration and sends the encoded values in the -`Authorization` header. - -### User Interface - -In the user interface, you simply select the `Basic Authorization` method and enter the username and password. - -### JSON - -A sample Basic authentication looks like this: - -{: .image-popup} -![Basic](/extend/generic-extractor/configuration/api/authentication/basic.png) - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "basic" - } - }, - "config": { - "#username": "JohnDoe", - "#password": "secret" - } -} -{% endhighlight %} - -The `username` and `password` fields are part of the [`config` section](/extend/generic-extractor/configuration/config/). -They are also prefixed by the hash `#` character, which means they are stored [encrypted](/overview/encryption/). -If the API expects something else than a username and password in the `Authorization` header, or if it requires -a custom authorization header, use the [Default Headers option](/extend/generic-extractor/configuration/api/#headers). - -## Configuration Parameters -This `basic` type of authentication has no configuration parameters. The login and password must be provided in the -[`config` section](/extend/generic-extractor/configuration/config/) of the Generic Extractor configuration. - - - -## Basic Configuration Example -Assume you have an API which requires you to use the HTTP Basic authentication to send the login and password in -the `Authorization` header. Assume that your login is `JohnDo` and password is `secret`. The following -configuration solves the situation: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com", - "authentication": { - "type": "basic" - } - }, - "config": { - "debug": true, - "#username": "JohnDoe", - "#password": "secret", - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users" - } - ] - } - } -} -{% endhighlight %} - -The following HTTP header will be sent: - - Authorization: Basic Sm9obkRvZTpzZWNyZXQ= - -See [example [EX078]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/078-basic-auth). diff --git a/extend/generic-extractor/configuration/api/authentication/basic.png b/extend/generic-extractor/configuration/api/authentication/basic.png deleted file mode 100644 index 92af3c7a..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/basic.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/bearer.png b/extend/generic-extractor/configuration/api/authentication/bearer.png deleted file mode 100644 index c686dd51..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/bearer.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/bearer_token.md b/extend/generic-extractor/configuration/api/authentication/bearer_token.md index 0f8957b4..0f341964 100644 --- a/extend/generic-extractor/configuration/api/authentication/bearer_token.md +++ b/extend/generic-extractor/configuration/api/authentication/bearer_token.md @@ -1,45 +1,5 @@ --- title: Bearer Token Authentication permalink: /extend/generic-extractor/configuration/api/authentication/bearer_token/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/bearer_token/ --- - -Bearer token authentication sends a token in the `Authorization` header of each API request. - -This method is available through UI. You can select the `Bearer Token` method and fill in the token. - -{: .image-popup} -![img.png](/extend/generic-extractor/configuration/api/authentication/bearer.png) - - - -### JSON - -In the underlying JSON, the Bearer Token is implemented as follows: - -Place your token into the `config.#__BEARER_TOKEN` parameter. The `Authorization` header is then constructed using the `concat` function. - -{% highlight json %} -{ - "api": { - ..., - "http": { - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "Bearer ", - { - "attr": "#__BEARER_TOKEN" - } - ] - } - } - } - }, - "config": { - "#__BEARER_TOKEN": "secret", - "jobs": [...] - } -} -{% endhighlight %} - diff --git a/extend/generic-extractor/configuration/api/authentication/index.md b/extend/generic-extractor/configuration/api/authentication/index.md index 74858e50..5e770fef 100644 --- a/extend/generic-extractor/configuration/api/authentication/index.md +++ b/extend/generic-extractor/configuration/api/authentication/index.md @@ -1,73 +1,5 @@ --- title: Authentication permalink: /extend/generic-extractor/configuration/api/authentication/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/ --- - -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/).* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -Unless the API you want to extract from is completely public, you need an authentication and possibly authorization method. -There are many authentication methods available. Generic Extractor supports the following ones: - -- [URL Query](/extend/generic-extractor/configuration/api/authentication/query/) authentication --- sends credentials in the URL of each API request. -- [Basic HTTP](/extend/generic-extractor/configuration/api/authentication/basic/) authentication --- sends credentials in the `Authorization` header of each API request. -- [Login](/extend/generic-extractor/configuration/api/authentication/login/) authentication --- obtains temporary credentials (token) by logging in and then sends them in the URL or headers of each API request. -- [OAuth 1.0](/extend/generic-extractor/configuration/api/authentication/oauth10/) authentication --- authenticates with [OAuth 1.0 scheme](#oauth). -- [OAuth 2.0](/extend/generic-extractor/configuration/api/authentication/oauth20/) authentication --- authenticates with [OAuth 2.0 scheme](#oauth). -- [OAuth 2.0 Login](/extend/generic-extractor/configuration/api/authentication/oauth20-login/) authentication --- -crossover between the [OAuth 2.0](/extend/generic-extractor/configuration/api/authentication/oauth20/) and -[Login](/extend/generic-extractor/configuration/api/authentication/login/) authentication. - -**NOTE:** The [UI](/extend/generic-extractor/configuration/#user-interface) also offers additional authentication methods that are implemented as a template: -- [Bearer Token](/extend/generic-extractor/configuration/api/authentication/bearer_token/) -- [API Key](/extend/generic-extractor/configuration/api/authentication/api_key/) -- [oAuth 2.0 Client Credentials](/extend/generic-extractor/configuration/api/authentication/oauth_cc/) - -Use the authentication method supported by the target API. If the API supports multiple -authentication methods, the [URL Query](/extend/generic-extractor/configuration/api/authentication/query/) and -[Basic HTTP](/extend/generic-extractor/configuration/api/authentication/basic/) methods are the easiest to set up but also the least secure. - -**User Interface** - -Most of the authentication methods are available via the user interface: - -{: .image-popup} -![Auth ui](/extend/generic-extractor/configuration/api/authentication/auth_ui.png) - - -An example authentication JSON configuration looks like this: - -{% highlight json %} -{ - "api": { - "authentication": { - "type": "query", - "query": { - "apiKey": "2267709" - } - } - }, - "config": { - ... - } -} -{% endhighlight %} - -## OAuth -Generic Extractor also supports authentication using the [OAuth](https://en.wikipedia.org/wiki/OAuth) standard. -Due to the principles of OAuth, this authentication method is supported only for [published components](/extend/generic-extractor/publish/). -The OAuth protocol defines a scheme in which credentials are exchanged between the following: - -- Consumer (Generic Extractor) -- Service provider (the API itself) -- End-user (the person authenticating against the API) - -The OAuth specification defines what kind of information is exchanged in which steps. It is not a precise -specification and leaves quite some freedom for implementation. Also, there are two versions of -OAuth --- 1.0 and 2.0. They are completely incompatible (both the authentication steps and the exchanged fields differ). -Generic Extractor supports both [OAuth 1.0](/extend/generic-extractor/configuration/api/authentication/oauth10/) -and [OAuth 2.0](/extend/generic-extractor/configuration/api/authentication/oauth20/). If you are developing a new component using Generic Extractor -[templates](/extend/generic-extractor/publish/#submission) and want to use and test OAuth authentication, -[inject the necessary credentials](https://help.keboola.com/extend/common-interface/oauth/#credentials-injection) simply by passing them -in the configuration's `authorization` property. diff --git a/extend/generic-extractor/configuration/api/authentication/login.md b/extend/generic-extractor/configuration/api/authentication/login.md index e6f674da..d1714869 100644 --- a/extend/generic-extractor/configuration/api/authentication/login.md +++ b/extend/generic-extractor/configuration/api/authentication/login.md @@ -1,663 +1,5 @@ --- title: Login permalink: /extend/generic-extractor/configuration/api/authentication/login/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/login/ --- - -* TOC -{:toc} - -Use the Login authentication to send a one-time **login request** to obtain temporary credentials -for authentication of all the other API requests. - - -## User Interface - -Note that this configuration option is not yet covered. You can add the JSON configuration using the `Custom` auth method. - -{: .image-popup} -![Login](/extend/generic-extractor/configuration/api/authentication/login.png) - -However, this method is reused in the oAuth 2.0 Client Credentials authentication method available in the UI. - -## JSON - -A sample Login authentication looks like this: - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "format": "json", - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - } - } - }, - "config": { - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `login` type of authentication: - -- `loginRequest` (required, object) --- a [job-like](/extend/generic-extractor/configuration/config/jobs/) object describing the login request; it has the following properties: - - `endpoint` (required, string) --- an API endpoint for the login request; the same rules as for the [Job `endpoint`](/extend/generic-extractor/configuration/config/jobs/#specifying-endpoint) apply here. - - `params` (optional, object) --- an object with key-value properties containing request parameters; object keys are parameters names; values are transformed the [same way as in jobs](/extend/generic-extractor/configuration/config/jobs/#request-parameters). - - `method` (optional, string) --- an HTTP method to send the request; this defines how the [parameters are sent](/extend/generic-extractor/configuration/config/jobs/#request-parameters) to the API. The default value is `GET`. - - `headers` (optional, object) --- an object with key-value properties containing HTTP headers. The names will be used as HTTP header names, and the values will be used as the value of the respective header. -- `format` (optional, string) --- defines the expected format of `loginRequest`; the allowed values are `json` (default) and `text`. If the format is text, then it is converted to a json with field `data` (see [example](#configuration-with-headers-and-text-response). This conversion will also be applied in case the response is a JSON [scalar](/extend/generic-extractor/tutorial/json/#data-values). But in that case, format `json` has to be used (see [example [EX129]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/129-login-auth-scalar)). -- `apiRequest` (optional, object) --- an object which defines how the result of the **login request** will be used in the actual API request; it contains the following properties: - - `headers` (optional, object) --- an object with key-value properties containing HTTP headers. The names are header names, the values are paths in the JSON response from which the actual values are extracted. - - `query` (optional, object) --- an object with key-value properties containing URL query parameters. The names are parameter names, and the values are paths in the JSON response from which the actual values are extracted. -- `expires` (optional, mixed) --- either an integer value specifying a fixed number of seconds after which the **login request** will be sent again (see an [example](#expiration-basic)); or, an object with the following properties: - - `response` (required, string) --- a path in the JSON response which contains the expiration time. It can be either: - - a string which can be processed by the [`strtotime` function](https://www.php.net/manual/en/function.strtotime.php) (see an [example](#expiration-from-response)), or - - a numeric [timestamp](https://en.wikipedia.org/wiki/Unix_time) (with `"relative": false`), or - - a number of seconds for which the credentials are valid (with `"relative": true`). - - `relative` (optional, boolean) --- When true, the expiration time is relative to the current time. The default value is `false`. - -Note that the values in `apiRequest.headers` and `apiRequest.query` take precedence over the values specified in the -`api.http.defaultOptions.headers` (see an [example](#parameter-overriding)). If `expires` is not set, the login request -is called only once before all other requests. To call the login request before every request (e.g., to obtain access token from refresh token), set `"expires": 0`. - - - -## Examples -Below are several examples showing you how to use various login authentication related features in Generic Extractor. - -### Configuration with Headers -Let's say you have an API which requires every API call to be authorized with the `X-ApiToken` header. The value of that header (an API -token) is obtained by calling the `/login` endpoint with the headers `X-Login` and `X-Password`. The `/login` endpoint response looks -like this: - -{% highlight json %} -{ - "authorization": { - "token": "a1b2c3d435f6" - } -} -{% endhighlight %} - -The following API configuration deals with the authentication: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - } - } -} -{% endhighlight %} - -The first request will be sent to `/login` with the HTTP headers: - - X-Login: JohnDoe - X-Password: TopSecret - -All consecutive requests will be sent to the endpoints specified in the [`jobs`](/extend/generic-extractor/configuration/config/jobs/) section and -will contain the header: - - X-ApiToken: a1b2c3d435f6 - -See [example [EX079]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/079-login-auth-headers). - -### Configuration with Headers and Text Response -Let's say you have an API like the above, but it returns the login response as a plain text: - - a1b2c3d435f6 - -The following API configuration deals with the authentication: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "format": "text", - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "data" - } - } - } - } -} -{% endhighlight %} - -The first request will be sent to `/login` with the HTTP headers: - - X-Login: JohnDoe - X-Password: TopSecret - -The plain text response `a1b2c3d435f6` is converted to the following JSON: - -{% highlight json %} -{ - "data": "a1b2c3d435f6" -} -{% endhighlight %} - -Therefore you can access it in the headers configuration using the `"response": "data"` reference. -All consecutive requests will be sent to the endpoints specified in the [`jobs`](/extend/generic-extractor/configuration/config/jobs/) section and -will contain the header: - - X-ApiToken: a1b2c3d435f6 - -See [example [EX128]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/128-login-auth-text). - -### Configuration with Query Parameters -Let's say you have an API which requires an [HTTP POST](https://en.wikipedia.org/wiki/POST_(HTTP)) request with `username` and -`password` to the endpoint `/login/form`. -On a successful login, it returns the following response: - -{% highlight json %} -{ - "authentication": [ - { - "secret": "a1b2c3d435f6" - }, - { - "token": { - "id": 123 - } - } - ] -} -{% endhighlight %} - -The actual API requests then must contain the `secretKey` and `tokenId` parameters in the URL. -The following `authentication` configuration takes care of the situation: - -{% highlight json %} -"authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login/form", - "method": "FORM", - "params": { - "username": "JohnDoe", - "password": "TopSecret" - } - }, - "apiRequest": { - "query": { - "secretKey": { - "response": "authentication.0.secret" - }, - "tokenId": { - "response": "authentication.1.token.id" - } - } - } -} -{% endhighlight %} - -The first API request will be sent as: - - POST /login/form - - username=JohnDoe&password=TopSecret - -The [FORM method](/extend/generic-extractor/configuration/config/jobs/#form) sends the parameters -as [application/x-www-form-urlencoded](https://en.wikipedia.org/wiki/POST_(HTTP)#Use_for_submitting_web_forms). -The `apiRequest.query` settings then map the response values to the parameters of the other API calls, -so the second API call will be sent as: - - GET /users?secretKey=a1b2c3d435f6&tokenId=123 - -See [example [EX080]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/080-login-auth-query). -Notice that the example uses completely different URL for the login request. - -### Parameter Overriding -The above examples show how to use query parameters and headers separately. However, they can be mixed freely; they can also be -mixed with parameters and headers entered elsewhere in the configuration. The following example shows how parameters from -different places are merged together: - -
- Click to expand the example. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "http": { - "defaultOptions": { - "headers": { - "X-Mode": "development", - "X-Account-Id": 123 - }, - "params": { - "debug": "1", - "orderBy": "default", - "secretKey": "123" - } - } - }, - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "POST", - "params": { - "username": "JohnDoe", - "password": "TopSecret" - } - }, - "apiRequest": { - "query": { - "apiToken": { - "response": "authorization.token" - }, - "secretKey": { - "response": "authorization.secretKey" - }, - "customerId": { - "response": "authorization.accountId" - } - }, - "headers": { - "X-SecretKey": { - "response": "authorization.secretKey" - }, - "X-Account-Id": { - "response": "authorization.accountId" - } - } - } - } - }, - "config": { - "debug": true, - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "users", - "params": { - "orderBy": "userName", - "secretKey": "none", - "customerId": "234" - } - } - ] - } - } -} -{% endhighlight %} - -
-
-The example configuration sends its first request as: - - POST /login - - {"username":"JohnDoe","password":"TopSecret"} - -with this header: - - Content-Type: application/json - -Notice that in the case of the **login request** both `headers` and `params` from `api.defaultOptions` are ignored. Only the -`headers` and `params` from `api.authentication.loginRequest` are used (and encoded as JSON because of `"method": "POST"`). - -The second API call is sent as: - - GET /users?debug=1&orderBy=userName&secretKey[0]=none&secretKey[1]=a1b2c3d435f6&customerId[0]=234&customerId[1]=abc&apiToken=987654 - -with this header: - - X-Mode: development - X-Account-Id: abc - X-SecretKey: a1b2c3d435f6 - -The request URL contains the following query parameters: - -- `debug=1` --- coming from the `api.http.defaultOptions.params` option -- `orderBy=userName` --- coming from the `config.jobs.params` option, which overrides the value specified in `api.http.defaultOptions.params` -- `secretKey[0]=none` --- coming from the `config.jobs.params` option, which overrides the value specified in `api.http.defaultOptions.params` -- `secretKey[1]=a1b2c3d435f6` --- coming from the `api.authentication.apiRequest.query` option -- `customerId[0]=234` --- coming from the `config.jobs.params` option -- `customerId[1]=abc` --- coming from the `api.authentication.apiRequest.query` option -- `apiToken=987654` --- coming from the `api.authentication.apiRequest.query` option - -The request headers contain: - -- `X-Mode` --- coming from the `api.http.defaultOptions.headers` option -- `X-Account-Id` --- coming from the `api.authentication.apiRequest.headers` option, which overrides the values specified in `api.http.defaultOptions.headers` -- `X-SecretKey` --- coming from the `api.authentication.apiRequest.headers` option - -As you can see, the headers specified elsewhere are **overwritten** by the `api.authentication.apiRequest` while the parameters -specified elsewhere are **merged** with the `api.authentication.apiRequest`. - -See [example [EX081]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/081-login-auth-headers-query-override). - -### Expiration Basic -It is possible that the credentials provided by the **login request** have a time-limited validity. This is handled by the `expires` -option. If the obtained credentials are always valid for a certain period of time, for example for 1 hour, modify the [first -example](#configuration-with-headers) to this: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - }, - "expires": "3600" - } - } - } -} -{% endhighlight %} - -This causes Generic Extractor to call the **login request** every hour. - -See [example [EX082]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/082-login-auth-expires). - -### Expiration from Response -In case the credentials provided by the **login request** have a time-limited validity, use the `expires` option. -If the validity of the credentials is returned in the response, modify the [first example](#configuration-with-headers) to this: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - }, - "expires": { - "response": "authorization.expires" - } - } - } - } -} -{% endhighlight %} - -This assumes that the response of the **login request** looks like this: - -{% highlight json %} -{ - "authorization": { - "token": "a1b2c3d435f6", - "expires": "2017-02-20 12:34:45" - } -} -{% endhighlight %} - -See [example [EX083]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/083-login-auth-expires-date). - -### Relative Expiration from Response -In case the API returns credentials validity in the **login request** and that validity is expressed in seconds, -use the `expires` option together with setting `relative` to `true`. -The result is the behavior of the [first example](#expiration-basic) but the value is taken -from the response as in the [second example](#expiration-from-response). - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - }, - "expires": { - "response": "authorization.expires", - "relative": true - } - } - } - } -} -{% endhighlight %} - -This assumes that the response of the **login request** looks like this: - -{% highlight json %} -{ - "authorization": { - "token": "a1b2c3d435f6",/ - "expires": 3600 - } -} -{% endhighlight %} - -See [example [EX084]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/084-login-auth-expires-seconds). - -### Login Authentication with Functions -Suppose you have an API which requires you to send a username and password separated by a colon and -base64 encoded --- for example, `JohnDoe:TopSecret` (base64 encoded to `Sm9obkRvZTpUb3BTZWNyZXQ=`) in the -`X-Authorization` header to an `/auth` endpoint. The login endpoint then returns a token -which can be used with other API calls. - -The following configuration reads both the login and password parameters from the -[`config` section](/extend/generic-extractor/functions/#configuration-attributes) and -uses the `login` authorization method to send them to the special `/auth` endpoint. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "auth", - "headers": { - "X-Authorization": { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "#login" - }, - ":", - { - "attr": "#password" - } - ] - } - ] - } - } - }, - "apiRequest": { - "headers": { - "X-Api-Token": "token" - } - } - } - }, - "config": { - "#login": "JohnDoe", - "#password": "TopSecret", - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - } -} -{% endhighlight %} - -See [example [EX100]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/100-function-login-headers). - -### Login Authentication with Login and API Request -Suppose you have an API similar to the one in the [previous example](#login-authentication-with-functions). -It requires you to send a username and password separated by a colon and -base64 encoded --- for example, `JohnDoe:TopSecret` (base64 encoded to `Sm9obkRvZTpUb3BTZWNyZXQ=`) in the -`X-Authorization` header to an `/auth` endpoint. The difference is that the login endpoint returns a token -which must be further processed. The other API requests expects that the received token is concatenated again -with the user name --- for example, `JohnDoe:d868d581b2f` and an SHA1 hash is generated (e.g. -`09e4e6977b72ecc9fa2120f49a4a74f5c268d277`). This value must be sent as an `auth` query parameter. - -The `loginRequest` part of the following configuration is the same as in the -[previous example](#login-authentication-with-functions) --- it reads both the login and password parameters -from the `config` section and uses the `login` authorization method to send them to the special `/auth` endpoint. -The `apiRequest` part of the configuration uses the [`sha1`](/extend/generic-extractor/functions/#sha1) function on -the result of the [`concat`](/extend/generic-extractor/functions/#concat) function. The `concat` function takes -the login parameter from the [`config` section](/extend/generic-extractor/functions/#configuration-attributes) -(via `"attr": "#login"` reference) and the token parameter from the -[response of the login request](/extend/generic-extractor/functions/#login-authentication-context) -(via `"response": "authorization.token"` reference). - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "auth", - "headers": { - "X-Authorization": { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "#login" - }, - ":", - { - "attr": "#password" - } - ] - } - ] - } - } - }, - "apiRequest": { - "query": { - "auth": { - "function": "sha1", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "#login" - }, - ":", - { - "response": "authorization.token" - } - ] - } - ] - } - } - } - } - }, - "config": { - "#login": "JohnDoe", - "#password": "TopSecret", - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - } -} -{% endhighlight %} - -See [example [EX117]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/117-function-login-params-response) -or [example [EX118]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/118-function-login-headers-response), which uses headers. diff --git a/extend/generic-extractor/configuration/api/authentication/login.png b/extend/generic-extractor/configuration/api/authentication/login.png deleted file mode 100644 index 49d15e83..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/login.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth10-diagram.png b/extend/generic-extractor/configuration/api/authentication/oauth10-diagram.png deleted file mode 100644 index 8ed840a1..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth10-diagram.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth10.md b/extend/generic-extractor/configuration/api/authentication/oauth10.md index a77285af..77ad5261 100644 --- a/extend/generic-extractor/configuration/api/authentication/oauth10.md +++ b/extend/generic-extractor/configuration/api/authentication/oauth10.md @@ -1,120 +1,5 @@ --- title: OAuth 1.0 Authentication permalink: /extend/generic-extractor/configuration/api/authentication/oauth10/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/oauth10/ --- - - - -**Note** that this configuration option is not yet supported and the test endpoint button will not work. - -OAuth 1.0 authentication is one of the [two OAuth methods](/extend/generic-extractor/configuration/api/authentication/#oauth) and -is supported only for [published components](/extend/generic-extractor/publish/). -It is configured by setting the `type` key to `oauth10`: - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "oauth10" - } - }, - "config": { - ... - } -} -{% endhighlight %} - -No other configuration parameters are necessary (nor available). The OAuth authentication process is -described by the [following diagram](https://oauth.net/core/1.0/#anchor9): - -![OAuth 1.0 Diagram](/extend/generic-extractor/configuration/api/authentication/oauth10-diagram.png) - -In the diagram, only the step `G` represents the actual communication with the API (extraction of data). -The final authorization section of the Generic Extractor configuration is generated between -steps `F` and `G`. When a component is published, the steps `A` - `F` of the process are handled by -Keboola (and the end-user). - -To **develop and test** a new component with the OAuth authorization, go through -the steps `A` - `F` manually. At the last step, you obtain a response containing the fields -`oauth_token` and `oauth_token_secret`, e.g.: - -{% highlight json %} -{ - "oauth\_token": "JohnDoe1234", - "oauth\_token\_secret": "TopSecret5678" -} -{% endhighlight %} - -Then, inject the OAuth credentials into the configuration root: - -{% highlight json %} -{ - "parameters": { - "api": { - ... - }, - "config": { - ... - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"oauth\_token\":\"JohnDoe1234\",\"oauth\_token\_secret\":\"TopSecret5678\"}", - "appKey": 1234, - "#appSecret": "TopSecret" - } - } - } -} -{% endhighlight %} - -The `authorization` field has a single property `oauth_api`, which has a single property `credentials` with three child properties: - -- `#data` --- contains the response from the service provider, the response is a JSON string (not an object!). -- `appKey` --- contains the [Consumer Key](https://oauth.net/core/1.0/#anchor3). -- `#appSecret` --- contains the [Consumer Secret](https://oauth.net/core/1.0/#anchor3) (use an empty string if -not used by the service provider). - -With the above configuration, Generic Extractor generates the `Authorization` header; the signature -method used is [HMAC-SHA1](https://oauth.net/core/1.0/#anchor16). For example: - - Authorization: OAuth oauth_consumer_key="1234", oauth_nonce="72469d96572dabb4d0ea02b057ea4f246d722b72", oauth_signature="zl0y5CyySCPj8IqODV3Egjqgg6Q%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1492904452", oauth_token="JohnDoe1234", oauth_version="1.0" - -The full configuration is, e.g.: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "oauth10" - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"oauth_token\":\"userToken\",\"oauth_token_secret\":\"tokenSecret\"}", - "appKey": 1234, - "#appSecret": "TopSecret" - } - } - } -} -{% endhighlight %} - -See [example [EX102]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/102-oauth1) or -learn [more about Keboola-OAuth integration](https://help.keboola.com/extend/common-interface/oauth/). diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-diagram.png b/extend/generic-extractor/configuration/api/authentication/oauth20-diagram.png deleted file mode 100644 index bee41534..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth20-diagram.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-login-console.png b/extend/generic-extractor/configuration/api/authentication/oauth20-login-console.png deleted file mode 100644 index 83c16a72..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth20-login-console.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-1.png b/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-1.png deleted file mode 100644 index 88117a4e..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-1.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-2.png b/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-2.png deleted file mode 100644 index 742cfb52..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-2.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-3.png b/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-3.png deleted file mode 100644 index 5e56f439..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-3.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20-login.md b/extend/generic-extractor/configuration/api/authentication/oauth20-login.md index 5db93316..8486f4ae 100644 --- a/extend/generic-extractor/configuration/api/authentication/oauth20-login.md +++ b/extend/generic-extractor/configuration/api/authentication/oauth20-login.md @@ -1,236 +1,5 @@ --- title: Login Using OAuth 2.0 Authentication permalink: /extend/generic-extractor/configuration/api/authentication/oauth20-login/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/oauth20-login/ --- - -* TOC -{:toc} - -**Note** that this configuration option is not yet supported and the test endpoint button will not work. - -The OAuth Login method is useful when you need to send a one-time **login request** to obtain temporary credentials -for authentication of all the other API requests. A sample OAuth Login authentication looks like this: - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "oauth20.login", - "loginRequest": { - "endpoint": "login", - "method": "GET", - "headers": { - "X-Login": "JohnDoe", - "X-Password": "TopSecret" - } - }, - "apiRequest": { - "headers": { - "X-ApiToken": { - "response": "authorization.token" - } - } - } - } - }, - "config": { - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The configuration parameters are identical to the [Login](/extend/generic-extractor/configuration/api/authentication/login/) method. -The difference, however, is in the [function context](/extend/generic-extractor/functions/#oauth-20-login-authentication-context). -The **login request** is assumed to require the OAuth2 authorization and its response must be in JSON format (plaintext is not supported). - -## Examples -The following examples demonstrate how to use OAuth with a basic login request and Google API in Generic Extractor. - -### Basic Configuration -The following configuration shows how to set up an OAuth **login request**: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://mock-server:80/105-oauth2-login/", - "authentication": { - "type": "oauth20.login", - "loginRequest": { - "endpoint": "token", - "headers": { - "X-Refresh-Token": { - "user": "refresh_token" - }, - "X-App-Key": { - "consumer": "client_id" - } - } - }, - "apiRequest": { - "headers": { - "X-Access-Token": { - "response": "credentials.access_token" - } - } - } - } - }, - "config": { - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"status\": \"ok\",\"refresh_token\": \"1234abcd5678efgh\"}", - "appKey": "someId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -First, an OAuth login is negotiated. The result of this authentication is a response from the API (inserted into -the `authorization.oauth_api.credentials.#data` property): - -{% highlight json %} -{ - "status": "ok", - "refresh_token": "1234abcd5678efgh" -} -{% endhighlight %} - -This is sent to the `/token` endpoint with the following headers: - - X-Refresh-Token: 1234abcd5678efgh - X-App-Key: someId - -This API call then returns the following response: - -{% highlight json %} -{ - "credentials": { - "validUntil": "2017-10-04 12:45:09", - "access_token": "mkoijn098uhbygv" - } -} -{% endhighlight %} - -From that, the value of the `credentials.access_token` property is taken, inserted into the `X-Access-Token` header -and sent to other API requests (`/users`). - -See [example [EX105]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/105-oauth2-login). - - -### Google API Configuration -The following example shows how to set up the OAuth authentication for Google APIs. The access token is refreshed with each API call. - -#### Generate access tokens -First, visit [Google API Console](https://console.developers.google.com/apis/credentials) to obtain OAuth 2.0 credentials, such as a -**Client ID** and **Client secret**. - -Add `https://developers.google.com/oauthplayground` to Authorized redirect URIs: - -{: .image-popup} -![Google API Console](/extend/generic-extractor/configuration/api/authentication/oauth20-login-console.png) - -Then, generate **access** and **refresh** tokens using [Google OAuth 2.0 Playground](https://developers.google.com/oauthplayground/). - -Provide your Client ID and Client secret in the settings of OAuth 2.0 Playground: - -{: .image-popup} -![Google OAuth 2.0 Playground 1](/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-1.png) - -Make sure the Offline Access option is checked and close the settings dialog. - -On the left side, choose which scopes you would like to authorize and click on **Authorize APIs**. - -{: .image-popup} -![Google OAuth 2.0 Playground 1](/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-2.png) - -Then exchange the authorization code for tokens: - -{: .image-popup} -![Google OAuth 2.0 Playground 1](/extend/generic-extractor/configuration/api/authentication/oauth20-login-playground-3.png) - -Use the generated tokens in the Generic Extractor configuration. - -#### Configuration -Paste the JSON response with the generated access and refresh tokens as a string under the `#data` key in -`authorization.oauth_api.credentials`. -Escape double quotes `"` in the JSON response and preferably remove newlines too, so it looks like this: - -`{\"access_token\": \"MY_ACCESS_TOKEN\",\"refresh_token\": \"MY_REFRESH_TOKEN\"}` - -Here is a complete configuration example for AdSense: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://www.googleapis.com", - "authentication": { - "type": "oauth20.login", - "loginRequest": { - "endpoint": "/oauth2/v4/token", - "method": "FORM", - "headers": { - "Content-Type": "application/x-www-form-urlencoded" - }, - "params": { - "client_id": { - "consumer": "client_id" - }, - "client_secret": { - "consumer": "client_secret" - }, - "refresh_token": { - "user": "refresh_token" - }, - "grant_type": "refresh_token" - } - }, - "apiRequest": { - "query": { - "access_token": { - "response": "access_token" - } - } - } - } - }, - "config": { - "debug": true, - "outputBucket": "adsense", - "jobs": [ - { - "endpoint": "/adsense/v1.4/reports/", - "dataField": "rows", - "dataType": "rows" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"access_token\": \"MY_ACCESS_TOKEN\",\"refresh_token\": \"MY_REFRESH_TOKEN\"}", - "appKey": "MY_CLIENT_ID", - "#appSecret": "MY_CLIENT_SECRET" - } - } - } -} -{% endhighlight %} - -See [example [EX114]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/114-oauth2-google). diff --git a/extend/generic-extractor/configuration/api/authentication/oauth20.md b/extend/generic-extractor/configuration/api/authentication/oauth20.md index 464ecbf0..198fd951 100644 --- a/extend/generic-extractor/configuration/api/authentication/oauth20.md +++ b/extend/generic-extractor/configuration/api/authentication/oauth20.md @@ -1,269 +1,5 @@ --- title: OAuth 2.0 Authentication permalink: /extend/generic-extractor/configuration/api/authentication/oauth20/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/oauth20/ --- - -* TOC -{:toc} - -OAuth 2.0 Authentication is one of [two OAuth methods](/extend/generic-extractor/configuration/api/authentication/#oauth) and -is supported only for [components registered in the developer portal](/extend/generic-extractor/publish/). -It is configured by setting the `type` key to `oauth20`: - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "oauth20" - } - }, - "config": { - ... - } -} -{% endhighlight %} - -The OAuth 2.0 authentication process is described by the [following diagram](https://docs.spring.io/spring-social/docs/1.0.0.M3/reference/html/serviceprovider.html): - -![Diagram - OAuth 2.0 authentication](/extend/generic-extractor/configuration/api/authentication/oauth20-diagram.png) - -In the diagram, step `6` represents the end of authentication and the actual communication with -the API (extraction of data) may begin. -The final authorization section of the Generic Extractor configuration is generated between -steps `5` and `6`. When a component is published, steps `1` --- `6` of the process are handled by -Keboola (and the end-user). - -To **develop and test** a new component with the OAuth authorization, go through -steps `1` --- `6` manually. At step `5`, you will obtain a response which needs to be put -in the `authorization.oauth_api.credentials.data` section of the configuration. The response can be -either plaintext or a JSON. Let's say you obtain a simple plaintext string: - - SomeToken1234abcd567ef - -The following configuration needs to be supplied to Generic Extractor: - -{% highlight json %} -{ - "parameters": { - "api": { - ... - }, - "config": { - ... - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "SomeToken1234abcd567ef", - "appKey": "clientId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -The `authorization` field has a single property `oauth_api` with a single property `credentials`. This -has three child properties: - -- `#data` --- contains the response from the service provider; the response is a plaintext string or a JSON string (not an object!). -- `appKey` --- contains the Client ID (use an empty string if not used by the service provider). -- `#appSecret` --- contains the Client Secret (use an empty string if not used by the service provider). - -Note that the properties `appKey` and `#appSecret` must exist even if not used by the API; set them -to empty strings. For more information about OAuth 2, see the [official documentation](https://oauth.net/2/) -or learn [more about Keboola-OAuth integration](https://help.keboola.com/extend/common-interface/oauth/). - -## Configuration Parameters -The following configuration parameters are supported for the `oauth20` authentication type: - -- `format` (optional, string) --- If the OAuth service provider response is JSON, use the only possible -value -- `json`. Otherwise do not specify format at all (plaintext is assumed). -- `headers` (optional, object) --- Object whose properties represent the key-value pairs sent as HTTP headers. -- `query` (optional, object) --- Object whose properties represent the key-value pairs of the URL query. - -At least one of the `headers` or `query` options should always be specified; otherwise no authentication -will be sent with the API requests. Both fields also allow and practically require using -[functions](/extend/generic-extractor/functions/) to generate an OAuth signature. Specific authentication values -are available in the [OAuth function context](/extend/generic-extractor/functions/#oauth-20-authentication-context). - -## Examples -The following two examples demonstrate the support for OAuth 2 in Generic Extractor. - -### Bearer Authentication -The most basic OAuth authentication method is with "Bearer Token". If you have an API which supports -this authentication method, the following configuration can be used: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://example.com/", - "authentication": { - "type": "oauth20", - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "Bearer ", - { - "authorization": "data" - } - ] - } - } - } - }, - "config": { - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "SomeToken1234abcd567ef", - "appKey": "clientId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -The response obtained from the service provider (the API) is a plaintext string `SomeToken1234abcd567ef`, which -is simply a token to be used to access other API calls. The `api.authentication.headers` section creates -the header `Authorization: Bearer SomeToken1234abcd567ef` using the -[`concat` function](/extend/generic-extractor/functions/#concat). - -See [example [EX103]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/103-oauth2-bearer). - -### HMAC Authentication -If you have an API which requires an [HMAC](https://en.wikipedia.org/wiki/Hash-based_message_authentication_code) -signed token, generate the correct signature using [functions](/extend/generic-extractor/functions). -The following example assumes you obtain the following response from the API upon authentication: - -{% highlight json %} -{ - "status": "ok", - "access_token": "testToken", - "mac_secret": "iAreSoSecret123" -} -{% endhighlight %} - -The user token is represented by the `access_token` and the token secret (MAC secret) is contained in the -`mac_secret` property. The following configuration generates the MAC signed `Authorization` header: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://example.com/", - "authentication": { - "type": "oauth20", - "format": "json", - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "MAC id=\"", - { - "authorization": "data.access_token" - }, - "\", ts=\"", - { - "authorization": "timestamp" - }, - "\", nonce=\"", - { - "authorization": "nonce" - }, - "\", mac=\"", - { - "function": "md5", - "args": [ - { - "function": "hash_hmac", - "args": [ - "sha256", - { - "function": "implode", - "args": [ - "\n", - [ - { - "authorization": "timestamp" - }, - { - "authorization": "nonce" - }, - { - "request": "method" - }, - { - "request": "resource" - }, - { - "request": "hostname" - }, - { - "request": "port" - }, - "\n" - ] - ] - }, - { - "authorization": "data.mac_secret" - } - ] - } - ] - } - ] - } - } - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"status\": \"ok\",\"access_token\": \"testToken\", \"mac_secret\": \"iAreSoSecret123\"}", - "appKey": "clientId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -The above configuration generates the following header: - - Authorization: MAC id="testToken", ts="1492958193", nonce="605cce2a2f687253", mac="ae96f93def8f02770f30e858e074b2a7 - -The configuration probably looks rather complicated. Most of it is to generate the `mac` value in the above header. -The first step is the [`implode` function](/extend/generic-extractor/functions/#implode) generating a -[Normalized request string](https://tools.ietf.org/html/draft-ietf-oauth-v2-http-mac-01#section-3.2.1). This is then -passed to the [`hash_hmac` function](/extend/generic-extractor/functions/#hash_hmac) along with the -parameters `sha256` (specifying the hashing algorithm) and the hashing key taken from the `authorization` property -`data.mac_secret`. The last (topmost) step is the [`concat` function](/extend/generic-extractor/functions/#concat); it -concatenates all parts of the `Authorization` header. - -See [example [EX104]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/104-oauth2-hmac). diff --git a/extend/generic-extractor/configuration/api/authentication/oauth_cc.md b/extend/generic-extractor/configuration/api/authentication/oauth_cc.md index a975bb51..d4431af5 100644 --- a/extend/generic-extractor/configuration/api/authentication/oauth_cc.md +++ b/extend/generic-extractor/configuration/api/authentication/oauth_cc.md @@ -1,145 +1,5 @@ --- title: oAuth 2.0 Client Credentials Authentication permalink: /extend/generic-extractor/configuration/api/authentication/oauth_cc/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/oauth_cc/ --- - -oAuth 2.0 Client Credentials authentication performs the [oAuth 2.0 client_credentials flow](https://auth0.com/docs/get-started/authentication-and-authorization-flow/client-credentials-flow). - -This method is available through the UI and is implemented via the [Login](/extend/generic-extractor/configuration/api/authentication/login/) method. - -{: .image-popup} -![img.png](/extend/generic-extractor/configuration/api/authentication/oauth_cc.png) - -### Configuration Parameters - -- `Login Request type` - - `Basic Auth`: The client_id and client_secret are sent in the Authorization header as a Basic authorization, e.g. `Authorization: Basic base64(client_id:client_secret)`. - - `Post Form`: The client_id and client_secret are sent as form data in the POST request body, e.g., `{login_url}?client_id=client_id&client_secret=client_secret`. -- `Client ID`: The client_id value -- `Client Secret`: The client_secret value -- `Access Token URL`: The URL where the access token is requested, e.g., `https://login-demo.io/oauth/v2/oauth-token` -- `Scope`: The scope of the access token. It is specific per API and can be left empty. - -### JSON - -In the underlying JSON, the method is implemented as follows: - -Place your secret into the `config.#__CLIENT_SECRET` and ID into the `config.#__CLIENT_ID` parameter. The `Authorization` header is then constructed using the `concat` function. - -**Basic Auth** - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "login", - "format": "json", - "loginRequest": { - "endpoint": "https://login-demo.io/oauth/v2/oauth-token", - "method": "FORM", - "headers": { - "Accept": "application/json", - "Authorization": { - "function": "concat", - "args": [ - "Basic ", - { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "__CLIENT_ID" - }, - ":", - { - "attr": "#__CLIENT_SECRET" - } - ] - } - ] - } - ] - } - }, - "params": { - "grant_type": "client_credentials", - "scope": "read" - } - }, - "apiRequest": { - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "Bearer ", - { - "response": "access_token" - } - ] - } - } - } - } - }, - "config": { - "__CLIENT_ID": "CLIENT ID" - "#__CLIENT_SECRET": "secret", - "jobs": [...] - } -} -{% endhighlight %} - - - -**Post Form** - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "login", - "format": "json", - "loginRequest": { - "endpoint": "https://login-demo.io/oauth/v2/oauth-token", - "method": "FORM", - "headers": { - "Accept": "application/json" - }, - "params": { - "grant_type": "client_credentials", - "scope": "read", - "client_id": { - "attr": "__CLIENT_ID" - }, - "client_secret": { - "attr": "#__CLIENT_SECRET" - } - } - }, - "apiRequest": { - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "Bearer ", - { - "response": "access_token" - } - ] - } - } - } - } - }, - "config": { - "__CLIENT_ID": "CLIENT ID" - "#__CLIENT_SECRET": "secret", - "jobs": [...] - } -} -{% endhighlight %} - diff --git a/extend/generic-extractor/configuration/api/authentication/oauth_cc.png b/extend/generic-extractor/configuration/api/authentication/oauth_cc.png deleted file mode 100644 index eb301e27..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/oauth_cc.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/authentication/query.md b/extend/generic-extractor/configuration/api/authentication/query.md index 2a0cf704..77e875e8 100644 --- a/extend/generic-extractor/configuration/api/authentication/query.md +++ b/extend/generic-extractor/configuration/api/authentication/query.md @@ -1,91 +1,5 @@ --- title: Query Authentication permalink: /extend/generic-extractor/configuration/api/authentication/query/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/authentication/query/ --- - -Query Authentication provides the simplest authentication method, in which -the credentials are sent in the [request URL](/extend/generic-extractor/tutorial/rest#url). -This method is most often used with APIs that authenticate using API tokens and -signatures. Dynamic values of query parameters can be generated using -[user functions](/extend/generic-extractor/functions/). - - -## User Interface -In the user interface, you simply select the `Query` method and enter the key-value pairs of the query parameters. - -{: .image-popup} -![img.png](/extend/generic-extractor/configuration/api/authentication/query.png) - -***Note:** Unless you need multiple parameters, the [API Key authentication method](/extend/generic-extractor/configuration/api/authentication/api_key) may be more suitable.* - - -## JSON - -A sample Query authentication configuration looks like this: - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "query", - "query": { - "apikey": "2267709" - } - } - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `query` type of authentication: - -- `query` (required, object): An object whose properties represent key-value pairs of the URL query. - -## Basic Configuration Example -Let's say you have an API that requires an `api-token` parameter (with value 2267709) to be sent with -each request. The following authentication configuration does exactly that: - -{% highlight json %} -"authentication": { - "type": "query", - "query": { - "api-token": "2267709" - } -} -{% endhighlight %} - -For this use case, it is also possible to use the [`defaultOptions` setting](/extend/generic-extractor/configuration/api/#default-parameters). -However, we recommend using the `authentication` setting for credentials so that the Generic Extractor -configuration remains organized. - -See [example [EX077]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/077-query-auth). - -## Configuration With Encrypted Token Example -Usually, you want the value used for authentication to be encrypted (the `api-token` parameter with the value 2267709 in our example), so you do not expose it to other users or store it in the configuration -versions history. The following authentication configuration, combined with the parameter defined in the [`config`](https://developers.keboola.com/extend/generic-extractor/configuration/config/) section, does -that (the value with the prefix `#` is encrypted upon saving the configuration): - -{% highlight json %} -{ - "api": { - ..., - "authentication": { - "type": "query", - "query": { - "api-token": { - "attr": "#token" - } - } - } - }, - "config": { - ..., - "#token": "2267709" - } - } -} -{% endhighlight %} - - -See [example [EX094]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/094-function-config-headers). diff --git a/extend/generic-extractor/configuration/api/authentication/query.png b/extend/generic-extractor/configuration/api/authentication/query.png deleted file mode 100644 index 6bba5c4d..00000000 Binary files a/extend/generic-extractor/configuration/api/authentication/query.png and /dev/null differ diff --git a/extend/generic-extractor/configuration/api/index.md b/extend/generic-extractor/configuration/api/index.md index b9040a73..fcb83ebb 100644 --- a/extend/generic-extractor/configuration/api/index.md +++ b/extend/generic-extractor/configuration/api/index.md @@ -1,392 +1,5 @@ --- title: API Configuration permalink: /extend/generic-extractor/configuration/api/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/ --- - -* TOC -{:toc} - -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/basic/).* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -The API section of Generic Extractor configuration **describes global characteristics of an API**. These include -[HTTP headers](/extend/generic-extractor/tutorial/rest/#headers), authentication and pagination methods. - -A sample API configuration can look like this: - -{% highlight json %} -{ - ..., - "api": { - "baseUrl": "https://example.com/v3.0/", - "caCertificate": "-----BEGIN CERTIFICATE-----\nMIIFaz....", - "pagination": { - "method": "offset", - "offsetParam": "offset", - "limitParam": "count" - }, - "authentication": { - "type": "basic" - }, - "retryConfig": { - "account": 3 - }, - "http": { - "headers": { - "Accept": "application/json" - }, - "defaultOptions": { - "params": { - "company": 123 - } - }, - "requiredHeaders": ["X-AppKey"], - "ignoreErrors": [405], - "connectTimeout": 30, - "requestTimeout": 300 - } - } -} -{% endhighlight %} - -## Base URL -The `baseUrl` configuration **defines the URL to which the API requests should be sent**. We -recommend that the URL ends with a slash so that the `jobs.endpoint` can be set easily. -See the [`endpoint` configuration](/extend/generic-extractor/configuration/config/jobs/#endpoint) for a detailed description of -how `api.baseUrl` and `jobs.endpoint` work together. - -## CA certificate -The `caCertificate` configuration **defines custom certificate authority bundle in -[`crt`/`pem` format](https://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file)**. -It allows connecting to a HTTPS server with a untrusted/self-signed certificate. -The value is not certificate of the server, but a certificate of the certificate authority used to generate the server certificate. -You can define a single root certificate, or a bundle of root and intermediate certificates -(see [EX141](https://github.com/keboola/generic-extractor/tree/master/doc/examples/141-https-self-signed)). - -## Client certificate -The `#clientCertificate` configuration **defines the client certificate and private key**. This is required -if the server requires two-way SSL authentication, so in addition to the verification of the server, -the server also verifies the client (see [EX142](https://github.com/keboola/generic-extractor/tree/master/doc/examples/142-https-client-cert)). - -**Value is the client certificate, followed by the private key. Both -in [`crt`/`pem` format](https://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file)**. - -Example: -```json -{ - "api": { - "baseUrl": "https://my-server.com", - "#clientCertificate": "-----BEGIN CERTIFICATE-----\n...\n----END CERTIFICATE-----\n-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n" - } -} -``` - -## Pagination -Pagination (or scrolling) **describes how the API pages through a large set of results**. Because -there are many different pagination strategies, the configuration is described on a -[separate page](/extend/generic-extractor/configuration/api/pagination/). - -## Authentication -Authentication (authorization) needs to be configured for any API which is not public. -Because there are many authorization methods used by different APIs, there are also many -[configuration options](/extend/generic-extractor/configuration/api/authentication/). - -## Retry Configuration -By default, Generic Extractor **automatically retries failed HTTP requests** --- repeatedly, and on most errors. -This is one of the big advantages over writing your own extractor from scratch. Tweak the retry setting to optimize -the speed of an extraction or to avoid unwanted flooding of the API. - -Every HTTP response contains a [Status code](/extend/generic-extractor/tutorial/rest/#http-status) and, -optionally, a Header describing the situation or further actions. Status codes 2xx (beginning with 2; e.g., 200 -OK) represent success and no action is needed for them. Status codes 3xx (e.g., 301 Moved Permanently) represent -redirection and are automatically handled by Generic Extractor (the redirection is followed). - -This leaves us with status codes 4xx (e.g., 404 Not Found) and 5xx (e.g., 500 Internal Server Error). The 4xx codes -represent the codes whose error is on the client side. 5xx represent errors on the server side. When -retrying, this distinction is really irrelevant because we need to use the codes that represent transient/temporary -errors. Unfortunately, there is no definitive official list of those. When it comes to communicating with -a real world API, the typical examples of transient errors are: - -- Network outage/malfunction -- Target server maintenance/outage -- API throttling/rate limiting - -The rate limiting behaviour is not universally agreed upon. A nice API should return a -`503 Service Unavailable` status together with a `Retry-After` HTTP header specifying number of -seconds to wait before the next request. This is, however, not supported by many APIs. -**Adjusting to the API rate limiting is the main reason for changing Retry Configuration**. - -The next aspect to consider is "when to retry". Even if the error is transient, retrying -immediately (within few milliseconds) usually makes no sense because the error is probably still not gone. -There are two retry strategies: - -- Either the API sends a `Retry-After` header (or its equivalent), or -- Generic Extractor uses an [exponential backoff algorithm](https://en.wikipedia.org/wiki/Exponential_backoff). - -### API Retry Strategy -Per the HTTP specification, the API may send the [`Retry-After`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After) -header which should contain number of seconds to pause/sleep before the next request. Generic Extractor -supports some extensions to this. First, the *Retry Header* name may be customized. Second, the header -value may be as follows: - -- Number of seconds before the next request -- [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time) of the time of the next request -- String date in [RFC 1123 format](https://www.php.net/manual/en/class.datetime.php#datetime.constants.rfc1123) of the -time of the next request - -The second and third options are often called **Rate Limit Reset** as they describe when the next successful request -can be made (i.e., the limit is reset). - -### Backoff Strategy -The exponential backoff in Generic Extractor is defined as `truncate(2^(retry\_number - 1)) * 1000` seconds. -This means that the first retry (zero-based index) will be after 0 seconds (`(2^(0-1)) = 0.5`, truncated to 0). -The retry delays are the following: - -|retry|1|2|3|4|5|6|7|8|9|10|11|12| -|---|---|---|---|---|---|---|---|---|---|---|---|---| -|delay|0s|1s|2s|4s|8s|16s|32s|64s|128s (~2min)|256s (~4min)|512s (~8.5min)|1024s (~17min)| - -The default number of retries is **10** which means that the retries stop after -511 seconds (~8.5 minutes). - -### Configuration -The default Retry configuration `retryConfig` is: - -{% highlight json %} -{ - "http": { - "retryHeader": "Retry-After", - "codes": [500, 502, 503, 504, 408, 420, 429], - "maxRetries": 10 - }, - "curl": { - "codes": [28, 6, 7, 35, 52, 56] - } -} -{% endhighlight %} - -The above defined `curl.codes` cover the common network errors. You can find a full list of -supported codes in the [cURL documentation](https://curl.haxx.se/libcurl/c/libcurl-errors.html). -There is no way to set the actual backoff strategy as it is derived automatically from the -content of the HTTP header specified in `retryHeader`. Generic Extractor will fallback to the -exponential backoff strategy in case the header contents are invalid (that includes, e.g., a typo -in the header name). Make sure to check that the backoff is correct --- the times can be verified -in the [debug](/extend/generic-extractor/running/#debug-mode) messages: - - Http request failed, retrying in 1s - -If the exponential backoff is used, you will see its sequence of times. -See an [example](/extend/generic-extractor/configuration/api/#retry-configuration). - -## Default HTTP Options -The `http` configuration option allows you to set the timeouts, default headers and parameters sent with each API call -(defined later in the [`jobs` section](/extend/generic-extractor/configuration/config/jobs/#request-parameters)). - -### Headers -The `http.headers` configuration allows you to **set the default headers sent with -each API call**. The configuration is an object where names are the names of -the headers and values are their values --- for instance: - -{% highlight json %} -"http": { - "headers": { - "Accept": "application/json", - "Accept-Encoding": "gzip" - } -} -{% endhighlight %} - -See the full [example](/extend/generic-extractor/configuration/api/#default-headers). - -### Request Parameters -The `http.defaultOptions.params` configuration allows you to **set the -[request parameters](/extend/generic-extractor/tutorial/rest/#url) to be -sent with each API request**. The same rules apply as to the -[`jobs.params`](/extend/generic-extractor/configuration/config/jobs/#request-parameters). - -See an [example](/extend/generic-extractor/configuration/api/#default-headers). - -### Required Headers -Similar to the `http.headers` option, the `http.requiredHeaders` option allows you to **set the HTTP header -for every API request**. The difference is that the `requiredHeaders` configuration specifies **only the header names**. -The actual values must be provided in the [`config`](/extend/generic-extractor/configuration/config/) -configuration section. This is useful in case the header values change dynamically or they are provided as part -of [template configuration](/extend/generic-extractor/publish/). - -If the `api` configuration section looks like this: - -{% highlight json %} -"http": { - "requiredHeaders": ["Accept", "Accept-Encoding"] -} -{% endhighlight %} - -then the header values must be provided in the `config` configuration section: -{% highlight json %} -"http": { - "headers": { - "Accept": "application/json", - "Accept-Encoding": "gzip" - } -} -{% endhighlight %} - -Failing to provide the header values in the `config` section will cause an error: - - Missing required header Accept in config.http.headers! - -See the full [example](/extend/generic-extractor/configuration/api/#required-headers). - -### Ignore Errors -The `ignoreErrors` option allows you to force Generic Extractor to ignore certain extraction errors. -The option lists HTTP codes for which any errors occurring during downloading -and JSON parsing the response will be ignored. The `ignoreErrors` option error is an array of HTTP -response status codes; the default value is an empty array. - -If the `ignoreErrors` is set to a non-empty array -- for instance: - -{% highlight json %} -"http": { - "ignoreErrors": [404] -} -{% endhighlight %} - -Then the following happens: - -- A response with status 2XX is processed normally. -- A response with status 404 is processed as if it were a success response. - - If parsing of the response body JSON succeeds, it is added as any other row. - - If parsing of the response body JSON fails, it is added as a row with the `errorData` field. -- A response with status 4XX (other than 404) causes the extraction to fail. -- A response with status 5XX causes the request to be [retried](#retry-configuration). If that does -not help, it causes the extraction to fail. - -If the `ignoreErrors` contains 5XX status codes, the [Retry rules](#retry-configuration) are still applied. -But regardless of the outcome of the retries, the response will be considered as success. - -See [example [EX132]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/132-ignore-errors). - -**Important**: Use this feature with **caution**! It is designed to workaround weird or buggy REST -API implementations and should not be used blindly if other solutions may be applied (e.g., -[`responseFilter`](/extend/generic-extractor/configuration/config/jobs/#response-filter). When ignoring errors, -**you might miss even those errors that require your attention.** - -### Connect Timeout - -The `connectTimeout` option is a float describing the number of seconds to wait while trying to connect to a server. -Default value is `30` seconds. Use `0` to wait indefinitely, we do not recommend it. - -{% highlight json %} -"http": { - "connectTimeout": 30 -} -{% endhighlight %} - - -### Request Timeout - -The `requestTimeout` option is a float describing the total timeout of the request in seconds. -Default value is `300` seconds. Use `0` to wait indefinitely, we do not recommend it. - -{% highlight json %} -"http": { - "requestTimeout": 300 -} -{% endhighlight %} - -## Examples - -### Retry Configuration -Assume that you have an API which implements throttling in the following way: when -the number of requests is exceeded, it returns an empty response with the status code `202` and -a timestamp when a new requests can be made in the `X-RetryAfter` HTTP header. -Then create the following API configuration to make Generic Extractor handle the -situation: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com/", - "retryConfig": { - "http": { - "retryHeader": "X-RetryAfter", - "codes": [500, 502, 503, 504, 408, 420, 429, 202] - }, - "maxRetries": 3 - } -} -{% endhighlight %} - -Notice that it is necessary to add the response code `202` to the existing default codes. I.e., setting -`"codes": [202]` is likely very wrong. - -See [example [EX037]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/037-retry-header). - -### Default Headers -Assume that you have an API which returns a JSON response only if the client sends an -`Accept: application/json` header. Additionally, if the client sends an -`Accept-Encoding: gzip` header, the HTTP transmission will be compressed (and thus faster). -The following configuration sends both headers with every API request: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com/", - "http": { - "headers": { - "Accept": "application/json", - "Accept-Encoding": "gzip" - } - } -} -{% endhighlight %} - -See [example [EX038]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/038-default-headers). - -### Default Parameters -Assume that you have an API requiring all requests to contain a filter -for the account to which they belong. This is done by passing the `account=XXX` parameter. -The following configuration sends the parameter with every API request: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com/", - "http": { - "defaultOptions": { - "params": { - "account": 123 - } - } - } -} -{% endhighlight %} - -For this use case, the [query authentication](/extend/generic-extractor/configuration/api/authentication/query/) -may also be used. - -See [example [EX039]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/039-default-parameters). - -### Required Headers -Assume that an API requires the header `X-AppKey` to be sent with each -API request. The following API configuration can be used: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com", - "http": { - "requiredHeaders": ["X-AppKey"] - } -}, -{% endhighlight %} - -Then the actual header value must be added to the `config` section. - -{% highlight json %} -"http": { - "headers": { - "X-AppKey": "ThisIsSecret" - } -} -{% endhighlight %} - -For this use case, the [authentication](/extend/generic-extractor/configuration/api/authentication/) may also be used. -See [example [EX040]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/040-required-headers). diff --git a/extend/generic-extractor/configuration/api/pagination/cursor.md b/extend/generic-extractor/configuration/api/pagination/cursor.md index 67e463d5..90d69c2e 100644 --- a/extend/generic-extractor/configuration/api/pagination/cursor.md +++ b/extend/generic-extractor/configuration/api/pagination/cursor.md @@ -1,149 +1,5 @@ --- title: Cursor Scroller permalink: /extend/generic-extractor/configuration/api/pagination/cursor/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/cursor/ --- - -* TOC -{:toc} - -The Cursor Scroller can be used with an API which expects the client to maintain a cursor (pointer) -to the last obtained item. For example, on the first request, it returns items with ID 1-100; for the second -request, you must tell the API to start with ID 101. - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "cursor", - "idKey": "id", - "param": "startWith", - "increment": 1 - }, - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `cursor` method of pagination: - -- `idKey` (required, string) --- path to the key which contains the value of the cursor; the path is entered relative to the exported items. -- `param` (required, string) --- name of the [query string](/extend/generic-extractor/tutorial/rest/#url) parameter in which the above **cursor value** should be sent to the API. -- `increment` (optional, integer) --- value by which the cursor value will be incremented/decremented; the default value is `0`. -- `reverse` (optional, boolean) --- when `true`, the cursor is reversed; the default value is `false`. - -In default mode, Generic Extractor examines the response and finds the **maximum** value in the -property specified in the `idKey`. Then it adds `increment` to the value and sends it to the -API in a parameter whose name is in the `param` value. If the cursor is set to reverse, -the **minimum** value of the `idKey` is taken. The `increment` is added to that value, so it should be probably -set to negative when using `reverse=true`. -The request parameter specified in the `param` configuration overwrites the parameter of the same name defined in the -[job parameters](/extend/generic-extractor/configuration/config/jobs/#request-parameters). Other job parameters are carried over without modification -(see an [example](#reverse-configuration)). - -### Stopping Condition -The pagination ends **when the `dataField` of the response contains no items**. Because of this, each -run with the `cursor` scroller produces a similar warning: - - Warning: datafield 'items' contains no data! - -This is expected behavior. [Common stopping conditions](/extend/generic-extractor/configuration/api/pagination/#stopping-strategy) also apply. - -## Examples -This section contains two API pagination examples where the Cursor Scroller is used. - -### Basic Configuration -Let's say you have an API which has an endpoint `/users` returning the following response: - -{% highlight json %} -{ - "items": [ - { - "name": "Jimmy Doe", - "fields": { - "id": 345 - } - }, - { - "name": "Jenny Doe", - "fields": { - "id": 456 - } - } - ] -} -{% endhighlight %} - -For the next page of results, the API requires you to send a request to `/users?continueAfter=456`. The following -configuration handles the case: - -{% highlight json %} -"pagination": { - "method": "cursor", - "idKey": "fields.id", - "param": "continueAfter" -} -{% endhighlight %} - -Notice that the `idKey` parameter is relative to the extracted array of items (`fields.id` and not `items[].fields.id`). - -See [example [EX060]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/060-pagination-cursor-basic). - -### Reverse Configuration -Some APIs return items starting with the newest item and therefore need to be queried for offset in -reverse order. Let's say a request to `/users?startWith=last` will produce: - -{% highlight json %} -{ - "items": [ - { - "id": 345, - "name": "Jimmy Doe" - }, - { - "id": 456, - "name": "Jenny Doe" - } - ] -} -{% endhighlight %} - -To retrieve the next set of results, send a request to `GET /users?startWith=344` --- i.e. with the -ID lower than the lowest one already retrieved. The following configuration does exactly that: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "cursor", - "idKey": "id", - "param": "startWith", - "increment": -1, - "reverse": true - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "startWith": "last" - } - } - ] - } - } -} -{% endhighlight %} - -The important part is `"reverse": true` which causes Generic Extractor to look for the lowest value of the -property specified in `idKey` (user id). Another important part --- `"increment": -1` causes ID to be lowered -by 1 between the requests. Also notice that the initial value of the API parameter `startWith` is specified -in the `jobs.params` configuration and it is overridden by the scroller in the subsequent requests. - -See [example [EX061]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/061-pagination-cursor-reverse). diff --git a/extend/generic-extractor/configuration/api/pagination/index.md b/extend/generic-extractor/configuration/api/pagination/index.md index 628fedb8..54d8e351 100644 --- a/extend/generic-extractor/configuration/api/pagination/index.md +++ b/extend/generic-extractor/configuration/api/pagination/index.md @@ -1,387 +1,5 @@ --- title: Pagination permalink: /extend/generic-extractor/configuration/api/pagination/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/ --- - -* TOC -{:toc} - -*If new to Generic Extractor, learn about [pagination in our tutorial](/extend/generic-extractor/tutorial/pagination/) first.* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various configuration options.* - -[Pagination](https://en.wikipedia.org/wiki/Pagination), or paging, describes **how an API splits a large list of items into -separate pages**. Pagination may also be called scrolling or traversing (scrolling through a large result set). Sometimes -it is also referred to as setting a [cursor](https://en.wikipedia.org/wiki/Cursor_(databases)) (pointing to a current -result). - -Almost every API has some form of pagination because returning extensive lists of large results is impractical for many -reasons, such as memory overflow issues and long transfer and processing times. So, unless you only want to do an ad-hoc query to -extract thousands of items at most, setting pagination is important. - -When configuring Generic Extractor, there is a slight distinction between pagination and scrolling: - -- **Pagination** describes paging of the entire API. -- **Scrolling** (scroller) describes paging of a single resource. - -As long as the API uses the same pagination method for all resources, there is no need to distinguish between the -two. Setting up pagination for Generic Extractor boils down to two crucial questions: - -- **How to obtain the next set of items?** (paging strategy) -- **How to determine that all items were obtained and scrolling can stop?** (stopping strategy) - -An example pagination configuration looks like this: - -{% highlight json %} -{ - ..., - "pagination": { - "method": "offset", - "limit": "2" - } -} -{% endhighlight %} - -## Paging Strategy -Generic Extractor supports the following paging strategies (scrollers); they are configured -using the `method` option: - -- [`response.url`](/extend/generic-extractor/configuration/api/pagination/response-url/) --- uses a URL provided in the response. -- [`offset`](/extend/generic-extractor/configuration/api/pagination/offset/) --- uses the page size (limit) and **item offset** (like in SQL). -- [`pagenum`](/extend/generic-extractor/configuration/api/pagination/pagenum/) --- uses the page size (limit) and **page number**. -- [`response.param`](/extend/generic-extractor/configuration/api/pagination/response-param/) --- uses a specific value (token) provided in the response. -- [`cursor`](/extend/generic-extractor/configuration/api/pagination/cursor/) --- uses the identifier of the item in response to maintain a scrolling cursor. -- [`multiple`](/extend/generic-extractor/configuration/api/pagination/multiple/) --- allows to set different scrollers for different API endpoints. - -### Choosing Paging Strategy -If the API responses contain direct links to the next set of results, use the -[`response.url` method](/extend/generic-extractor/configuration/api/pagination/response-url/). -This applies to the APIs following the [JSON API specification](https://jsonapi.org/). The response usually -contains a `links` section: - -{% highlight json %} -{ - "results": [ - ... - ], - "links": { - "next": "http://example.com/posts?page=2" - } -} -{% endhighlight %} - -If the API response contains a parameter used to obtain the next page, use the -[`response.param` method](/extend/generic-extractor/configuration/api/pagination/response-param/). -It is preferred to use an -authoritative value provided by the API than any of the following methods. -This can be some kind of scrolling token or even a page number of the next page, for example: - -{% highlight json %} -{ - "results": [ - ... - ], - "scrolling": { - "next_page": 2 - } -} -{% endhighlight %} - -If the API does not provide a scrolling hint within the response, use one of the -`offset`, `pagenum` or `cursor` methods: - -- Use the [`pagenum` method](/extend/generic-extractor/configuration/api/pagination/pagenum/) if the API expects the **page** -number/index. For example, `/users?page=2` retrieves the 2nd page regardless of how many items the page contains. -- Use the [`offset` method](/extend/generic-extractor/configuration/api/pagination/offset/) if the API expects the **item** -number/index. For example, `/users?startWith=20` retrieves the 20th and following items. -- Use the [`cursor` method](/extend/generic-extractor/configuration/api/pagination/cursor/) if the API expects an item **identifier**. -For example, `/users?startWith=20` retrieves an item with ID 20 and the following items. - -If the API uses different paging methods for different endpoints, use the -[`multiple` method](/extend/generic-extractor/configuration/api/pagination/multiple/) together with -any of the above methods. - -## Stopping Strategy -Generic Extractor stops scrolling - -- based on the `nextPageFlag` condition configuration. -- based on the `forceStop` condition configuration. -- based on the `limitStop` condition configuration. -- when the **same result** is obtained twice. - -Apart from those, each pagination method may have its own -[stopping strategy](#combining-multiple-stopping-strategies). - -The **same result** condition deals with the situation when there is no clear limit to -stop the scrolling. Generic Extractor keeps requesting higher and higher pages from the API. -Let's say that there are 150 pages of results in total. When Generic Extractor asks for page 151, various -situations can arise: - -- Most common --- API returns an **empty page**; scrolling with -[`pagenum`](/extend/generic-extractor/configuration/api/pagination/pagenum/) and -[`offset` methods](/extend/generic-extractor/configuration/api/pagination/pagenum/) will stop, and other methods will probably stop -too (depends on how empty the response is). -- Less common --- API returns an **error** --- in this case a different stopping condition such as [`nextFlag`](#next-page-flag) or -[`forceStop`](#force-stop) has to be used. -- Less common --- API keeps returning the **last page**, the extraction is stopped when a page is obtained twice (see -[example [041]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/041-paging-stop-same)). If the API returns -the last page and it is the same as the previous page, the extraction is stopped. You will see this in the Generic Extractor logs as -the following message: - - Job '1234567890' finished when last response matched the previous! - -- Even less common --- API keeps returning the **first page**, the extraction is stopped when a page is obtained twice (see -[example [EX042]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/042-paging-stop-same-2)). If the API returns -the first page, it is not same as the previous page and therefore another request is sent to `users?offset=6&limit=2`. Then the result -is the same as the previous page, the same check kicks in and the extraction is stopped too. However, the results from the first -page will be duplicated. - -### Next Page Flag -The above describes automatic behavior of Generic Extractor regarding scrolling stopping. -Using **Next Page Flag** allows you to do a **manual setup of the stopping strategy**: Generic Extractor analyzes the response, -looks for a particular field (the flag) and decides whether to continue scrolling based on the value or presence of that flag. - -Next Page Flag is configured using three options: - -- **`field`** (required) --- name of a field containing any value. The field must be in the root of the response. - It will be converted to [boolean](/extend/generic-extractor/tutorial/json/#data-values). -- **`stopOn`** (required) --- value to which the field will be compared to. When the values are equal, the scrolling stops. -- **`ifNotSet`** --- assumed value of the `field` in case it is not present in the response. It defaults to the `stopOn` value. - -The boolean conversion has the following rules: - -- `false`, `0`, `null`, string `"0"`, empty array `[]` is **`false`**. -- Everything else is **`true`**. - -Example `nextPageFlag` setting: - -{% highlight json %} -"pagination": { - "nextPageFlag": { - "field": "moreItems", - "stopOn": false, - "ifNotSet": true - }, - ... -} -{% endhighlight %} - -See our [Next Page Flag Examples](#next-page-flag-examples). - -### Force Stop -Force stop configuration allows you to stop scrolling when some extraction limits are hit. -The supported options are: - -- `pages` --- maximum number of pages to extract -- `time` --- maximum number of seconds the extraction should run -- `volume` --- maximum number of bytes which can be extracted - -This is an example or the `forceStop` setting: - -{% highlight json %} -"pagination": { - "forceStop": { - "pages": 20, - "time": 3600 - }, - ... -} -{% endhighlight %} - -The volume of the response is measured as number of bytes in compressed JSON. Therefore the response - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ] -} -{% endhighlight %} - -is compressed (minified) to: - -{% highlight json %} - {"items":[{"id":123,"name":"John Doe"},{"id":234,"name":"Jane Doe"}]} -{% endhighlight %} - -which makes it 69 bytes long. - - -The following is a **force stop example** configuration that will stop scrolling after extracting two pages of results, or -after extracting 69 bytes of minified JSON data (whichever comes first). - -{% highlight json %} -"pagination": { - "forceStop": { - "pages": 2, - "volume": 69 - }, - "method": "offset", - "limit": "2" -} -{% endhighlight %} - -See [example [EX048]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/048-force-stop) -and [example [EX116]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/116-multiple-conditions-multiple-jobs) -(combining multiple conditions) -and [example [EX140]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/140-pagination-forcestop-child-filter) -(combining with child jobs). - -### Limit Stop -Limit stop configuration allows you to stop scrolling when a specified number of items is extracted. -The supported options are: - -- `count` (required, integer) --- total number of items to extract -- `field` (required, string) --- path to the key which contains the value with total number of items - -The two options are **mutually exclusive**, but one of them is required. In both cases, the total number of items may not be -honored exactly. If the total amount is not divisible by the page size, then the leftover from the last page (if any) -is extracted too (see example [EX127](https://github.com/keboola/generic-extractor/tree/master/doc/examples/127-pagination-stop-field) and [EX138](https://github.com/keboola/generic-extractor/tree/master/doc/examples/138-pagination-stop-field-child-filter) (combining with child jobs)). -This is an example or the `limitStop` setting: - -{% highlight json %} -"pagination": { - "limitStop": { - "field": "items.count" - }, - ... -} -{% endhighlight %} - -The above configuration will search the response for the key `count` inside the key `items`. The obtained value is -expected to be the total number of items to extract. In the sample response below, it will be `4`: - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "scroller": { - "count": 4, - "offset": 0 - } -} -{% endhighlight %} - -Note that if the field does not exist in the response (e.g., you misspell it in the configuration), paging stops after the first page. -See [example [EX126]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/126-pagination-stop-limit) -(a modified version of [EX049](https://github.com/keboola/generic-extractor/tree/master/doc/examples/049-pagination-offset-rename). -For `count` configuration, see [example [EX127]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/127-pagination-stop-field) -(a modified version of [EX051](https://github.com/keboola/generic-extractor/tree/master/doc/examples/051-pagination-pagenum-basic). - -### Combining Multiple Stopping Strategies -All stopping strategies are evaluated simultaneously and for the scrolling to continue, none of -the stopping conditions must be met. In other words, the scrolling continues until any of the -stopping conditions is true. To this you need to account specific stopping strategies for -each scroller. For example, the scrolling of this configuration: - -{% highlight json %} -"pagination": { - "nextPageFlag": { - "field": "isLast", - "stopOn": true - }, - "forceStop": { - "pages": 20 - }, - "method": "offset", - "limit": "10" -} -{% endhighlight %} - -will stop if **any** of the following is true: - -- An empty page is encountered (`offset` scroller specific). -- A page contains less then 10 items (`offset` scroller specific). -- A page contains the same items as the previous page. -- 20 pages were extracted (`forceStop`). -- The `isLast` field is present in the response and is true (`nextPageFlag`). -- The `isLast` field is not present in the response. - -## Next Page Flag Examples -In this section, we want to show you the following examples of the Next Page Flag stopping strategy: - -- Has-More Scrolling -- Non-Boolean Has-More Scrolling -- Is-Last Scrolling - -### Has-More Scrolling -Assume that the API returns a response which contains a `hasMore` field. The field is present in -every response and has always the value `true` except for the last response where it is `false`. -The following pagination configuration can be used to configure the stopping strategy: - -{% highlight json %} -"pagination": { - "nextPageFlag": { - "field": "hasMore", - "stopOn": false, - "ifNotSet": false - }, - ... -} -{% endhighlight %} - -It means that the scrolling will **continue** till the field `hasMore` is present in the response and true. -In this case, setting `ifNotSet` is not necessary. - -See [example [EX045]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/045-next-page-flag-has-more) -and [example [EX139]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/139-pagination-hasmore-child-filter)] (combining with child jobs). - -### Non-Boolean Has-More Scrolling -Assume that the API returns a response which contains a `hasMore` field. The field is present only in the -last response and has the value `"no"` there. -The following pagination configuration can be used to configure the stopping strategy: - -The configuration: - -{% highlight json %} -"pagination": { - "nextPageFlag": { - "field": "hasMore", - "stopOn": true, - "ifNotSet": false - }, - ... -} -{% endhighlight %} - -means that the scrolling will **continue** until the field `hasMore` is present. This takes advantage of the -boolean conversion which converts the value `"no"` to true. If the field `hasMore` is not present, it defaults -to false. In this case setting `ifNotSet` is mandatory. - -See [example [EX046]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/046-next-page-flag-has-more-2). - -### Is-Last Scrolling -Assume that the API returns a response which contains an `isLast` field. The field is present only in the -last response and has the value `true` there. -The following pagination configuration can be used to configure the stopping strategy: - -{% highlight json %} -"pagination": { - "nextPageFlag": { - "field": "isLast", - "stopOn": true, - "ifNotSet": false - }, - ... -} -{% endhighlight %} - -The scrolling will **stop** when the field `isLast` is present in the response and true. -Because the field `isLast` is not present at all times, the `ifNotSet` configuration is required. - -See [example [EX047]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/047-next-page-flag-is-last). - diff --git a/extend/generic-extractor/configuration/api/pagination/multiple.md b/extend/generic-extractor/configuration/api/pagination/multiple.md index 3e38d23b..6aea31b4 100644 --- a/extend/generic-extractor/configuration/api/pagination/multiple.md +++ b/extend/generic-extractor/configuration/api/pagination/multiple.md @@ -1,119 +1,5 @@ --- title: Multiple Scrollers permalink: /extend/generic-extractor/configuration/api/pagination/multiple/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/multiple/ --- - -* TOC -{:toc} - -Setting the pagination method to `multiple` allows you to use **multiple scrollers on a single API**. -This type of pagination contains the definition of all scrollers used in the entire configuration. -Each [job](/extend/generic-extractor/configuration/config/jobs/) is then assigned a -[`scroller`](/extend/generic-extractor/configuration/config/jobs/#scroller) in its configuration. - -This is useful mainly if the API has inconsistent pagination methods among various API calls. -It may be also useful in case you need to vary parameters --- for instance, set different page sizes for -different endpoints. - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "multiple", - "scrollers": { - "resource_scroller": { - "method": "offset", - "limit": 100 - }, - "search_scroller": { - "method": "pagenum" - } - } - }, - ... - }, - ... -} -{% endhighlight %} - -## Configuration -The following configuration parameters are supported for the `multiple` method of pagination: - -- `scrollers` (required, object) --- object with configuration of the scrollers (see below) -- `default` (optional, string) --- name of a scroller used for all jobs without a specified scroller; if not -specified, then the jobs with no assigned scroller will not use any type of pagination. - -The `scrollers` configuration is an object whose keys are arbitrary scroller names. The values of the -keys are standard scroller configurations. Any of the supported -[paging strategies](/extend/generic-extractor/configuration/api/pagination/#paging-strategy) can be used, and -multiple paging strategies can be mixed. The configurations are the same as if there was a single scroller. -The name of the scroller must be used in a specific [job `scroller` parameter](/extend/generic-extractor/configuration/config/jobs/#scroller). - -A `default` scroller can be set (must be one of the names defined in `scrollers`). In that case, all jobs -without an assigned scroller will use the default one. - -### Stopping Condition -There are no specific stopping conditions for the `multiple` pagination. Each scroller acts upon its -normal stopping conditions. - -## Examples -Assume you have an API with several endpoints (`/users`, `/orders`, `/search`, etc.). Most endpoints -use the offset pagination strategy (meaning that the results are split into pages of the same size, and -the next page is obtained by setting offset to a multiple of the page size). The `/search` endpoint uses the -page number pagination strategy because the retrieved pages are not of equal size. The following -configuration extracts from two endpoints with different paging strategies. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "multiple", - "scrollers": { - "list_scroller": { - "method": "offset", - "limit": "2" - }, - "search_scroller": { - "method": "pagenum" - } - }, - "default": "list_scroller" - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users" - }, - { - "endpoint": "search", - "scroller": "search_scroller" - } - ] - } - } -} -{% endhighlight %} - -The `api.pagination.scrollers` defines both pagination methods: - -{% highlight json %} -"scrollers": { - "list_scroller": { - "method": "offset", - "limit": "2" - }, - "search_scroller": { - "method": "pagenum" - } -} -{% endhighlight %} - -It is then important to actually use the scroller in the `job.scroller` configuration for the endpoint `/search`. -The endpoint `/users` has no assigned scroller, therefore it uses the default one, which is `list_scroller`. - -See [example [EX062]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/062-pagination-multiple-scrollers). \ No newline at end of file diff --git a/extend/generic-extractor/configuration/api/pagination/offset.md b/extend/generic-extractor/configuration/api/pagination/offset.md index e4bd94fa..0a35f79b 100644 --- a/extend/generic-extractor/configuration/api/pagination/offset.md +++ b/extend/generic-extractor/configuration/api/pagination/offset.md @@ -1,176 +1,5 @@ --- title: Offset Scroller permalink: /extend/generic-extractor/configuration/api/pagination/offset/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/offset/ --- - -* TOC -{:toc} - -The Offset scroller handles a pagination strategy in which the API splits the results into pages -of the same size (limit parameter) and navigates through them using the **item offset** parameter. This -is similar to paging in SQL language. If you need to use the page offset, use the -[Page Number Scroller](/extend/generic-extractor/configuration/api/pagination/pagenum/). - -An example configuration: - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "offset", - "limit": 100, - "limitParam": "count", - ... - }, - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `offset` method of pagination: - -- `limit` (required, integer) --- page size -- `limitParam` (optional, string) --- name of the parameter in which the API expects the page size; the default value is `limit`. -- `offsetParam` (optional, string) --- name of the parameter in which the API expects the item offset; the default value is `offset`. -- `firstPageParams` (optional, boolean) --- when false, the first page is retrieved without the page parameters; the default value is `true`. -- `offsetFromJob` (optional, boolean) --- when true, the offset parameter value is taken from the job parameters; the default value is `false`. - -The limit value is configured by the `limit` parameter, but it may be overridden in -the [job parameters](/extend/generic-extractor/configuration/config/jobs/#request-parameters). The offset value is computed automatically starting from 0, but it may be overridden in the job parameters if `offsetFromJob` is set to `true`. - -**Important:** Do not set the limit parameter above the limit supported by the API. To give an example, if the API returned -100 items at most and you set the limit 1000, it would cause the extraction to stop after the first page. This is because the -[underflow condition](/extend/generic-extractor/configuration/api/pagination/#stopping-strategy) would be triggered. - -### Stopping Condition -Scrolling is stopped **when the result contains less items than requested** --- specified in the -`limit` configuration (underflow). This also includes an instance when no items are returned, or the -response is empty. - -Let's say that you have an API endpoint `users` which takes the parameters `limit` and `offset`. -There are four users in total. The response looks as follows: - -{% highlight json %} -[ - { - "id": 345, - "name": "Jimmy Doe" - }, - { - "id": 456, - "name": "Jenny Doe" - } -] -{% endhighlight %} - -Querying `users?offset=0&limit=2` returns the first two users. Querying `users?offset=2limit` returns -the second two users. Generic Extractor will then query `users?offset=4&limit=2`. - -If the response is empty (the API returns an empty page, `[])`, the **underflow** check kicks in -and the extraction is stopped. See [example [043]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/043-paging-stop-underflow). - -Note that the **emptiness** is evaluated on the extracted array as [auto-detected](/extend/generic-extractor/configuration/config/jobs/#data-field) or -specified by the [`dataField`](/extend/generic-extractor/configuration/config/jobs/#data-field) configuration. -That means that the entire response -may be non-empty. See [example [044]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/044-paging-stop-underflow-struct). - -You will also see the following warning in the logs: - - WARNING: dataField `results.users.items` contains no data! - -which is expected. - -All [common stopping conditions](/extend/generic-extractor/configuration/api/pagination/#stopping-strategy) apply as well. - -## Examples -This section contains three examples of API pagination using the Offset Scroller. - -### Basic Scrolling -This is the simplest scrolling setup: - -{% highlight json %} -"pagination": { - "method": "offset", - "limit": "20" -} -{% endhighlight %} - -The first request is sent with the parameters `limit=20` and `offset=0`, for example, `/users?limit=20&offset=0`. -The next request has `limit=20` and `offset=20`, for example, `/users?limit=20&offset=20`. -See [example [EX043]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/043-paging-stop-underflow) and -[example [EX044] with a more structured response](https://github.com/keboola/generic-extractor/tree/master/doc/examples/044-paging-stop-underflow-struct). - -### Renaming Parameters -The `limitParam` and `offsetParam` configuration options allow you to rename the limit and -offset for the needs of a specific API: - -{% highlight json %} -"pagination": { - "method": "offset", - "limitParam": "count", - "offsetParam": "skip", - "limit": "100" -} -{% endhighlight %} - -Here the API expects the parameters `count` and `skip`. The first request will be sent with the parameters `count=100` -and `skip=0`, for example, `/users?count=2&skip=0`. - -See [example [EX049]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/049-pagination-offset-rename). - -### Overriding Limit and Offset -It is possible to override both the limit and offset parameters of a specific API job. -This is useful in case you want to use different limits for different API endpoints. - -In the configuration below, the first API request to the `users` endpoint will be -`GET /users?count=2&skip=2`. This is because the values `count=2` and `skip=2` are taken from the -job `params`. - -Notice that the job `params` names must correspond to the names of the offset and limit parameters -(`skip` and `count` in this case). The limit parameter is always overridden to 5, no setting is necessary. -The offset parameter is overridden to 2; this requires setting `offsetFromJob`. -Without it being set, the `jobs.params.skip` value would be ignored. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "offset", - "limitParam": "count", - "offsetParam": "skip", - "offsetFromJob": true, - "limit": "20" - } - }, - "config": { - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "count": 2, - "skip": 2 - } - }, - { - "endpoint": "orders", - "dataField": "items", - "params": { - "count": 10 - } - } - ] - } - } -} -{% endhighlight %} - -The entire endpoint configuration means that the first two items of the `users` endpoint will be skipped. -For the `orders` endpoint, the `skip` (offset) parameter is not overridden, and therefore it starts at zero. -The `count` (limit) parameter is set to 10. Therefore the first request to that endpoint will be -`GET /orders?count=10&skip=0`. - -See [example [EX050]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/050-pagination-offset-override). diff --git a/extend/generic-extractor/configuration/api/pagination/pagenum.md b/extend/generic-extractor/configuration/api/pagination/pagenum.md index a2eb20f7..8d5ef50b 100644 --- a/extend/generic-extractor/configuration/api/pagination/pagenum.md +++ b/extend/generic-extractor/configuration/api/pagination/pagenum.md @@ -1,123 +1,5 @@ --- title: Page Number Scroller permalink: /extend/generic-extractor/configuration/api/pagination/pagenum/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/pagenum/ --- - -* TOC -{:toc} - -The Page Number Scroller handles a pagination strategy in which the API splits the results into pages -of the same size (limit parameter) and navigates through them using the **page offset** parameter. -If you need to use the item offset, use the [Offset Scroller](/extend/generic-extractor/configuration/api/pagination/offset/). - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "pagenum", - "limit": 100, - "limitParam": "count", - ... - }, - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `pagenum` method of pagination: - -- `limit` (optional, integer) --- page size -- `limitParam`(optional, string) --- name of the parameter in which the API expects the page size; the default value is `limit`. -- `pageParam` (optional, string) --- name of the parameter in which the API expects the page number; the default value is `page`. -- `firstPageParams` (optional, boolean) --- when `false`, the first page will be retrieved without the page parameters; the default -value is `true`. -- `firstPage` (optional, integer) --- index of the first page; the default value is `1`. - -### Stopping Condition -The `pagenum` scroller uses similar stopping condition as the [`offset` scroller](/extend/generic-extractor/configuration/api/pagination/offset/#stopping-condition). -Scrolling is stopped in case of an underflow --- when the result contains **less items than requested** (including zero). However, -in the `pagenum` scroller, the **`limit` parameter is not required** and has **no default value**. This means that if you omit it, -the scrolling will stop only if an empty page is encountered. - -## Examples -This section contains three API pagination examples where the Page Number Scroller is used. - -### Basic Scrolling -The most simple scrolling setup is the following: - -{% highlight json %} -"pagination": { - "method": "pagenum" -} -{% endhighlight %} - -The first request is sent with the parameter `page=1`, for example `/users?page=1`. -The next request will have `page=2`, for example `/users?page=2`. - -See [example [EX051]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/051-pagination-pagenum-basic). - -### Renaming Parameters -The `limitParam` and `pageParam` configuration options allow you to rename the limit and -offset for the needs of a specific API: - -{% highlight json %} -"pagination": { - "method": "pagenum", - "limit": 20, - "limitParam": "count", - "pageParam": "set" -} -{% endhighlight %} - -Here the API expects the parameters `count` and `set`. The first request will be sent with the parameters `count=20` -and `set=1`; for example, `/users?set=1&count=20`. - -**Important:** Without setting a value for the `limit` option, the `limitParam` will not be sent at all -(no matter how you name it). - -See [example [EX052]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/052-pagination-pagenum-rename). - -### Overriding Parameters -It is possible to override the limit parameter of a specific API job. -This is useful when you want to use different limits for different API endpoints. - -In the following configuration, the first request is sent to `/users?count=2` because the -`limit` parameter was renamed to `count`. Then the default value of `count` was overridden for the -`users` API endpoint in `jobs.params.count`. - -The `firstPageParams` is set to false, which means that -the page parameter (named `count`) is **not** sent in the first request. The second API -request is sent to `/users?count=2&set=1`. Because the `firstPage` option is set to `0`, the -second page index is `1`. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "pagenum", - "limit": 200, - "limitParam": "count", - "pageParam": "set", - "firstPage": 0, - "firstPageParams": false - } - }, - "config": { - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "count": 2 - } - } - ] - } - } -} -{% endhighlight %} - -See [example [EX053]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/053-pagination-pagenum-override). diff --git a/extend/generic-extractor/configuration/api/pagination/response-param.md b/extend/generic-extractor/configuration/api/pagination/response-param.md index b7524d22..87e79933 100644 --- a/extend/generic-extractor/configuration/api/pagination/response-param.md +++ b/extend/generic-extractor/configuration/api/pagination/response-param.md @@ -1,244 +1,5 @@ --- title: Response Parameter Scroller permalink: /extend/generic-extractor/configuration/api/pagination/response-param/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/response-param/ --- - -* TOC -{:toc} - -The Response Parameter Scroller can be used with APIs that provide a certain kind -of value in the response which must be used in the next request. - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "response.param", - "responseParam": "links.next", - "queryParam": "page" - }, - ... - }, - ... -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `response.param` method of pagination: - -- `responseParam` (required, string) --- path to the key which contains the value used for scrolling -- `queryParam` (required, string) --- name of the [query string](/extend/generic-extractor/tutorial/rest/#url) parameter in which -the above value should be sent to the API; the `queryParam` **overrides** the values from the [job -parameters](/extend/generic-extractor/configuration/config/jobs/#request-parameters) -(see an [example](#overriding-parameters)). -- `includeParams` (optional, boolean) --- when `true`, the job parameters -**are added** to the provided URL. The default value is `false`. -- `scrollRequest` (optional, object) --- [job-like](/extend/generic-extractor/configuration/config/jobs/) object (supported fields are -`endpoint`, `method` and `params`) which allows to sent an initial scrolling request (see an [example](#using-scroll-request)). - -### Stopping Condition -The pagination ends **when the value of `responseParam` parameters is empty** --- the key is not present at all, is null, is -an empty string, or is `false`. Take care when configuring the `responseParam` parameter. If you, for example, misspell the name of -the key, the extraction will not go beyond the first page. -[Common stopping conditions](/extend/generic-extractor/configuration/api/pagination/#stopping-strategy) also apply. - -## Examples -The following API pagination examples demonstrate the use of the Response Parameter Scroller. - -### Basic Configuration -Assume you have an API which returns, for instance, the next page number inside the response: - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "scrolling": { - "next_page": 2 - } -} -{% endhighlight %} - -The following configuration can handle such situation: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "response.param", - "responseParam": "scrolling.next_page", - "queryParam": "page" - } - }, - "config": { - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataField": "items" - } - ] - } - } -} -{% endhighlight %} - -The first request is sent to `/users`. For the second request, the value found in the response -in the property `scrolling.next_page` is sent as the `page` parameter. Therefore the request -is sent to `/users?page=2`. - -See [example [EX057]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/057-pagination-response-param-basic). - -### Overriding Parameters -The following configuration passes the parameter `orderBy` to every request: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "response.param", - "responseParam": "scrolling.next_page", - "includeParams": true, - "queryParam": "page" - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "page": "start", - "orderBy": "id" - } - } - ] - } - } -} -{% endhighlight %} - -The `includeParams` configuration set to `true` causes the parameters from the `job.params` settings to -be sent with every request. If you set `includeParams` to false, they will be sent only with -the first request. - -Also notice that the `page` parameter from `job.params` is overridden by the `page` parameter specified -in the `pagination.queryParam`. Therefore the first request is sent to `/users?page=start&orderBy=id` -and the second request to `/users?page=2&orderBy=id`. - -See [example [EX058]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/058-pagination-response-param-override). - -### Using Scroll Request -The response param scroller supports sending of an initial scrolling request. This can be used -in situations where the API requires special initialization of a scrolling endpoint; -for instance, the [Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/search-request-scroll.html). -Another example is an API which has something like a search endpoint which needs an initial request and -then allows you to scroll through the results (this is not exactly [RESTful](/extend/generic-extractor/tutorial/rest/) though). - -Let's consider an API which --- to list users --- requires that you send a POST request to the -`/search` endpoint with the configuration: - -{% highlight json %} -{ - "object": "users", - "orderBy": "id" -} -{% endhighlight %} - -It will then respond with a **search token** representing an internal cursor: - -{% highlight json %} -{ - "scroll": { - "token": "b97d814f1a715d939f3f96bc574445de", - "totalCount": 4 - } -} -{% endhighlight %} - -To obtain the actual result, send a request to the `/results` endpoint with the parameter -`scrollToken=b97d814f1a715d939f3f96bc574445de`. The response looks like this: - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "scroll": { - "token": "4015e9ce43edfb0668ddaa973ebc7e87" - } -} -{% endhighlight %} - -The following configuration is able to handle the situation: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "response.param", - "responseParam": "scroll.token", - "queryParam": "scrollToken", - "scrollRequest": { - "endpoint": "results", - "method": "GET" - } - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "search", - "method": "POST", - "dataField": "items", - "dataType": "users", - "params": { - "object": "users", - "orderBy": "id" - } - } - ] - } - } -} -{% endhighlight %} - -The configuration is actually turned upside-down. The `jobs` section defines the initial search request -(`POST` to `/search` with the required parameters `object` and `orderBy`). The first request sent to the API -is therefore: - - POST /search - - {"object":"users","orderBy":"id"} - -When the response contains a `scroll.token` field, the scroller starts to act and overrides the above -configuration with the one provided in the `scrollRequest` configuration. The next request is therefore -a `GET` to `/results?scrollToken=b97d814f1a715d939f3f96bc574445de`. The `queryParam` configuration -causes the `scrollToken` request parameter to be added. This will repeat until the `scroll.token` field in -the response is empty. - -See [example [EX059]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/059-pagination-response-param-scroll-request). diff --git a/extend/generic-extractor/configuration/api/pagination/response-url.md b/extend/generic-extractor/configuration/api/pagination/response-url.md index 2596d152..36f7dc28 100644 --- a/extend/generic-extractor/configuration/api/pagination/response-url.md +++ b/extend/generic-extractor/configuration/api/pagination/response-url.md @@ -1,205 +1,5 @@ --- title: Response URL Scroller permalink: /extend/generic-extractor/configuration/api/pagination/response-url/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/api/pagination/response-url/ --- - -* TOC -{:toc} - -The Response URL Scroller can be used with APIs that provide the URL of the -next page in the response. This scroller is suitable for APIs supporting the -[JSON API specification](https://jsonapi.org/format/#fetching-pagination). - -{% highlight json %} -{ - "api": { - "pagination": { - "method": "response.url", - "urlKey": "links.next" - }, - ... - } -} -{% endhighlight %} - -## Configuration Parameters -The following configuration parameters are supported for the `response.url` pagination method: - -- `urlKey` (optional, string) --- path in the response to the field which contains the URL of the next request; -the default value is `next_page`. -- `delimiter` (optional, string) --- char used as the delimiter of the nested keys in the `urlKey`; -the default value is `.`. -- `paramIsQuery` (optional, boolean) - - if `true` --- URL is assumed to be only [query string](/extend/generic-extractor/tutorial/rest/#url) parameters; - the parameters in the response **override** the [parameters in the job](/extend/generic-extractor/configuration/config/jobs/#request-parameters). - - if `false` --- URL with a path is assumed. `false` is the default value; the parameters in the response - **are overridden** by the parameters in the job. -- `includeParams` (optional, boolean) - - if `true` --- job parameters **are added** to the parameters of the URL provided in the response; the default value is `false`. - -See the [examples below](#examples). - -### Stopping Condition -The pagination ends **when the value of the `urlKey` parameter is empty** --- the key is not present at all, is null, -is an empty string or is `false`. Be careful when configuring the `urlKey` parameter. If you, for example, misspell the -key name, the extraction will not go beyond the first page. -[Common stopping conditions](/extend/generic-extractor/configuration/api/pagination/#stopping-strategy) also apply. - -## Examples -This section provides three API pagination examples where the Response URL Scroller is used. - -### Basic Configuration -To configure pagination for an API that supports the [JSON API specification](https://jsonapi.org/format/#fetching-pagination), -use the configuration below: - -{% highlight json %} -"pagination": { - "method": "response.url", - "urlKey": "links.next" -} -{% endhighlight %} - -The configuration expects a response to contain a `links.next` field with the URL of the next page, e.g.: - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "links": { - "next": "/users?page=2" - } -} -{% endhighlight %} - -The URL may be either an *absolute link* (`http://example.com/users?page=2`) or an *absolute path* (`/users?page=2`). -If the URL is *relative* (`users?page=2`), it is appended to the endpoint URL. - -See [example [EX054]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/054-pagination-response-url-basic). - -### Merging Parameters -To pass additional parameters to each of the page URLs, use the `includeParams` parameter: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "pagination": { - "method": "response.url", - "urlKey": "links.next", - "includeParams": true - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "account": 123 - } - } - ] - } - } -} -{% endhighlight %} - -A sample response: - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "links": { - "next": "/users?page=2" - } -} -{% endhighlight %} - -In the above configuration, the `account` parameter is sent with every API request. If it were not for the -`includeParams` option, it would be sent **only with the first request**. Note that adding -a `jobs.params.page` parameter would overwrite the `page` parameter in the response URL and thus -would probably break the paging. - -See [example [EX055]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/055-pagination-response-url-params). - -### Overriding Parameters -Sometimes the API does not pass the entire URL, but only the [query string](/extend/generic-extractor/tutorial/rest/#url) -parameters which should be used for querying the next page. - -{% highlight json %} -{ - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "links": { - "next": "?page=2" - } - } -{% endhighlight %} - -Then use the `paramsIsQuery` configuration so that your Generic Extractor can produce a -valid URL: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://mock-server:80/056-pagination-response-url-params-override/", - "pagination": { - "method": "response.url", - "urlKey": "links.next", - "paramIsQuery": true, - "includeParams": true - } - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataField": "items", - "params": { - "account": 123, - "page": "start" - } - } - ] - } - } -} -{% endhighlight %} - -Also notice that with the above -configuration the `page` parameter specified in the job is used only for the first page because it -is overridden by the `page` parameter given in the response. That is to say that the first request is sent to -`/users?account=123&page=start` and the second request is sent to `/users?account=123&page=2`. - -See [example [EX056]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/056-pagination-response-url-params-override). diff --git a/extend/generic-extractor/configuration/aws-signature/index.md b/extend/generic-extractor/configuration/aws-signature/index.md index 7a936e9c..416ef5fc 100644 --- a/extend/generic-extractor/configuration/aws-signature/index.md +++ b/extend/generic-extractor/configuration/aws-signature/index.md @@ -1,36 +1,5 @@ --- title: AWS Signature permalink: /extend/generic-extractor/configuration/aws-signature/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/aws-signature/ --- - -* TOC -{:toc} - -Generic extractor allows signaturing requests by [**AWS**](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html). -Signature is the process to add authentication information to your requests. When you use AWS tools, extractor sign your API request. - -A sample AWS signature configuration looks like this: - -{% highlight json %} -{ - ..., - "aws": { - "signature": { - "credentials": { - "accessKeyId": "testAccessKey", - "#secretKey": "testSecretKey", - "serviceName": "testService", - "regionName": "testRegion" - } - } - } -} -{% endhighlight %} - -See [example [EX143]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/143-aws-signature-request). - -## AWS Signature Credentials -- **accessKeyId** --- AWS access key ID -- **#secretKey** --- AWS secret access key -- **serviceName** --- Signing to a particular service name -- **regionName** --- Signing to a particular region name diff --git a/extend/generic-extractor/configuration/config/index.md b/extend/generic-extractor/configuration/config/index.md index 35fc1cd1..52cc7fa3 100644 --- a/extend/generic-extractor/configuration/config/index.md +++ b/extend/generic-extractor/configuration/config/index.md @@ -1,279 +1,5 @@ --- title: Extraction Configuration permalink: /extend/generic-extractor/configuration/config/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/config/ --- - -* TOC -{:toc} - -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/).* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -The `config` section of Generic Extractor configuration **describes the actual extraction**, including properties of HTTP requests, -and mapping between source JSON and target CSV. - -A sample `config` configuration can look like this: - -{% highlight json %} -{ - ..., - "config": { - "debug": false, - "outputBucket": "ge-tutorial", - "incrementalOutput": false, - "compatLevel": 2, - "jobs": [ - ... - ], - "mappings": { - ... - }, - "http": { - ... - }, - "userData": { - ... - } - } -} -{% endhighlight %} - -Apart from the properties listed below, the `config` section can contain any number of -other properties which are not used by Generic Extractor itself, but may be referenced -from within [functions](/extend/generic-extractor/functions/). - -The keys prefixed by the hash character `#` are [automatically encrypted](/overview/encryption/) when the -configuration is saved. It is advisable to store sensitive information in such fields. Note, however, they -are not automatic aliases to un-encrypted fields. That means that when using a `#password` field, you -must always refer to it as `#password` (for instance, in [functions](/extend/generic-extractor/functions)). -Also, you cannot encrypt any Generic Extractor configuration fields (such as `jobs`, `mappings`, ...). - -## Jobs -The Jobs configuration describes the API endpoints (resources) which will be extracted. This -includes configuring the HTTP method and parameters. The `jobs` configuration is -**required** and is described in a [separate article](/extend/generic-extractor/configuration/config/jobs/). - -## Output Bucket -The `outputBucket` option defines the name of the [Storage Bucket](https://help.keboola.com/storage/buckets/) -in which the extracted tables will be stored. The configuration is **required** unless -the extractor is [published](/extend/generic-extractor/publish/) as a standalone component with the -[Default Bucket](https://help.keboola.com/extend/common-interface/folders/#default-bucket) option. - -The following configuration will make Generic Extractor put all extracted tables in the `ge-tutorial` bucket -(the names of the tables are defined by the [`dataType`](/extend/generic-extractor/configuration/config/jobs/#dataType) setting): - -{% highlight json %} -{ - ..., - "config": { - "outputBucket": "ge-tutorial", - ... - } -} -{% endhighlight %} - -If you omit the `outputBucket` configuration, you will receive an error similar to this: - - CSV file 'campaigns' file name is not a valid table identifier, either set output mapping for 'campaigns' or make sure that the file name is a valid Storage table identifier. - -## Mappings -The Mappings configuration describes how the JSON response is converted into -CSV files that will be imported into Storage. The `mappings` configuration is **optional** and -is described in a [separate article](/extend/generic-extractor/configuration/config/mappings/). - -## Debug -The `debug` boolean option allows you to turn on more verbose logging which shows -all HTTP requests sent by Generic Extractor. The default value is `false`. -Read more about running Generic Extractor in a [separate article](/extend/generic-extractor/running/). - -## HTTP -The `http` option allows you to set the HTTP headers sent with every request. This primarily serves the purpose of providing values for [`api.http.requiredHeaders` option](/extend/generic-extractor/configuration/api/#required-headers). -It is also possible to use the `http` option without `api.http.requiredHeaders` in -which case it is essentially equal to [`api.http.headers`](/extend/generic-extractor/configuration/api/#default-headers). - -{% highlight json %} -{ - ..., - "config": { - "http": { - "headers": { - "X-AppKey": "ThisIsSecret" - } - }, - ... - } -} -{% endhighlight %} - -See [example [EX074]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/074-http-headers). - -## Incremental Output -The `incrementalOutput` boolean option allows you to load the extracted data into -[Storage](http://help.keboola.com/storage/) incrementally. This flag in no way affects the data extraction. -When `incrementalOutput` is set to `true`, the contents of the target table in Storage will not be cleared. -The default value is `false`. - -How to configure Generic Extractor to extract data in increments from an API -is described in a [dedicated article](/extend/generic-extractor/incremental/). - -See [example [EX075]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/075-incremental-output). - -## User Data -The `userData` option allows you to add arbitrary data to extracted records. -It is an object with arbitrary property names which are added as columns to all records extracted -from parent jobs. The property values are the columns values. It is also possible to use -[functions](/extend/generic-extractor/functions/) as `userData` property values. - -The following configuration: - -{% highlight json %} -{ - "config": { - "userData": { - "tag": "fullExtract", - "mode": "development" - } - } -} -{% endhighlight %} - -and the following response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -will produce the following `users` table: - -|id|name|tag|mode| -|123|John Doe|fullExtract|development| -|234|Jane Doe|fullExtract|development| - -The `userData` values are added to the parent jobs only. They will not affect the -[child jobs](/extend/generic-extractor/configuration/config/jobs/children). If the result table contains -columns with the same names as the `userData` properties. If there is already a column with the same name, -the `userData` column will be renamed. - -See [example [EX076]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/076-user-data). - -## Compatibility Level -As we develop the Generic Extractor, some of the new features might lead to minor differences in extraction results. -When such a situation arises, a new *compatibility level* is introduced. The `compatLevel` setting allows -you to force the old compatibility level and **temporarily** maintain the old behavior. The current -compatibility level is **3**. The `compatLevel` setting is intended only to ease updates and migrations, -never use it in new configurations (any version of old behavior is considered unsupported). - -When a new Level is introduced, the following will happen: - -- Configurations that explicitly specify `compatLevel` will stay unchanged. -- All other configurations will automatically use the latest level. - -Note that there is an exception: all configurations running before Level 3 was introduced will use compatibility -Level 1. This means that they use the legacy (Level 1) JSON parser, and you will see the following warning in -events: `Using legacy JSON parser, because it is in configuration state.` - -### Level 2 -Level 2 has different behavior in [`responseFilter`](/extend/generic-extractor/configuration/config/jobs/#response-filter) handling. -In current behavior (level 3 and above), a filtered JSON property consistently produces a valid JSON. -Previously (level 2 and below), a scalar value was not filtered. Given the data: - -{% highlight json %} -[ - { - "id": 1, - "data": { - "a": "b" - } - }, - { - "id": 2, - "data": "c" - } -] -{% endhighlight %} - -With `responseFilter` set to `data`, the level 2 version produces the following table: - -|id|data| -|---|---| -|1|{"a":"b"}| -|2|c| - -Level 3 and above produces: - -|id|data| -|---|---| -|1|{"a":"b"}| -|2|"c"| - -That means that the `data` column is always a valid JSON string. -Compare the results of examples -[EX121](https://github.com/keboola/generic-extractor/tree/master/doc/examples/121-inconsistent-object-legacy) -and -[EX122](https://github.com/keboola/generic-extractor/tree/master/doc/examples/122-multiple-filters-legacy). -using compatibility level 2 with the result produced by examples -[EX016](https://github.com/keboola/generic-extractor/tree/master/doc/examples/016-inconsistent-object) -and -[EX018](https://github.com/keboola/generic-extractor/tree/master/doc/examples/018-multiple-filters) -which use the current JSON parser. - - -### Level 1 -Level 1 uses a JSON parser which cannot handle duplicate columns properly. This applies to a number of situations: - -- The response contains properties which are evaluated to the same name, e.g.: -{% highlight json %} -{ - "some.property": "first", - "some_property": "second" -} -{% endhighlight %} -- The response contains nested properties which are evaluated to the same name, e.g.: -{% highlight json %} -{ - "some_property": "first", - "some": { - "property": "second" - } -} -{% endhighlight %} -- The response contains names which are generated internally by Generic Extractor (`parent_id`, `JSON_parentId`), e.g. -- in a child job: -{% highlight json %} -{ - "parent_id": 1, - "name": "someName" -} -{% endhighlight %} -- The user data contains a column which is present in the response. - -In either of these situations, the Level 1 extractor generates an empty column with hash, and -the original (or first encountered) column values are overwritten. In the current version -(Level 2 and above), both columns are retained. The second encountered column has a -numbered suffix. If you are upgrading from a Level 1 extractor, delete the column -with hash from the target Storage table, otherwise you'll get an error (`Some columns are missing in -the csv files`). - -There are also some differences in the naming of very long columns. For example, a property -`data.modules.#DistributionGroups.outputs.groupCharacteristics.persistent` is shortened to -`d__m__DistributionGroups_outputs_groupCharacteristics_persistent` in a Level 1 extractor, and -to `DistributionGroups_outputs_groupCharacteristics_persistent` in Level 2 and above. - -Compare the results of examples -[EX124](https://github.com/keboola/generic-extractor/tree/master/doc/examples/124-naming-conflict-legacy) -and -[EX125](https://github.com/keboola/generic-extractor/tree/master/doc/examples/125-user-data-legacy), -using compatibility level 1 with the result produced by examples -[EX025](https://github.com/keboola/generic-extractor/tree/master/doc/examples/025-naming-conflict) -and -[EX076](https://github.com/keboola/generic-extractor/tree/master/doc/examples/076-user-data) -which use the current JSON parser. \ No newline at end of file diff --git a/extend/generic-extractor/configuration/config/jobs/children.md b/extend/generic-extractor/configuration/config/jobs/children.md index d9599b8f..1381f754 100644 --- a/extend/generic-extractor/configuration/config/jobs/children.md +++ b/extend/generic-extractor/configuration/config/jobs/children.md @@ -1,1203 +1,5 @@ --- title: Child Jobs permalink: /extend/generic-extractor/configuration/config/jobs/children/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/config/jobs/children/ --- - -* TOC -{:toc} - -*If new to Generic Extractor, learn about [jobs in our tutorial](/extend/generic-extractor/tutorial/jobs/) first.* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -Child jobs allow you to **iterate/traverse over sub-resources of an API resource**. Because child jobs may contain -other child jobs, you may query for sub-sub-resources in a virtually unlimited depth. - -For instance, when downloading a list of users, you can download details of each user or a list of orders for each -user. See the Generic Extractor tutorial for a basic [example of using child -jobs](/extend/generic-extractor/tutorial/jobs/#child-jobs). - -Apart from two additional fields, `placeholders` and `recursionFilter`, configuring a child job is no different than -configuring [any other job](/extend/generic-extractor/configuration/config/jobs). - -
- Click to see a sample job configuration. - -{% highlight json %} -{ - ..., - "config": { - "jobs": [ - { - "endpoint": "users", - "method": "get", - "dataField": "items", - "dataType": "users", - "params": { - "type": "active" - }, - "responseFilter": "additional.address/details", - "responseFilterDelimiter": "/", - "children": [ - { - "endpoint": "users/{user_id}/orders", - "dataField": "items", - "recursionFilter": "id>20", - "placeholders": { - "user_id": "id" - } - } - ] - } - ] - } -} -{% endhighlight %} -
-
- -## Placeholders -In a child job, the `endpoint` configuration must contain a **placeholder** enclosed in curly braces `{}`. -For example, the following endpoint defines the placeholder **user-id**: - -{% highlight json %} -{ - ..., - "endpoint": "user/{user-id}" -} -{% endhighlight %} - -The **placeholder name** is rather arbitrary (it should not contain any special characters though). To assign it -a value, use the `placeholders` configuration. It is an object whose properties are placeholder names. The value -of each `placeholders` object property is a **property path** in the parent job response. -The placeholder in the child `endpoint` will be replaced by the **value** of that parent property. The property -path is configured relative to the extracted object ([see an example](#accessing-deeply-nested-id)). The child -`endpoint` is configured relative to the [`api.baseUrl` configuration](/extend/generic-extractor/configuration/api/#base-url), -not relative to the parent endpoint. - -The following configuration: - -{% highlight json %} -{ - ..., - "endpoint": "user/{user-id}", - "placeholders": { - "user-id": "userId" - } -} -{% endhighlight %} - -means that Generic Extractor sends as many requests to the `/user/XXX` endpoint as there -are result objects in the parent API response. The `XXX` will be replaced by the `userId` value -of each individual response. Placeholders must be used in child jobs so that each child job sends a different API request. - -{% comment %} -TODO: Un-comment this when this https://github.com/keboola/generic-extractor/issues/49 is fixed. - -The `placeholders` object can be specified as an object whose values are strings with the paths. It is -also possible to use a more complicated structure where the value is another object with `path` property. -The following configuration is equivalent to the above one: - -{% highlight json %} -{ - ..., - "endpoint": "user/{user-id}", - "placeholders": { - "user-id": { - "path": "userId" - } - } -} -{% endhighlight %} - -This is useful when using [User Defined functions](/extend/generic-extractor/user-functions/) -{% endcomment %} - -**Note:** It is technically possible to define a child job without using `placeholders` configuration -or without having a placeholder in the `endpoint`. But then all the child requests would be the same and -that is usually not what you intend to do. - -### Placeholder Level -Optionally, the placeholder name may be prefixed by a nesting **level**. Nesting allows you to -refer to properties in other objects than the direct parent. The level is written as the -placeholder name prefix, delimited by a colon `:`. For example, `2:user-id`. - -The default level is 1, meaning that the placeholder `user-id` is equivalent to `1:user-id` and -that the property path will be searched in the direct parent of the child job. The level -is counted from the child 'upwards'. Therefore a placeholder `2:user-id` means that -the property path will be searched in the parent of the child parent (two levels up). -See the [corresponding examples](#nesting-level). - -## Filter -The configuration option `recursionFilter` allows you to skip some child jobs. This can be -useful in these cases: - -- Some resources are not accessible to you and querying them would cause an error in the extraction. -- Some resources return inconsistent or incomplete responses. -- You are not interested in some of the resources and want to speed up the extraction. - -The `responseFilter` configuration contains a string expression with a filter condition composed of the following: - -- Name of a property from the parent response -- Comparison operator: `<`, `>`, `<=`, `>=`, `==` (equal), `!=` (not equal), `~~` (like), `!~` (unlike) -- Value to compare -- Logical operators: `|` (or), `&` (and); optionally, they may be used to join multiple conditions. - -An example response filter may be `type!=employee` or `product.value>150`. To test for an empty value (`null`, `false`, `""`, `0`) do not use any value -- e.g. `type!=` filter matches an empty value. - -**Important:** The expression is whitespace sensitive. Therefore `type != employee` filters the `"type "` property -to not contain the value `" employee"` (which is probably not what you intended to do). String comparisons are always -**case sensitive**. - -## Examples -This section contains a number of examples using child jobs. - -### Basic Example -Let's say that you have an API with two endpoints: - -- `/users/` --- Returns a list of users. -- `/user/?` --- Returns user details with a given user ID. - -The `users` endpoint returns a response like this: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -The `user/123` endpoint returns a response like this: - -{% highlight json %} -{ - "id": 123, - "name": "John Doe", - "address": { - "city": "London", - "country": "UK", - "street": "Whitehaven Mansions" - } -} -{% endhighlight %} - -Now use the following configuration, retrieving the user list and user -details for each user: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "placeholders": { - "user-id": "id" - } - } - ] - } - ] - } - } -} -{% endhighlight %} - -The `jobs` section defines a single job for the `users` resource. This job has child jobs for -the `users/{user-id}` resource. The `user-id` placeholder in the endpoint URL is -replaced by the value of the `id` property of each user in the parent job response. This means that -Generic Extractor makes three API calls: - -- `users` -- `users/123` -- `users/234` - -The [`dataField`](/extend/generic-extractor/configuration/config/jobs/#data-field) is set to a dot to retrieve the -entire response as a single object. Running Generic Extractor produces the following tables: - -users: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user__user-id: - -|id|name|address\_city|address\_country|address\_street|parent\_id| -|---|---|---|---|---|---| -|123|John Doe|London|UK|Whitehaven Mansions|123| -|234|Jane Doe|St Mary Mead|UK|High Street|234| - -Notice that the table representing child resources contains all the responses -merged into a single table; the [usual merging rules](/extend/generic-extractor/configuration/config/jobs/#merging-response) apply. - -Also notice that a new column --- `parent_id` was added, containing the **placeholder value** used -to retrieve the resource. The `parent_id` column is not always named `parent_id`. -Its name is created by joining the `parent_` prefix to the **placeholder path**. - -To create a friendly name for the table, it is good to use the [dataType](/extend/generic-extractor/configuration/config/jobs/#data-type) -property (see the next example). The auto-generated name is rather ugly. - -See [example [EX021]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/021-basic-child-job). - -### Basic Job With Data Type -To avoid automatic table names, it is advisable to always use the `dataType` property for -child jobs: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} -In the above configuration, `dataType` is set to `user-detail`, hence you will obtain the -following tables: - -users: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user-detail: - -|id|name|address\_city|address\_country|address\_street|parent\_id| -|---|---|---|---|---|---| -|123|John Doe|London|UK|Whitehaven Mansions|123| -|234|Jane Doe|St Mary Mead|UK|High Street|234| - -See [example [EX022]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/022-basic-child-job-datatype). - -### Basic Job With Array Values -It is also possible that the main job returns objects which contain direct references -to the children: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "children": ["a1", "a2"] - }, - { - "id": 234, - "name": "Jane Doe", - "children": ["a3"] - } -] -{% endhighlight %} - -The following configuration is the same as in the -[previous example](/extend/generic-extractor/configuration/config/jobs/children/#basic-example): - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{child-id}", - "dataField": ".", - "placeholders": { - "child-id": "children" - } - } - ] - } -] -{% endhighlight %} - -The child jobs will iterate both over the returned array of objects and -the array of each `children`. Therefore the following tables will be extracted: - -users: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user-child: - -|id|name|address_city|address_country|address_street|parent_children| -|---|---|---|---|---|---| -|a1|John Doe|London|UK|Whitehaven Mansions|a1| -|a2|Jane Doe|St Mary Mead|UK|High Street|a2| -|a3|Jimmy Doe|Scaryville|Nowhere|Cemetery Lane|a3| - -See [example [EX135]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/135-basic-child-job-array). - -### Accessing Nested ID -If the placeholder value is nested within the response object, you can use -dot notation to access child properties of the response object. For instance, if the -parent response with a list of users returns a response similar to this: - -{% highlight json %} -[ - { - "name": "John Doe", - "user-info": { - "id": 123, - "active": true - } - }, - { - "name": "Jane Doe", - "user-info": { - "id": 234, - "active": false - } - } -] -{% endhighlight %} - -you have to modify the `placeholders` definition: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "user-info.id" - } - } - ] - } -] -{% endhighlight %} - -Setting the placeholder to `"user-id": "user-info.id"` means that the `user-id` placeholder -will be replaced by the value of the `id` property inside the `user-info` object in the parent response. -If you fail to set a correct path for the placeholder, you will receive an error: - - `No value found for user-id in the parent result. (level: 1)` - -When you set the correct path, you will get the following tables: - -users: - -|name|user-info\_id|user-info\_active| -|---|---|---| -|John Doe|123|1| -|Jane Doe|234|| - -user detail: - -|id|name|address\_city|address\_country|address\_street|parent\_user-info\_id| -|---|---|---|---|---|---| -|123|John Doe|London|UK|Whitehaven Mansions|123| -|234|Jane Doe|St Mary Mead|UK|High Street|234| - -Notice that the parent reference column name is the concatenation of the `parent` prefix and -`user-info_id` placeholder path (with special characters replaced by the underscore `_`). - -See [example [EX023]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/023-child-job-nested-id). - -### Accessing Deeply Nested Id -The placeholder path is configured **relative to** the extracted object. Assume that the -parent endpoint returns a complicated response like this: - -{% highlight json %} -{ - "active-users": { - "items": [ - { - "name": "John Doe", - "user-info": { - "id": 123, - "active": true - } - }, - { - "name": "Jane Doe", - "user-info": { - "id": 234, - "active": true - } - } - ], - "description": "Active Users" - }, - "inactive-users": { - "items": [ - { - "name": "Jimmy Doe", - "user-info": { - "id": 345, - "active": false - } - } - ], - "description": "Inactive Users" - } -} -{% endhighlight %} - -The following job definition extracts the `active-users` array together with the details for each user: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "dataField": "active-users.items", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "user-info.id" - } - } - ] - } -] -{% endhighlight %} - -Notice that the placeholder path remains set to `user-info.id` because it is relative to -the parent object, which itself is located at the path `active-users.items`. This -may be confusing because the endpoint property in that child job is set relative to the -`api.baseUrl` and not to the parent URL. - -See [example [EX024]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/024-child-job-deeply-nested-id). - -### Naming Conflict -Because a new column is added to the table representing child properties, it is possible that you -run into a naming conflict. That is, if the child response with user details looks like this: - -{% highlight json %} -{ - "id": 123, - "name": "John Doe", - "parent_id": "admins", - "address": { - "city": "London", - "country": "UK", - "street": "Whitehaven Mansions" - } -} -{% endhighlight %} - -and you use the following job configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} - -the output for the child job will contain the column `parent_id`. At the same time, Generic Extractor will attempt -to create the column `parent_id` with the placeholder value, overwriting the original column. That column will be lost. - -See [example [EX025]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/025-naming-conflict). - -### Nesting Level -By default, the placeholder value is taken from the object retrieved in the parent job. As long as the child -jobs are nested only one level deep, there is no other option anyway. Let's see what happens with a deeper nesting. - -Assume that you have an API with the following endpoints: - -- `/users/` --- Returns a list of users. -- `/user/?` --- Returns user details with given user ID. -- `/user/?/orders` --- Returns a list of user orders. -- `/user/?/orders/?` --- Returns order detail with given user and order ID. - -The `users` endpoint returns a response like this: - -{% highlight json %} -[ - { - "userId": 123, - "name": "John Doe" - }, - { - "userId": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -The `user/123` endpoint returns a response like this: - -{% highlight json %} -{ - "userId": 123, - "name": "John Doe", - "description": "Good ol' father John" -} -{% endhighlight %} - -The `user/123/orders` endpoint returns a response like this: - -{% highlight json %} -[ - { - "orderId": "1234", - "price": "$12" - }, - { - "orderId": "1345", - "price": "$1212" - } -] -{% endhighlight %} - -The `user/123/order/1234` endpoint returns a response like this: - -{% highlight json %} -{ - "orderId": 1234, - "price": "$12", - "timestamp": "2017-05-06 8:21:45", - "state": "cancelled" -} -{% endhighlight %} - -Then you can create a job configuration with three nested children to retrieve all the API resources: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{1:user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "1:user-id": "userId" - }, - "children": [ - { - "endpoint": "user/{2:user-id}/orders", - "dataType": "orders", - "placeholders": { - "2:user-id": "userId" - }, - "children": [ - { - "endpoint": "user/{3:user-id}/order/{1:order-id}", - "dataType": "order-detail", - "dataField": ".", - "placeholders": { - "3:user-id": "userId", - "1:order-id": "orderId" - } - } - ] - } - ] - } - ] - } -] -{% endhighlight %} - -The `jobs` configuration retrieves all users from the `users` API endpoint. The first child retrieves -details for each user (from `user/?` endpoint) and stores them in the `user-detail` table. The second -child retrieves each user orders (from `user/?/orders` endpoint) and stores them in the `orders` table. -Finally, the deepest nested child returns details of each order (for each user) from the -`user/?/order/?` endpoint and stores them in the `order-detail` table. Therefore the following four tables -will be produced: - -users: - -|userId|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user-detail: - -|userId|name|description|parent\_userId| -|---|---|---|---| -|123|John Doe|Good ol' father John|123| -|234|Jane Doe|Good young mommy Jenny|234| - -orders: - -|orderId|price|parent\_userId| -|---|---|---| -|1234|$12|123| -|1345|$1212|123| -|2345|$42|234| - -order-detail: - -|orderId|price|timestamp|state|parent\_userId|parent\_orderId| -|---|---|---|---|---|---| -|1234|$12|2017-05-06 8:21:45|cancelled|123|1234| -|1345|$1212|2017-12-24 12:30:53|delivered|123|1345| -|2345|$42|2017-01-12 2:12:43|cancelled|234|2345| - -Notice that each table contains additional columns with the placeholder property path prefixed with `parent_`. - -See [example [EX026]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/026-basic-deeper-nesting). - -### Nesting Level Alternative -Because the required user and order IDs are present in multiple requests (in the list and in the detail), there -are multiple ways how the jobs may be configured. For example, the following configuration produces the -exact same result as the above configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "userId" - }, - "children": [ - { - "endpoint": "user/{user-id}/orders", - "dataType": "orders", - "children": [ - { - "endpoint": "user/{user-id}/order/{order-id}", - "dataType": "order-detail", - "dataField": ".", - "placeholders": { - "order-id": "orderId" - } - } - ] - } - ] - } - ] - } -] -{% endhighlight %} - -Even though the above configuration is less explicit and not really recommended, it is still acceptable. -Placeholders are defined globally, which means that the second nested child job to `user/{user-id}/orders` does -not define any because it relies on those defined by its parent job (which happen to be correct). Also the -deepest child defines only the `order-id` placeholder because, again, the `user-id` placeholder was defined in -some of its parents. - -Although the placeholders are defined globally, the ones defined in child jobs override the ones in the parent -jobs. For example, in the following (probably **very incorrect**) configuration, the `1:user-id` placeholder in -the deepest child will really contain the `orderId` value. - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{1:user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "1:user-id": "userId" - }, - "children": [ - { - "endpoint": "user/{2:user-id}/orders", - "dataType": "orders", - "placeholders": { - "2:user-id": "userId" - }, - "children": [ - { - "endpoint": "user/{1:user-id}/order/{2:order-id}", - "dataType": "order-detail", - "dataField": ".", - "placeholders": { - "1:user-id": "orderId", - "2:order-id": "userId" - } - } - ] - } - ] - } - ] - } -] -{% endhighlight %} - -See [example [EX027]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/027-basic-deeper-nesting-alternative). - -### Deep Job Nesting -Let's look at how to retrieve more nested API resources: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{1:user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "1:user-id": "id" - }, - "children": [ - { - "endpoint": "user/{2:user-id}/orders", - "dataType": "orders", - "placeholders": { - "2-user-id": "id" - }, - "children": [ - { - "endpoint": "user/{3:user-id}/order/{1:order-id}", - "dataType": "order-detail", - "dataField": ".", - "placeholders": { - "3:user-id": "id", - "1:order-id": "id" - }, - "children": [ - { - "endpoint": "user/{4:user-id}/order/{2:order-id}/items", - "dataType": "order-items", - "placeholders": { - "4:user-id": "id", - "2:order-id": "id" - }, - "children": [ - { - "endpoint": "user/{5:user-id}/order/{3:order-id}/item/{1:item-id}", - "dataType": "item-detail", - "dataField": ".", - "placeholders": { - "5:user-id": "id", - "3:order-id": "id", - "1:item-id": "id" - } - } - ] - } - ] - } - ] - } - ] - } - ] - } -] -{% endhighlight %} - -The above configuration assumes that all API resources simply have an `id` property (unlike in the -previous example, where the users had `userId` and the orders had `orderId`). This makes the configuration look -rather cryptic. Read the deepest child placeholder configuration - - "5:user-id": "id", - "3:order-id": "id", - "1:item-id": "id" - -as: - -- Go five levels up, pick the `id` property from the response and put it in place of the `user-id` in the endpoint URL. -- Go three levels up, pick the `id` property from the response and put it in place of the `order-id` in the endpoint URL. -- Go one level up, pick the `id` property from the response and put it in place of the `item-id` in the endpoint URL. - -**Important:** Once you run into using placeholders with the same property path, their order becomes important. -This is because the property path is used as the name of an additional column in the extracted table. Because -the property path is `id` in all cases, it will lead to the column `parent_id` in all cases, and therefore it -will get overwritten. With the above configuration, the following `item-detail` table will be produced: - -|id|code|name|parent_id| -|---|---|---|---| -|345|PA10|Pick Axe|345| -|456|TB20|Tooth Brush|456| - -where the `parent_id` column refers to the `1:item-id` placeholder. If you used this placeholder configuration: - -{% highlight json %} -"placeholders": { - "1:item-id": "id", - "3:order-id": "id", - "5:user-id": "id" -} -{% endhighlight %} - -you would obtain the following `item-detail` table: - -|id|code|name|parent_id| -|---|---|---|---| -|345|PA10|Pick Axe|123| -|456|TB20|Tooth Brush|123| - -where the `parent_id` column refers the `5:user-id` placeholder. - -See [example [EX028]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/028-advanced-deep-nesting). - -### Nested Array - -Suppose now that the endpoint `/users` returns a more complicated response: - -{% highlight json %} -{ - "members": { - "description": "Active System Members", - "tags": [ - "active", - "crm" - ], - "count": "2", - "items": [ - { - "name": "John Doe", - "user-info": { - "id": 123, - "active": true - } - }, - { - "name": "Jane Doe", - "user-info": { - "id": 234, - "active": false - } - } - ] - } -} -{% endhighlight %} - -The API also has an endpoint `/user/{userId}` which returns details about a specific user. If -you want to obtain all the fields from the above response and also the details about each user, -you have to create a rather tricky configuration. Even though you may be tempted to start with -the following job configuration: - -{% highlight json %} -{ - "endpoint": "users", - "dataField": "." -} -{% endhighlight %} - -this is not possible because the root of the response in the `members` field is not -an array and therefore it cannot create child jobs. For that reason the job configuration must start with: - -{% highlight json %} -{ - "endpoint": "users", - "dataField": "members.items", -} -{% endhighlight %} - -The `members.items` is an array which now can be used as a source for child jobs: - -{% highlight json %} -{ - "endpoint": "users", - "dataField": "members.items", - "dataType": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "user-info.id" - } - } - ] -} -{% endhighlight %} - -Notice that the placeholder path (`user-info.id`) is entered relative to the `dataField` setting -(`members.items`). Now, to extract the other fields from the `/users` response (other than `member.items`), -create another job: - -{% highlight json %} -{ - "endpoint": "users", - "dataField": ".", - "dataType": "users-2" -} -{% endhighlight %} - -Note that the `dataType` must be different than in the first job because the structure of the response is different. -You will receive a number of tables: - -users (first job): - -|name|user-info\_id|user-info\_active| -|---|---|---| -|John Doe|123|1| -|Jane Doe|234|| - -user-detail (first job children): - -|id|name|address\_city|address\_country|address\_street|parent\_user-info\_id| -|---|---|---|---|---|---| -|123|John Doe|London|UK|Whitehaven Mansions|123| -|234|Jane Doe|St Mary Mead|UK|High Street|234| - -users-2 (second job): - -|members\_description|members\_tags|members\_count|members\_items| -|---|---|---|---| -|Active System Members|users-2.members\_c6eb0647a7f2fb2cbe02ba62d56e3312|2|users-2.members\_c6eb0647a7f2fb2cbe02ba62d56e3312| - -users-2\_members\_items (second job, generated from array node `items`): - -|name|user-info\_id|user-info\_active|JSON\_parentId| -|---|---|---|---| -|John Doe|123|1|users-2.members_c6eb0647a7f2fb2cbe02ba62d56e3312| -|Jane Doe|234||users-2.members_c6eb0647a7f2fb2cbe02ba62d56e3312| - -The `users-2\_members\_items` contains the same results as the `users` table, but it also contains the -`JSON\_parentId` column which allows you to link the user list to the list description in the `users-2` table. -This makes the response in the `users` table quite useless, but the job is still required to generate -the child jobs to obtain the `user-detail` table. See [example [EX106]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/106-child-jobs-array). - -### Simple Filter -Let's assume that you have an API which has two endpoints: - -- `users` --- Returns a list of users. -- `users/?` --- Returns a user detail. - -The `users` endpoint returns a response like this: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "role": "parent", - "type": "admin" - }, - { - "id": 234, - "name": "Jane Doe", - "role": "parent", - "type": "administrator" - }, - { - "id": 345, - "name": "Jimmy Doe", - "role": "child", - "type": "user" - }, - { - "id": 456, - "name": "Janet Doe", - "role": "child", - "type": "user" - } -] -{% endhighlight %} - -The `user/123` endpoint returns a response like this: - -{% highlight json %} -{ - "id": 123, - "name": "John Doe", - "userRole": "parent", - "userType": "admin", - "description": "Father John" -} -{% endhighlight %} - -A simple child filter can be then set up using the following `jobs` configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "placeholders": { - "user-id": "id" - }, - "recursionFilter": "role==parent" - } - ] - } -] -{% endhighlight %} - -The `recursionFilter` setting will cause Generic Extractor to query only the sub-resources for which the -filter evaluates to true. The filter property name `type` refers to the parent response, but it -filters only the children. So, the following tables will be returned: - -users: - -|id|name|role|type| -|---|---|---|---| -|123|John Doe|parent|admin| -|234|Jane Doe|parent|administrator| -|345|Jimmy Doe|child|user| -|456|Janet Doe|child|user| - -user-detail: - -|id|name|userRole|userType|description|parent\_id| -|---|---|---|---|---|---| -|123|John Doe|parent|admin|Father John|123| -|234|Jane Doe|parent|administrator|Mother Jane|234| - -You can see from the above tables that the filter is applied to the child results only so that -the details are retrieved only for the desired users. - -See [example [EX029]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/029-simple-filter). - -### Not Like Filter -Apart from the standard comparison operators, the recursive filter allows to use -a **like** comparison operator `~`. It expects that the value contains a placeholder `%`, -which matches any number of characters. The following configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "recursionFilter": "type!~%min%", - "dataType": "user-detail", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} - -filters out all child resources not containing the string `min` in their parent type property. -The expression `%min%` matches any string which contains any number of characters (including none) -before and after the string `min`. The operator `!~` is negative like, therefore the -following `user-detail` table will be extracted: - -|id|name|userRole|userType|description|parent\_id| -|---|---|---|---|---|---| -|345|Jimmy Doe|child|user|Sonny Jimmy|345| -|456|Janet Doe|child|user|Missy Jennie|456| - -See [example [EX030]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/030-not-like-filter). - -### Combining Filters -Multiple filters can be combined using the -[logical](https://en.wikipedia.org/wiki/Boolean_algebra#Basic_operations) `&` (and) and `|` (or) operators. -For example, the following configuration retrieves details for users who have -both `id < 400` and `role = child`: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "dataType": "user-detail", - "recursionFilter": "id<400&role==child", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} - -The following `user-detail` will be produced: - -|id|name|userRole|userType|description|parent\_id| -|---|---|---|---|---|---| -|345|Jimmy Doe|child|user|Sonny Jimmy|345| - -See [example [EX031]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/031-combined-filter). - -### Multiple Filter Combinations -Although you can join a multiple filter expression with logical operators as in the -above example, there is no support for parentheses. The following configuration -combines multiple filters: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "recursionFilter": "role=parent|id>300&id<400", - "children": [ - { - "endpoint": "user/{user-id}", - "dataField": ".", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} - -The precedence of logical operators is defined so that the first operator occurring in the -expression takes precedence over the second. That is to say that the condition `role=parent|id>300&id<400` -is interpreted as `role=parent|(id>300&id<400)` because the operator `|` takes precedence -over the `&` operator. The condition `id>300&id<400|role==parent` is interpreted as -`id>300&(id<400|role==parent)` because the `&` operator takes precedence over the `|` operator. - -With the above configuration, the following `user-detail` table will be produced: - -|id|name|userRole|userType|description|parent\_id| -|---|---|---|---|---|---| -|123|John Doe|parent|admin|Father John|123| -|234|Jane Doe|parent|administrator|Mother Jane|234| -|345|Jimmy Doe|child|user|Sonny Jimmy|345| - -Because the described system of operator precedence may lead to rather unusual behaviour, -we recommend that you keep the recursive filter simple. - -See [example [EX032]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/032-multiple-combined-filter). diff --git a/extend/generic-extractor/configuration/config/jobs/index.md b/extend/generic-extractor/configuration/config/jobs/index.md index 0fbe9a5a..19607680 100644 --- a/extend/generic-extractor/configuration/config/jobs/index.md +++ b/extend/generic-extractor/configuration/config/jobs/index.md @@ -1,1538 +1,5 @@ --- title: Jobs permalink: /extend/generic-extractor/configuration/config/jobs/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/config/jobs/ --- - -* TOC -{:toc} - -*If new to Generic Extractor, learn about [jobs in our tutorial](/extend/generic-extractor/tutorial/jobs/) first.* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -The jobs section of the extraction configuration contains **descriptions of the API resources to be -extracted**. The `jobs` configuration property is an array of processed API endpoints. A -**single job represents a single [API resource](/extend/generic-extractor/tutorial/rest)**. - -
- Click to see a sample job configuration. - -{% highlight json %} -{ - ..., - "config": { - "jobs": [ - { - "endpoint": "users", - "method": "get", - "dataField": "items", - "dataType": "users", - "params": { - "type": "active" - }, - "responseFilter": "additional.address/details", - "responseFilterDelimiter": "/", - "children": [ - { - "endpoint": "users/{user_id}/orders", - "dataField": "items", - "recursionFilter": "id>20", - "placeholders": { - "user_id": "id" - } - } - ] - } - ] - } -} -{% endhighlight %} -
-
-Generic Extractor reads and processes the responses from the API endpoints in a pretty complex -way. Each response is processed in the following steps: - -1. Receive the response JSON. -2. Find the relevant object in the response as specified by the [`dataField` property](#data-field) or default rules. -3. Flatten the object structure into one or more tables. -4. Create the required tables in Storage and load data into them. - -## Merging Responses -The first two steps are the responsibility of [Jobs](/extend/generic-extractor/configuration/config/jobs/) -resulting in an array of objects. Generic Extractor then tries to find a common super-set of -properties of all objects, for example, with the following response: - -{% highlight json %} -[ - { - "id": 123, - "name": "foo", - "color": "green" - }, - { - "id": 321, - "name": "bar", - "size": "large" - } -] -{% endhighlight %} - -The super-set of object properties consists of `id`, `name`, `color` and `size`. In the Generic Extractor -configuration, this is [referred to as **`dataType`**](#data-type). If the `dataType` configuration is not set, a -name is automatically generated. Merging the object structure requires that the objects are compatible. - -The responses are merged into type-less tables. This means that values `42` and `apples` are perfectly compatible -because they get converted to a string. Also, the scalar and array values are compatible because the -scalar is [upgraded to an array](#upgrading-to-array). The following are incompatible combinations: - -- Scalar (simple) and object values -- Object and array values - -For example, this would not be allowed: - -{% highlight json %} -[ - { - "id": 123, - "name": "foo", - "color": "green" - }, - { - "id": 321, - "name": "bar", - "color": { - "items": ["red", "blue"] - } - } -] -{% endhighlight %} - -If you want to process the above response, use the -[`responseFilter` setting](/extend/generic-extractor/configuration/config/jobs/#response-filter). - -## Endpoint -The endpoint property is **required** and represents the URL of the resource. It can be either of the following: - -- URL fragment relative to the [`baseURL` property](/extend/generic-extractor/configuration/api/#baseurl) of the API definition -- Absolute URL from the domain specified in the [`baseURL` property](/extend/generic-extractor/configuration/api/#baseurl) of the API definition -- Full absolute URL - -Assume the following [API definition](/extend/generic-extractor/configuration/api/): - -{% highlight json %} -"api": { - "baseURL": "https://example.com/3.0/" -} -{% endhighlight %} - -### Relative URL Fragment -The relative endpoint **must not start** with a slash; so, with -`endpoint` set to `campaign`, the final resource URL would be -`https://example.com/3.0/campaign`. - -### Absolute Domain URL -The absolute endpoint **must start** with a slash. So, with `/endpoint` -set to `campaign`, the final resource URL would be `https://example.com/campaign`. -This means that the path part specified in the `baseURL` is ignored and fully -replaced by the value specified in `endpoint`. - -### Absolute Full URL -The full absolute URL must start with a protocol. So, with the endpoint set to -`https://eu.example.com/campaign`, this would be the final resource URL -and the path specified in the `baseURL` is completely ignored. - -### Specifying Endpoint -The following table summarizes possible outcomes: - -|`baseURL`|`endpoint`|actual URL| -|---------|----------|----------| -|`https://example.com/3.0/`|`campaign`|`https://example.com/3.0/campaign`| -|`https://example.com/3.0/`|`campaign/`|`https://example.com/3.0/campaign/`| -|`https://example.com/3.0/`|`/1.0/campaign`|`https://example.com/1.0/campaign`| -|`https://example.com/3.0/`|`https://eu.example.com/3.0/`|`https://eu.example.com/3.0/campaign`| -|`https://example.com/`|`campaign`|`https://example.com/campaign`| -|`https://example.com`|`campaign`|`https://example.comcampaign`| -|`https://example.com/`|`https://elpmaxe.com/endpoint`|`https://elpmaxe.com/endpoint`| - -It is highly recommended to use the relative URL fragments. This means that the -`baseURL` property of the `api` section **must end** with a slash. - -Use the other two options for handling exceptions in the API extraction (for instance, falling back -to an older API version). Note that using a different domain (or even a base path) may -interfere with the authentication --- depending on the specification of the target API. - -Also, closely follow the target API specification regarding trailing slashes. For some APIs, -both `https://example.com/3.0/campaign` and `https://example.com/3.0/campaign/` URLs may -be accepted and valid. For other APIs, however, only one version may be supported. - -## Request Parameters -The `params` section defines [request parameters](/extend/generic-extractor/tutorial/rest). They -may be optional or required, depending on the target API specification. The `params` section is -an object with arbitrary properties (or, more precisely, parameters understood by the target -API). It is also allowed to use [function calls](/extend/generic-extractor/functions/). - -Assume that `api.baseUrl` is set to `https://example.com/3.0/`, `jobs[].endpoint` -is set to `mock-api` and that the `param` parameters are set as follows: - -{% highlight json %} - "params": { - "startDate": "2016-01-20", - "types": ["new", "active", "finished"], - "filter": { - "query": "q=user:johnDoe", - "tags": { - "first": true, - "second": false - } - } - } -{% endhighlight %} - -See our [examples](/extend/generic-extractor/configuration/config/jobs/#examples-with-http-methods-and-parameters). - -## Method -The `method` parameter defines the [HTTP request method](/extend/generic-extractor/tutorial/rest/). -The following are the allowed values: - -- `GET` (default) -- `POST` -- `FORM` - -### GET -The HTTP method encodes the parameters in the URL. Therefore the above `params` definition gets transformed -in the following URL: - - https://example.com/3.0/mock-api?startDate=2016-01-20&types%5B0%5D=new&types%5B1%5D=active&types%5B2%5D=finished&filter%5Bquery%5D=q%3Duser%3AjohnDoe&filter%5Btags%5D%5Bfirst%5D=1&filter%5Btags%5D%5Bsecond%5D=0 - -or, in a more readable [URLDecoded](https://urldecode.org/) form: - - https://example.com/3.0/mock-api?/mock-server/web/users/12/orders/2/tickets/000/comments?startDate=2016-01-20&types[0]=new&types[1]=active&types[2]=finished&filter[query]=q=user:johnDoe&filter[tags][first]=1&filter[tags][second]=0 - -### POST -The HTTP POST method sends the parameters in the request body. They are sent as a JSON object in the same form -as entered in the configuration. For the above defined `params` property, the request body would be: - -{% highlight json %} -{ - "startDate": "2016-01-20", - "types": ["new", "active", "finished"], - "filter": { - "query": "q=user:johnDoe", - "tags": { - "first": true, - "second": false - } - } -} -{% endhighlight %} - -With the POST method, also [pagination](/extend/generic-extractor/configuration/api/pagination/) parameters are sent in the request body, and **not as URL query parameters**. If your API requires pagination parameters in the query, use the GET method or Custom Python component instead. -This method **does not support** function evaluation when `param` is a list of objects. - -Also, the `Content-Type: application/json` HTTP header is added to the request. -See our [examples](/extend/generic-extractor/configuration/config/jobs/#examples-with-http-methods-and-parameters). - -### FORM -The `FORM` method type sends the request the same way the HTTP POST method does. However, -the parameters from the `param` object are encoded as form data, mimicking the request being sent by -a web form. This method **does not support nested objects** in the `param` object. -For example, the following `params` field: - -{% highlight json %} - "params": { - "startDate": "2016-01-20", - "types": ["new", "active", "finished"] - } -{% endhighlight %} - -will be sent as the following POST request body: - - startDate=2016-01-20&types%5B0%5D=new&types%5B1%5D=active&types%5B2%5D=finished - -or, in a more readable [URLDecoded](https://urldecode.org/) form: - - startDate=2016-01-20&types[0]=new&types[1]=active&types[2]=finished - -Also, the `Content-Type: application/x-www-form-urlencoded` HTTP header will be added to the request. - -## Data Type -The `dataType` parameter assigns a name to the object(s) obtained from the endpoint. -Setting it is optional. If not set, a name will be generated automatically from the `endpoint` -value and parent jobs. - -Data types are used in [mappings](/extend/generic-extractor/configuration/config/mappings/) and for naming output -tables within their [output buckets](/extend/generic-extractor/configuration/config/#output-bucket). - -Note that you can use the same `dataType` for multiple resources, provided that the result objects may -be [merged into a single one](/extend/generic-extractor/configuration/config/mappings/). This can be used, -for example, in a situation where two API endpoints return the same resource: - -{% highlight json %} - "jobs": [ - { - "endpoint": "solved-tickets/", - "dataType": "tickets" - }, - { - "endpoint": "unsolved-tickets/", - "dataType": "tickets" - } - ] -{% endhighlight %} - -In the above case, only a single `tickets` table will be produced in the output bucket. It -will contain records from both API endpoints. - -## Data Field -The `dataField` parameter is used to determine what part of the API **response** will be -extracted. The following rules apply by default: - -- If the response is a single *array*, use the whole response. -- If the response is an [object](/extend/generic-extractor/tutorial/json/) and there is a single *array* property, -use that property. -- If the response is an object with none or multiple array properties, require that `dataField` is configured. - -Apart from cases where required, the `dataField` configuration may also be set to override the -above default behaviour. The `dataField` parameter contains a -[dot separated path](/extend/generic-extractor/tutorial/json/) to the response property you want to -extract. The `dataField` parameter may be written in two ways --- either as a simple string or -as an object with the `path` property. For instance, these two configurations are equivalent: - -{% highlight json %} - "jobs": [ - { - "endpoint": "solved-tickets/", - "dataField": "tickets" - } - ] -{% endhighlight %} - -{% highlight json %} - "jobs": [ - { - "endpoint": "solved-tickets/", - "dataField": { - "path": "tickets" - } - } - ] -{% endhighlight %} - -### Data Field Delimiter -The path to the response property is by default expected to be dot separated. That is --- a path -`members.active` refers to the property `active` nested inside the property `members`. If you need to refer to a -property containing a dot, you have to change the data field path delimiter to some other character. This can be -done using the `delimiter` property: - -{% highlight json %} - "jobs": [ - { - "endpoint": "solved-tickets/", - "dataField": { - "path": "members.active", - "delimiter": "|" - } - } - ] -{% endhighlight %} - -The above configuration refers to the property named `members.active`. To refer to the property `items` nested -inside the property `members.active` you have to use: - -{% highlight json %} - "jobs": [ - { - "endpoint": "solved-tickets/", - "dataField": { - "path": "members.active|items", - "delimiter": "|" - } - } - ] -{% endhighlight %} - -The `delimiter` character is completely arbitrary but must be something that is not used in the property names in the response. -See [example [EX120]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/120-datafield-separator). - -## Response Filter -The `responseFilter` option allows you to skip parts of the API response from processing. This can -be useful in these cases: - -- You do not want to flatten the JSON structure using the default -[JSON Parser](/extend/generic-extractor/configuration/config/jobs/#merging-responses) (as in the above examples). -- The API response is inconsistent and the objects cannot be flattened. - -The value of the `responseFilter` property is either a path to a property in the response, or -an array of such paths. The path is dot-separated unless set otherwise in the `responseFilterDelimiter` configuration. -If you want to refer to the items of an array, use `[]` --- see an [example below](#skip-flattening-in-nested-objects). -The same result can be achieved using `forceType` parameter in -[column mapping](/extend/generic-extractor/configuration/config/mappings/#column-mapping). - -## Children -The `children` configuration allows you to retrieve sub-resources of the processes API resource. -These **child jobs** (**nested jobs**) are executed for each object retrieved from the -parent response. The definition of child jobs is the same as the definition of parent jobs, -except for **placeholders**. The children configuration is described in a -[separate article](/extend/generic-extractor/configuration/config/jobs/children/). - -## Scroller - -The `scroller` parameter assigns a predefined scroller when -[`multiple` pagination](/extend/generic-extractor/configuration/api/pagination/multiple/) is used, -and is pointless when the `multiple` pagination method is not used. - -If `scroller` is not set, the pagination method specified in the [`api` configuration](/extend/generic-extractor/configuration/api/pagination/) -is used. If there is no pagination method specified, the job has no pagination. - -## Examples -The following examples show how simple objects are extracted from different objects. - -### Simple array -To extract data from the following API response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "married": true - }, - { - "id": 234, - "name": "Jane Doe", - "married": false - } -] -{% endhighlight %} - -do not set the `dataField` parameter at all, or set it to an empty string (`"dataField": ""`). -The following table will be extracted: - -|id|name|married| -|--|---|---| -|123|John Doe|1| -|234|Jane Doe|| - -Notice that the [boolean value](/extend/generic-extractor/tutorial/json/#data-values) `married` is converted -to `1` when true and left empty otherwise (`false` and `null`). - -See [example [EX001]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/001-simple-job). - -### Array within an object -To extract data from the following API response: - -{% highlight json %} -{ - "users": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ] -} -{% endhighlight %} - -do not set the `dataField` parameter at all, or set it to an empty string or to the value `users` -(`"dataField": ""` or `"dataField": "users"`). -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| -|234|Jane Doe| - -See [example [EX002]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/002-array-in-object). - -### Multiple arrays within an object -To extract data from the following API response: - -{% highlight json %} -{ - "users": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "userTypes": [ - "member", - "guest" - ] -} -{% endhighlight %} - -set the `dataField` parameter to the value `users` (`"dataField": "users"`). Not setting the -`dataField` parameter would result in an error -(`More than one array found in the response! Use the 'dataField' parameter to specify a key to the data array.`). -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| -|234|Jane Doe| - -See [example [EX003]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/003-multiple-arrays-in-object). - -### Array within a nested object -To extract data from the following API response: - -{% highlight json %} -{ - "members": { - "active": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "inactive": [ - { - "id": 345, - "name": "Jimmy Doe" - } - ] - } -} -{% endhighlight %} - -set the `dataField` parameter to the value `members.active` (`"dataField": "members.active"`). Not setting the -`dataField` parameter would result in a warning (`No data array found in the response!`). -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| -|234|Jane Doe| - -See [example [EX004]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/004-array-in-nested-object). - -### Two arrays within a nested object -To extract both `active` and `inactive` arrays from the above API response, use two jobs: - -{% highlight json %} -{ - "members": { - "active": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ], - "inactive": [ - { - "id": 345, - "name": "Jimmy Doe" - } - ] - } -} -{% endhighlight %} - -In the first job, set the `dataField` parameter to the value `members.active`. In the second job, set -the `dataField` parameter to the value `members.inactive`. The entire `jobs` section will look like this: - -{% highlight json %} - "jobs": [ - { - "endpoint": "users-5", - "dataField": "members.active" - }, - { - "endpoint": "users-5", - "dataField": "members.inactive" - } - ] -{% endhighlight %} - - -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| -|234|Jane Doe| -|345|Jimmy Doe| - -See [example [EX005]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/005-two-arrays-in-nested-object). - -### Simple object -You may encounter an API response like this: - -{% highlight json %} -{ - "id": 123, - "name": "John Doe" -} -{% endhighlight %} - -You have to set the `dataField` parameter to the value `.` (`"dataField": "."`). Not setting the -`dataField` parameter would result in a warning (`No data array found in the response!`) and no data extracted. -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| - -See [example [EX006]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/006-simple-object). - -### Nested object -You may encounter an API response like this: - -{% highlight json %} -{ - "user": { - "id": 123, - "name": "John Doe" - } -} -{% endhighlight %} - -Set the `dataField` parameter to the value `user` (`"dataField": "user"`). Not setting the -`dataField` parameter would result in a warning (`No data array found in the response!`) and no data extracted. -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| - -See [example [EX007]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/007-nested-object). - -### Single object in an array -You may encounter an API response like this: - -{% highlight json %} -{ - "member": { - "history": [ - { - "id": 123, - "name": "John Doe", - "version": 2 - }, - { - "id": 123, - "name": "Jonh Doe", - "version": 1 - } - ] - } -} -{% endhighlight %} - -To extract the first item from the `history` array, set the `dataField` parameter to the value `member.history.0`. -The following table will be extracted: - -|id|name|version| -|--|----|-------| -|123|John Doe|2 | - -See [example [EX008]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/008-single-object-in-array). - -### Nested array -You may encounter an API response like this: - -{% highlight json %} -{ - "members": [ - { - "type": "active", - "items": [ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } - ] - }, - { - "type": "inactive", - "items": [ - { - "id": 345, - "name": "Jimmy Doe" - } - ] - } - ] -} -{% endhighlight %} - -To extract the `items` from the `members` array, set the `dataField` parameter to the value `members.0.items`. -The following table will be extracted: - -|id|name| -|--|----| -|123|John Doe| -|234|Jane Doe| - -See [example [EX009]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/009-nested-array). - -## Examples with Complicated Objects -The above examples show how simple objects are extracted from different objects. Generic -Extractor can also extract objects with non-scalar properties. The default -[JSON to CSV mapping](/extend/generic-extractor/configuration/config/mappings/) flattens nested objects and produces secondary tables from nested arrays. - -### Object with nested array -You may encounter an API response like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "tags": ["active", "admin"] - }, - { - "id": 234, - "name": "Jane Doe", - "tags": ["active"] - } - ] -} -{% endhighlight %} - -To extract the `members` array, set the `dataField` parameter to the value `members` or to an empty value. -The following tables will be extracted: - -Users: - -|id|name|tags| -|---|---|----| -|123|John Doe|users-10_3ca896f39b257a4f2d2f4784e7680c87| -|234|Jane Doe|users-10_a15f4be71e739e1b2ea32bd4209d756e| - -Tags: - -|data|JSON_parentId| -|----|-------------| -|active|users-10_3ca896f39b257a4f2d2f4784e7680c87| -|admin|users-10_3ca896f39b257a4f2d2f4784e7680c87| -|active|users-10_a15f4be71e739e1b2ea32bd4209d756e| - -Each member contains a nested array of `tags` that cannot be serialized into a single -database (CSV) column. Therefore the [JSON-CSV mapper] creates another table for the -`tags` with tag values. It also generates a unique member identifier, puts it -in the `tags` column and uses it in a new `JSON_parentId` column. This -way, the 1:N relationship between Members and Tags is represented. - -See [example [EX010]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/010-object-with-nested-array). - -If the response contains an array nested in an array, Generic extractor is not able to process it. -In such case the, contents of the array are extracted as a JSON encoded string. See [example [EX130]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/130-unsupported-nested-array). -You will also see a warning in the extraction events, e.g.: - - Converting nested array 'rows.[]' to JSON string. - - -### Upgrading to array -You may encounter the following API response: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "tags": "active" - }, - { - "id": 234, - "name": "Jane Doe", - "tags": ["active", "admin"] - } - ] -} -{% endhighlight %} - -When you extract the `members` array (set the `dataField` parameter to the value `members` or to an empty value), -the following tables will be extracted: - -Users: - -|id|name|tags| -|---|---|----| -|123|John Doe|users-17_c6f3e32262682b6efd6c85ad97d2d503| -|234|Jane Doe|users-17_92df9d5b9af8821316172285b196318e| - -Tags: - -|data|JSON_parentId| -|----|-------------| -|active|users-17_c6f3e32262682b6efd6c85ad97d2d503| -|active|users-17_92df9d5b9af8821316172285b196318e| -|admin|users-17_92df9d5b9af8821316172285b196318e| - -As you can see, the scalar value `tags` in the first member object was automatically upgraded to -a single-element array because the `tags` property is an array elsewhere (second member) in the response. - -See [example [EX017]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/017-upgrading-array). - -### Object with nested object -You may encounter an API response like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "address": { - "street": "Elm Street", - "city": "New York" - } - }, - { - "id": 234, - "name": "Jane Doe", - "address": { - "street": "Bates Street", - "city": "Chicago", - "state": "USA" - } - } - ] -} -{% endhighlight %} - -To extract the `members` array, set the `dataField` parameter to the value `members` or to an empty value. -The following table will be extracted: - -|id|name|address\_street|address\_city|address_state| -|---|---|---|---|---| -|123|John Doe|Elm Street|New York|| -|234|Jane Doe|Bates Street|Chicago|USA| - -The properties of nested `address` objects are automatically flattened into the parent object. Therefore -the `address.city` property is flattened into the `address_city` column. - -See [example [EX011]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/011-object-with-nested-object). - -### Object with a deeply nested object -The above two examples show the basic principles of the JSON-CSV mapping used by Generic Extractor. -They are applied to all child properties. So, when you encounter an API response like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "contacts": [ - { - "type": "address", - "properties": { - "street": "Elm Street", - "city": "New York" - } - }, - { - "type": "email", - "primary": true, - "properties": { - "address": "john.doe@example.com" - } - } - ] - }, - { - "id": 234, - "name": "Jane Doe", - "contacts": [ - { - "type": "address", - "primary": false, - "properties": { - "street": "Bates Street", - "city": "Chicago", - "state": "USA" - } - }, - { - "type": "phone", - "primary": true, - "properties": { - "number": "123 456 789" - } - } - ] - } - ] -} -{% endhighlight %} - -the following two tables will be extracted: - -Users: - -|id|name|contacts| -|---|---|---| -|123|John Doe|users-12_8505d6585e28c00d461ba64f085d1055| -|234|Jane Doe|users-12_ec8c48efecb10334072f03a860113ea2| - -Contacts: - -|type|properties\_street|properties\_city|properties\_address|properties\_state|properties\_number|primary|JSON_parentId| -|---|---|---|---|---|---|---|---| -|address|Elm Street|New York|||||users-12_8505d6585e28c00d461ba64f085d1055| -|email|||john.doe@example.com|||1|users-12_8505d6585e28c00d461ba64f085d1055| -|address|Bates Street|Chicago||USA|||users-12_ec8c48efecb10334072f03a860113ea2| -|phone|||||123 456 789|1|users-12_ec8c48efecb10334072f03a860113ea2| - -The obtained table is rather sparse because the properties of the nested `contacts` -objects do not match exactly. For example, the `properties_number` column was created -as a result of flattening the `properties.number` object that is contained in the response -only once. Therefore the column has a single value. - -The rows in the *Contacts* table are again linked through an -auto-generated key to the parent *Users* table. Also notice that the -[Boolean value](/extend/generic-extractor/tutorial/json/#data-values) -`primary` is converted to `1` when true and left empty otherwise. - -See [example [EX012]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/012-deeply-nested-object). - -## Response Filter Examples - -### Skip flattening -If you have an API response like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "tags": ["active", "admin"] - }, - { - "id": 234, - "name": "Jane Doe", - "tags": ["active"] - } - ] -} -{% endhighlight %} - -and extract the `members` array with the -[default settings](/extend/generic-extractor/configuration/config/jobs/#an-object-with-nested-object), two tables will be -produced. If you set the response filter to `"responseFilter": "tags"`, then the `tags` property of the `members` -items will not be processed and will be stored as a [serialized](https://en.wikipedia.org/wiki/Serialization) -JSON string. The following table will be extracted: - -|id|name|tags| -|---|---|---| -|123|John Doe|["active","admin"]| -|234|Jane Doe|["active"]| - -The `tags` column contains serialized JSON fragments which can be processed by -the JSON capable database (e.g., [Snowflake](https://docs.snowflake.net/manuals/sql-reference/functions-semistructured.html)). - -See [example [EX013]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/013-skip-flatten). - -### Skip flattening in nested objects -If you have the following API response: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "contacts": [ - { - "type": "address", - "properties": { - "street": "Elm Street", - "city": "New York" - } - }, - { - "type": "email", - "primary": true, - "properties": { - "address": "john.doe@example.com" - } - } - ] - }, - { - "id": 234, - "name": "Jane Doe", - "contacts": [ - { - "type": "address", - "primary": false, - "properties": { - "street": "Bates Street", - "city": "Chicago", - "state": "USA" - } - }, - { - "type": "phone", - "primary": true, - "properties": { - "number": "123 456 789" - } - } - ] - } - ] -} -{% endhighlight %} - -and extract the `members` array with the -[default settings](/extend/generic-extractor/configuration/config/jobs/#an-object-with-a-deeply-nested-object), -two tables will be produced and the `properties` object will be flattened into a sparse table. -To avoid that, set the response filter to `"responseFilter": "contacts[].properties"`. This will -leave the `properties` child of the `contacts` array of the `members` array unprocessed. -The following two tables will be produced: - -Users: - -|id|name|contacts| -|---|---|---| -|123|John Doe|users-12_0b9650e0f68b0c6738843d5b4ff0a961| -|234|Jane Doe|users-12_cf76fb6794380244946d2bc4fa3aa04a| - -Contacts: - -|type|properties|primary|JSON_parentId| -|---|---|---|---| -|address|{"street":"Elm Street","city":"New York"}||users-12_0b9650e0f68b0c6738843d5b4ff0a961| -|email|{"address":"john.doe@example.com"}|1|users-12_0b9650e0f68b0c6738843d5b4ff0a961| -|address|{"street":"Bates Street","city":"Chicago","state":"USA"}||users-12_cf76fb6794380244946d2bc4fa3aa04a| -|phone|{"number":"123 456 789"}|1|users-12_cf76fb6794380244946d2bc4fa3aa04a| - -The `properties` column contains JSON serialized objects. When setting the `responseFilter` parameter, -remember to use the correct path to the properties you wish to skip from processing. That is to say that -setting `responseFilter` to - -- `contacts` skips the entire `contacts` property and does not create the *Contacts:* table at all. -- `properties` does nothing because there is no `properties` property under the `members` array items. -- `contacts.properties` does nothing because there is no `properties` property under the `contacts` array. - -The last two options might seem inconsistent. This is because the `responseFilter` path is set **relative to** -the objects of the processed array (not to the array itself, not to the JSON root). Thus the only correct -setting in this case is `contacts[].properties`. - -See [example [EX014]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/014-skip-flatten-nested). - -{% comment %} -TODO: Un-comment this when this is fixed: https://github.com/keboola/generic-extractor/issues/59 - -### Skip Boolean conversion -If you have an API response like this: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "married": true - }, - { - "id": 234, - "name": "Jane Doe", - "married": false - } -] -{% endhighlight %} - -and want to avoid the [default Boolean conversion](#simple-array), add the `married` property to -the response filter. Setting `"responseFilter": "married"` will cause Generic Extractor to -return the following table: - -|id|name|married| -|---|---|---| -|123|John Doe|true| -|234|Jane Doe|false| - -See [example [EX015]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/015-skip-boolean). - -{% endcomment %} - -### Inconsistent object -If you have an API response like this: - -{% highlight json %} -[ - { - "id": 123, - "name": "foo", - "color": "green" - }, - { - "id": 321, - "name": "bar", - "color": { - "items": ["red", "blue"] - } - } -] -{% endhighlight %} - -you will receive an error similar to `Error parsing response JSON: Unhandled type change from "scalar" to "object" in 'users-16.color'`. This means that the objects returned in the response are incompatible and cannot -be [merged into a table](#merging-responses) by Generic Extractor. - -To avoid the error and still retrieve the data, -use the `responseFilter` to skip the `color` property. When you set `"responseFilter": "color"`, you -will obtain the following table: - -|id|name|color| -|---|---|---| -|123|foo|"green"| -|321|bar|{"items":["red","blue"]}| - -See [example [EX016]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/016-inconsistent-object). - -### Multiple filters -You might have a complex API response like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "tags": { - "items": ["active", "admin"] - }, - "contacts": [ - { - "type": "address", - "properties": { - "street": "Elm Street", - "city": "New York" - } - }, - { - "type": "email", - "primary": true, - "properties": "john.doe@example.com" - } - ] - }, - { - "id": 234, - "name": "Jane Doe", - "tags": "none", - "contacts": [ - { - "type": "address", - "primary": false, - "properties": { - "street": "Bates Street", - "city": "Chicago", - "state": "USA" - } - }, - { - "type": "phone", - "primary": true, - "properties": "123 456 789" - } - ] - } - ] -} -{% endhighlight %} - -Because both `tags` and `contacts.properties` properties are inconsistent (sometimes using an object, -sometimes using a scalar value), you have to define multiple response filters. This can be done by using -an array of paths: - -{% highlight json %} -"responseFilter": [ - "contacts[].properties", - "tags" -] -{% endhighlight %} - -Then you will obtain the following tables: - -Users: - -|id|name|tags|contacts| -|---|---|---|---| -|123|John Doe|{"items":["active","admin"]}|users-18_19318ac6aa76a92c8d90e603f69e02f6| -|234|Jane Doe|"none"|users-18_3fdf6b12b11f85cb4eb9c34ce0322ecd| - -Contacts: - -|type|properties|primary|JSON_parentId| -|---|---|---|---| -|address|{"street":"Elm Street","city":"New York"}||users-18_19318ac6aa76a92c8d90e603f69e02f6| -|email|"john.doe@example.com"|1|users-18_19318ac6aa76a92c8d90e603f69e02f6| -|address|{"street":"Bates Street","city":"Chicago","state":"USA"}||users-18_3fdf6b12b11f85cb4eb9c34ce0322ecd| -|phone|"123 456 789"|1|users-18_3fdf6b12b11f85cb4eb9c34ce0322ecd| - -See [example [EX018]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/018-multiple-filters). - -### Setting delimiter -The default delimiter used for referencing nested properties is a dot `.`. If the names of -properties in the API response contain dots, it might be necessary to change the default delimiter. -The API response might look like this: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "primary.address": { - "street": "Elm Street", - "city": "New York" - }, - "secondary.address": { - "street": "Cemetery Ridge", - "city": "New York" - } - }, - { - "id": 234, - "name": "Jane Doe", - "primary.address": { - "street": " Blossom Avenue", - "state": "U.K." - }, - "secondary.address": { - "street": "1313 Webfoot Walk", - "city": "Duckburg", - "state": "Calisota" - } - } - ] -} -{% endhighlight %} - -If you want to filter the `secondary.address` field, you cannot set the `responseFilter` setting to -`secondary.address` because it would be interpreted as an `address` property of the `secondary` property. -If you set `"responseFilter": "secondary.address`, the extraction will work as if you did not set the -filter at all; it will be filtering the non-existent `address` property. - -For the filter to work correctly, set the `responseFilterDelimiter` to an arbitrary character not -used in the response property names. The following would be a valid configuration: - -{% highlight json %} -{ - ... - "responseFilter": "secondary.address", - "responseFilterDelimiter": "#" -} -{% endhighlight %} - -It might by tempting to change the response filter to `secondary#address`. However, this would be -incorrect as it would again mean that we are referring to an `address` property nested in the `secondary` -object. With the above settings you will obtain a table like this: - -|id|name|primary\_address\_street|primary\_address\_city|primary\_address\_state|secondary\_address| -|---|---|---|---|---|---| -|123|John Doe|Elm Street|New York||{"street":"Cemetery Ridge","city":"New York"}| -|234|Jane Doe|Blossom Avenue||U.K.|{"street":"1313 Webfoot Walk","city":"Duckburg","state":"Calisota"}| - -See [example [EX019]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/019-different-delimiter). - -### Setting delimiter --- more complex -For the custom set delimiter in the response filter, you need to have a complex API response. For example: - -{% highlight json %} -{ - "members": [ - { - "id": 123, - "name": "John Doe", - "primary.address": { - "street": "Elm Street", - "city": "New York", - "tags": [] - }, - "secondary.address": { - "street": "Cemetery Ridge", - "city": "New York", - "tags": ["work", "usaddress"] - } - }, - { - "id": 234, - "name": "Jane Doe", - "primary.address": { - "street": " Blossom Avenue", - "state": "U.K.", - "tags": ["home"] - }, - "secondary.address": { - "street": "1313 Webfoot Walk", - "city": "Duckburg", - "state": "Calisota" - } - } - ] -} -{% endhighlight %} - -To filter out all the `tags` properties, you need to set the following: - -{% highlight json %} -{ - "responseFilter": [ - "secondary.address#tags", - "primary.address#tags" - ], - "responseFilterDelimiter": "#" -} -{% endhighlight %} - -You will obtain a table similar to the one below: - -|id|name|primary\_address\_street|primary\_address\_city|primary\_address\_tags|primary\_address\_state|secondary\_address\_street|secondary\_address\_city|secondary\_address\_tags|secondary\_address\_state| -|123|John Doe|Elm Street|New York|||Cemetery Ridge|New York|["work","usaddress"]|| -|234|Jane Doe|Blossom Avenue||["home"]|U.K.|1313 Webfoot Walk|Duckburg||Calisota| - -See [example [EX020]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/020-setting-delimiter-complex). - -## Examples with HTTP Methods and Parameters - -### Request parameters -Assume that you have an API with the endpoint `users` which requires the -[GET parameter](/extend/generic-extractor/tutorial/rest/#url) `type` to specify which -users are to be retrieved. For example, a request to `/users?type=active` returns a response -with active users: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "married": true - }, - { - "id": 234, - "name": "Jane Doe", - "married": false - } -] -{% endhighlight %} - -To retrieve inactive users, send a request to `/users?type=inactive`. This -can be solved using the following jobs configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "params": { - "type": "active" - } - }, - { - "endpoint": "users", - "params": { - "type": "inactive" - } - } -] -{% endhighlight %} - -The [`params` configuration](/extend/generic-extractor/configuration/config/jobs/#request-parameters) option specifies the -parameters to be sent to the API. Therefore the `type` property is the name defined by the API itself. -The above configuration produces the following table: - -|id|name|married| -|---|---|---| -|123|John Doe|1| -|234|Jane Doe|| -|345|Jimmy Doe|| - -See [example [EX033]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/033-job-parameters) -or [example [EX136]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/136-post-request-functions) which is also using -[functions](/extend/generic-extractor/functions/). - -### POST request -You may encounter an API which is not exactly [RESTful](/extend/generic-extractor/tutorial/rest/) -and has to be queried using the [HTTP POST method](/extend/generic-extractor/tutorial/rest/#method). -Assume that you have an API with the endpoint `getUsers` that expects an empty HTTP POST request. The endpoint -then returns the following response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "married": true - }, - { - "id": 234, - "name": "Jane Doe", - "married": false - } -] -{% endhighlight %} - -Generic Extractor can handle this too, using the [`method` configuration](/extend/generic-extractor/configuration/config/jobs/#method): - -{% highlight json %} -"jobs": [ - { - "endpoint": "getUsers", - "method": "POST" - } -] -{% endhighlight %} - -The above configuration produces the following table: - -|id|name|married| -|---|---|---| -|123|John Doe|1| -|234|Jane Doe|| - -See [example [EX034]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/034-post-request). - -### Complex POST request -A not-exactly-[RESTful](/extend/generic-extractor/tutorial/rest/) API (see above) may require some JSON -parameters in the request. Let's say you have the `getUsers` endpoint which requires an HTTP POST request with -the following body: - -{% highlight json %} -{ - "filter": { - "type": "active" - }, - "return": { - "fields": ["id", "name"] - } -} -{% endhighlight %} - -The request returns the following JSON: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -The above situation can be handled by passing the entire request JSON to the -[`params` configuration](/extend/generic-extractor/configuration/config/jobs/#request-parameters). - -{% highlight json %} -"jobs": [ - { - "endpoint": "getUsers", - "method": "POST", - "params": { - "filter": { - "type": "active" - }, - "return": { - "fields": ["id", "name"] - } - } - } -] -{% endhighlight %} - -The above configuration produces the following table: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -See [example [EX035]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/035-complex-post). - -### Complex GET request -Sometimes even the HTTP GET requests require complex parameters. Suppose the API -endpoint `/users` requires the `filter` and `return` definitions. The API may describe -the configuration in many different ways, for instance: - -|filter|name|example value| -|---|---|---| -|Name of property for filtering|field|type| -|Filtering operator|operator|equal| -|Value to use in filter|value|active| - -|return|name|example value| -|---|---|---| -|Names of properties to return in response|fields|id,name| - -In the HTTP protocol, this would be encoded in the following [query string](/extend/generic-extractor/tutorial/rest/#url): - - filter[field]=type&filter[operator]=equal&filter[value]=active&return[fields][0]=id&return[fields][1]=name - -or, in the [URL Encoded](https://www.w3schools.com/tags/ref_urlencode.asp) form: - - filter%5Bfield%5D%3Dtype%26filter%5Boperator%5D%3Dequal%26filter%5Bvalue%5D%3Dactive%26return%5Bfields%5D%5B0%5D%3Did%26return%5Bfields%5D%5B1%5D%3Dname - -The following JSON is returned: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -The above situation can be handled by encoding the parameters in a JSON into the -[`params` configuration](/extend/generic-extractor/configuration/config/jobs/#request-parameters). - -{% highlight json %} -"jobs": [ - { - "endpoint": "getUsers", - "method": "POST", - "params": { - "filter": { - "field": "type", - "operator": "equal", - "value": "active" - }, - "return": { - "fields": ["id", "name"] - } - } - } -] -{% endhighlight %} - -The above configuration produces the following table: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -See [example [EX036]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/036-complex-get). diff --git a/extend/generic-extractor/configuration/config/mappings.md b/extend/generic-extractor/configuration/config/mappings.md index c7d8ce08..c5cd756b 100644 --- a/extend/generic-extractor/configuration/config/mappings.md +++ b/extend/generic-extractor/configuration/config/mappings.md @@ -1,1480 +1,5 @@ --- title: Mapping permalink: /extend/generic-extractor/configuration/config/mappings/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/config/mappings/ --- - -*If you are new to Generic Extractor, learn about [mapping in our tutorial](/extend/generic-extractor/tutorial/mapping/) first.* -*Use the [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various configuration options.* - - -* TOC -{:toc} - -Mapping allows you to **modify a response conversion process** in which Generic Extractor receives JSON responses, -[merges them](/extend/generic-extractor/configuration/config/jobs/#merging-responses), and -converts them to CSV files, which are then imported to Keboola. - -Manually define mapping if you wish to do the following: - -- Set up a primary key to simplify relations between result tables and speed up the extraction, -- Avoid extraction of unnecessary properties which make result tables cluttered, -- Split a single response into multiple result tables, -- Override the automatic conversion for any other reason. - -The automatic conversion between JSON and CSV (Storage Tables) is defined by the following rules (see an -[example](#automatic-mapping)): - -- If the value of a JSON field is a [scalar](/extend/generic-extractor/tutorial/json/#data-values), it is saved as \ -the value of the column with the name of the field. -- If the value of a JSON field is an [object](/extend/generic-extractor/tutorial/json/#data-values), each of the -object property values will be added as a value of a column with an auto-generated name. -- If the value of a JSON field is an [array](/extend/generic-extractor/tutorial/json/#data-values), a new table -will be created and linked by the `JSON_parentId` column. - -Mapping configuration allows you to manually modify or override this behavior for a -[`dataType`](/extend/generic-extractor/configuration/config/jobs/#data-type) -defined in a job. The following is a mapping configuration example: - -{% highlight json %} -"mappings": { - "users": { - "address.country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } -} -{% endhighlight %} - -## Configuration -The `mappings` configuration is a deeply nested object. The first level of keys are `dataType` -values used in the [job configurations](/extend/generic-extractor/configuration/config/jobs/#data-type). The -second level of keys are the names of the properties found (or expected) in the response. -Then, the value is an object with the following properties: - -- `type` (optional, string) --- Mapping type, either `column`, `table` or `user`. The default value is `column`. -- `mapping` (required, object) --- Mapping configuration; depends on the mapping type. - -The following configuration shows a sample mapping configuration for dataType `users` and column `id`: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "user_id" - } - } - } -} -{% endhighlight %} - -### User Interface -In the UI, the mapping can be created for each endpoint in the `Endpoints`.`Mapping section` by clicking `Create Mapping` toggle. - -{: .image-popup} -![Create mapping](/extend/generic-extractor/tutorial/create_mapping_toggle.png) - -#### Mapping Detection - -You may opt to generate the mapping automatically by clicking the `Infer Mapping` button in the top right corner. - -This operation will generate a mapping based on the enpoint's sample response, which may help as a starting point for further manual adjustments. - -In most cases, this method is sufficient and doesn't require any additional edits. - -{: .image-popup} -![Create mapping](/extend/generic-extractor/tutorial/create_mapping.png) - - -##### Primary key -You can specify a `.` separated path of the elements in the response to create a primary key. **NOTE** that if you are mapping child jobs, -the parent keys will automatically be included. - -##### Nesting level -Currently, the automatic detection outputs only single table mapping. You can control the nesting level by specifying -the `Nesting Level` property. For example, a depth of 1 transforms `{"address": {"street": "Main", "details": {"postcode": "170 00"}}}` into two columns: `address_street` and `address_details`. -All elements that have ambiguous types or are beyond the specified depth are stored in a single column as JSON, e.g., with the [`force_type`](https://developers.keboola.com/extend/generic-extractor/configuration/config/mappings/#mapping-without-processing) option. - -### Column Mapping -Column mapping represents a basic mapping type that allows you to select extracted -columns, rename them, and optionally set a primary key on them. The mapping -configuration requires: - -- `type` (optional, string) --- Can be omitted or must be `column`. -- `mapping` (required, object) --- Object with two properties: - - `destination` (required, string) --- Name of the column in the output table - - `primaryKey` (optional, boolean) --- If `true`, then a primary key will be set on the column. The default value is `false`. -- `forceType` (optional, boolean) --- If set to `true`, the property will not be processed and will be stored as an encoded -JSON (see an [example](#mapping-without-processing)). - -### User Mapping -User mapping has the same configuration as the [column mapping](#column-mapping). The only difference is -that it applies to *virtual properties*. This is useful mainly for working with auto-generated properties/columns -in child jobs (see an [example](#mapping-child-jobs)). - -### Table Mapping -Table mapping allows you to create a new table from a particular property of the response object. Table -mapping is, by default, used for arrays. The mapping configuration requires: - -- `type` (required, string) --- Must be set to `table`. -- `destination` (required, string) --- Name of the output table. -- `tableMapping` (required, object) --- Object with another mapping configuration (required unless `parentKey.disable` is -set to `true` --- see below). -- `parentKey` (optional, object) --- Configuration of the parent-child relationship between tables: - - `destination` (optional, string) --- Name of the column which links to the parent table. The default value is the name - of the parent table with the suffix `_pkey`. See an [example](#using-primary-keys). - - `primaryKey` (optional, boolean) --- Set to `true` to mark the link column as a primary key for the child table too. - The default value is `false`. See an [example](#using-primary-keys). - - `disable` (optional, boolean) --- Completely disables the parent-child relationship, disables configured - `tableMapping`. See an [example](#disabled-parent-key). - -The following configuration takes the `contacts` property from the response and makes a new table -(`user-contact`) from it; the `contacts.email` is mapped to the `email` column and the property -`contacts.phone` is mapped to the column `tel`. See more in the [examples](#table-mapping-examples). - -{% highlight json %} -"contacts": { - "type": "table", - "destination": "user-contact", - "tableMapping": { - "email": { - "type": "column", - "mapping": { - "destination": "email" - } - }, - "phone": { - "type": "column", - "mapping": { - "destination": "tel" - } - } - } -} -{% endhighlight %} - -## Examples -The following examples demonstrate how to map JSON responses to CSV files. - -### Automatic Mapping -Without any configuration, the following JSON response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "address": { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - "interests": [ - "girls", "cars", "flowers" - ] - }, - { - "id": 234, - "name": "Jane Doe", - "address": { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - }, - "interests": [ - "boys", "cars", "flowers" - ] - } -] -{% endhighlight %} - -is converted to the following CSV files (and subsequently Storage tables): - -users: - -|id|name|address\_street|address\_country|address\_city|interests| -|---|---|---|---|---|---| -|123|John Doe|Blossom Avenue|United Kingdom||users_dab021748b7f93c10476ebe151de4459| -|234|Jane Doe|Whiteheaven Mansions|United Kingdom|London|users_aeb1d126471eef24c0769437f4e7adaa| - -users_interests: - -|data|JSON_parentId| -|---|---| -|girls|users_dab021748b7f93c10476ebe151de4459| -|cars|users_dab021748b7f93c10476ebe151de4459| -|flowers|users_dab021748b7f93c10476ebe151de4459| -|boys|users_aeb1d126471eef24c0769437f4e7adaa| -|cars|users_aeb1d126471eef24c0769437f4e7adaa| -|flowers|users_aeb1d126471eef24c0769437f4e7adaa| - -The nested properties `address.street`, `address.county` and `address_city` were automatically -flattened into columns named as a concatenation of the parent and child property names. The -array property `interests` was turned into a separate table and linked using -`JSON_parentId` column and auto-generated keys. - -See [example [EX063]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/063-mapping-automatic). - -**Note:** When using automatic mapping, you may get result tables with **changing structure**. A typical example is -when the API returns a completely empty response in which case no tables are created for the job. -When Manual mapping is used, the generated table structure always honors the mapping setting. -See [example [EX137]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/137-mapping-tables-nested-empty). - -### Basic Manual Mapping -Maybe you are not interested in the user `interests` and want to simplify the user table -to three columns: `country`, `name` and `id`. The following mapping configuration does the trick: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ], - "mappings": { - "users": { - "address.country": { - "type": "column", - "mapping": { - "destination": "country" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "id": { - "mapping": { - "destination": "id", - "primaryKey": true - } - } - } - } - } - } -} -{% endhighlight %} - -The `mappings` settings has the key `users` which is the value of the `job.dataType` property. The keys in -the `users` objects are the names of the properties in the JSON response. The values are the mapping configurations for -each property. The mapping is always exhaustive; only the mentioned properties get processed, while the others are -completely ignored. The above configuration also sets the primary key on the `id` column. - -All three mapped properties are mapped to columns (the `id` property relies on the default value for `type`). -Notice that in the nested properties, you need to enter the name/path in the JSON response (`address.country`). -You cannot use the auto-generated name produced without any mapping (`address_country`), because the automatic -processing is turned off by the mapping. - -Take great care to **use the correct keys** in the mapping! If you misspell the first-level key, the entire configuration -will be ignored (it will refer to a non-existent data type). If you misspell the second-level key, you will get -an empty column in the result (referring to a non-existent property of the response). With the -correct settings, the following table will be produced: - -|country|name|id| -|---|---|---| -|United Kingdom|John Doe|123| -|United Kingdom|Jane Doe|234| - -See [example [EX064]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/064-mapping-basic). - -### Mapping Child Jobs -Let's say that you have an API endpoint `/users` which returns a response similar to: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -More details about the user can be retrieved through another endpoint --- `/user/{id}`, where `{id}` is -the user ID: - -{% highlight json %} -{ - "id": 123, - "name": "John Doe", - "address": { - "city": "London", - "country": "UK", - "street": "Whitehaven Mansions" - }, - "interests": [ - "girls", "cars", "flowers" - ] -} -{% endhighlight %} - -To handle this situation in Generic Extractor, use a [child job](/extend/generic-extractor/configuration/config/jobs/#children): - -{% highlight json %} -"jobs": [ - { - "endpoint": "users", - "dataType": "users", - "children": [ - { - "endpoint": "user/{user-id}", - "dataType": "user-detail", - "dataField": ".", - "placeholders": { - "user-id": "id" - } - } - ] - } -] -{% endhighlight %} - -The produced user-detail table will look like this: - -|id|name|address\_city|address\_country|address\_street|interests|parent_id| -|---|---|---|---|---|---|---| -|123|John Doe|London|UK|Whitehaven Mansions|user-detail_3484bd6e10690a3a2e77079f69ceaa42|123| -|234|Jane Doe|St Mary Mead|UK|High Street|user-detail_a7655e39a0399dc842b44365778cd295|234| - -Note that the name of the column `parent_id` depends on the [placeholder configuration](/extend/generic-extractor/configuration/config/jobs/children/#basic-example) -and is not always `parent_id` (see [example](/extend/generic-extractor/configuration/config/jobs/children/#basic-job-with-array-values)). - -Now you can use the following mapping to shape the table: - -{% highlight json %} -"mapping": { - "user-detail": { - "address.country": { - "type": "column", - "mapping": { - "destination": "country" - } - }, - "parent_id": { - "type": "user", - "mapping": { - "destination": "user_id" - } - } - } -} -{% endhighlight %} - -and get the following user-detail table: - -|country|user_id| -|---|---| -|UK|123| -|UK|234| - -The important part of the mapping configuration is that you **must** use `"type": "user"` -for the mapping type of the `parent_id` (`user_id`) column. This is because the -column `parent_id` does not really exist in the response as it is generated dynamically for the child job. - -See [example [EX065]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/065-mapping-child-jobs). - -### Mapping without Processing -The `forceType` configuration property allows you to skip a part of the API response from processing. -With the following API response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "address": { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - "interests": [ - "girls", "cars", "flowers" - ] - }, - { - "id": 234, - "name": "Jane Doe", - "address": { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - }, - "interests": [ - "boys", "cars", "flowers" - ] - } -] -{% endhighlight %} - -and the following mapping configuration: - -{% highlight json %} -"mappings": { - "users": { - "name": { - "mapping": { - "destination": "name" - } - }, - "id": { - "type": "column", - "mapping": { - "destination": "id", - "primaryKey": true - } - }, - "interests": { - "type": "column", - "mapping": { - "destination": "interests" - }, - "forceType": true - } - } -} -{% endhighlight %} - -the result table `users` contains the `interests` field unprocessed and left as JSON fragments: - -|name|id|interests| -|---|---|---| -|John Doe|123|["girls","cars","flowers"]| -|Jane Doe|234|["boys","cars","flowers"]| - -The same result can be achieved by using the [`responseFilter` job property](/extend/generic-extractor/configuration/config/jobs/#response-filter): - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "responseFilter": "interests" - } - ] - } - } -} -{% endhighlight %} - -See [example [EX073]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/073-mapping-forceType). - -### Table Mapping Examples - -#### Basic table mapping -Because all output columns must be listed in a mapping, using only column mapping settings skips -the `interests` property of the response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "address": { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - "interests": [ - "girls", "cars", "flowers" - ] - }, - { - "id": 234, - "name": "Jane Doe", - "address": { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - }, - "interests": [ - "boys", "cars", "flowers" - ] - } -] -{% endhighlight %} - -The `interests` property cannot be saved as a column therefore, a mapping of the `table` type must be used: - -{% highlight json %} -"mappings": { - "users": { - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "interests": { - "type": "table", - "destination": "user-interests", - "tableMapping": { - ".": { - "type": "column", - "mapping": { - "destination": "interest" - } - } - } - } - } -} -{% endhighlight %} - -The table mapping follows the same structure as normal mapping. Each item is another mapping -definition identified by the property name in the JSON file. Because the `interests` property -itself is an array, its value has no name, and therefore, the key is only a dot `"."`. The mapping -value is a standard [column mapping](/extend/generic-extractor/configuration/config/mappings/#column-mapping). -The above configuration produces the same result as automatic column mapping. - -See [example [EX066]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/066-mapping-tables-basic). - -#### Nested properties -Let's say that you have an API that returns a response like this (it will be used in the following two examples as well): - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "contacts": { - "email": "john.doe@example.com", - "phone": "987345765", - "addresses": [ - { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - }, - { - "id": 234, - "name": "Jane Doe", - "contacts": { - "email": "jane.doe@example.com", - "skype": "jane.doe", - "addresses": [ - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - } -] -{% endhighlight %} - -With the automatic mapping (without any `mappings` configuration), the following tables will be extracted: - -users: - -|id|name|contacts\_email|contacts\_phone|contacts\_addresses|contacts\_skype| -|---|---|---|---|---|---|---| -|123|John Doe|john.doe@example.com|987345765|users.contacts_912c86dec7acdb9d8a17c97eb464aec6|| -|234|Jane Doe|jane.doe@example.com||users.contacts_4cf9e859113127acb138872cc630e75f|jane.doe| - -users.contacts: - -|street|country|city|JSON_parentId| -|---|---|---|---| -|Blossom Avenue|United Kingdom||users.contacts_912c86dec7acdb9d8a17c97eb464aec6| -|Whiteheaven Mansions|United Kingdom|London|users.contacts_912c86dec7acdb9d8a17c97eb464aec6| -|Whiteheaven Mansions|United Kingdom|London|users.contacts_4cf9e859113127acb138872cc630e75f| - -This might not be exactly what you want. Perhaps you would like the contacts to be separate from the users and -addresses. This can be done using the following mapping configuration: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "contacts": { - "type": "table", - "destination": "user-contact", - "tableMapping": { - "email": { - "type": "column", - "mapping": { - "destination": "email" - } - }, - "phone": { - "type": "column", - "mapping": { - "destination": "tel" - } - }, - "addresses": { - "type": "table", - "destination": "user-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - } - } - } - } -} -{% endhighlight %} - -The above configuration defines that the `contacts` field will be mapped into a separate table -with the columns `email` and `tel` (value of `mapping.destination`). The `address` field will be -mapped into yet another separate table with the columns `street` and `country`. - -With the above configuration, the following tables will be created: - -users: - -|id|name|user-contact| -|---|---|---| -|123|John Doe|b5d72095c441b3a3d6f23ad8142c3f8b| -|234|Jane Doe|5f7f2ab65a680f1a9387a8fafe6b9050| - -user-contact: - -|email|tel|user-address|users_pk| -|---|---|---|---| -|john.doe@example.com|987345765|1c439a9a39548290f7b7a4513a9224e7|b5d72095c441b3a3d6f23ad8142c3f8b| -|jane.doe@example.com||605e865710f95dba665f6d0e8bc19f1a|5f7f2ab65a680f1a9387a8fafe6b9050| - -user-address: - -|street|country|user-contact_pk| -|---|---|---| -|Blossom Avenue|United Kingdom|1c439a9a39548290f7b7a4513a9224e7| -|Whiteheaven Mansions|United Kingdom|1c439a9a39548290f7b7a4513a9224e7| -|Whiteheaven Mansions|United Kingdom|605e865710f95dba665f6d0e8bc19f1a| - -See [example [EX067]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/067-mapping-tables-nested). - -#### Array items -The following examples deal with *arrays of objects*. If you need to deal with *array of scalar* values, see -the [corresponding example](#multiple-primary-key-columns). - -Consider the same API response as above: -
- Click to expand the response. - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "contacts": { - "email": "john.doe@example.com", - "phone": "987345765", - "addresses": [ - { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - }, - { - "id": 234, - "name": "Jane Doe", - "contacts": { - "email": "jane.doe@example.com", - "skype": "jane.doe", - "addresses": [ - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - } -] -{% endhighlight %} -
-
-Let's say that you know that the `addresses` array contains only two items at most, and therefore, -you want to mark them as the primary and secondary addresses: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "contacts": { - "type": "table", - "destination": "user-contact", - "tableMapping": { - "email": { - "type": "column", - "mapping": { - "destination": "email" - } - }, - "phone": { - "type": "column", - "mapping": { - "destination": "tel" - } - }, - "addresses.0": { - "type": "table", - "destination": "primary-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - }, - "addresses.1": { - "type": "table", - "destination": "secondary-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - } - } - } - } -} -{% endhighlight %} - - -The important part of the pretty long configuration is: - -{% highlight json %} -"addresses.0": { - "type": "table", - "destination": "primary-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } -} -{% endhighlight %} - -This picks the first item (remember that array indexes are -[zero-based](/extend/generic-extractor/tutorial/json/#references)) and places it in the -`primary-address` table. Analogously, the `addresses.1` mapping picks the second item from the `addresses` -array and stores it in the `secondary-address` table. - -See [example [EX068]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/068-mapping-tables-nested-array). - -#### Directly mapping array -The following examples deal with *arrays of objects*; if you need to deal with *array of scalar* values, see -the [corresponding example](#multiple-primary-key-columns). - -Consider the same API response as above: -
- Click to expand the response. - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "contacts": { - "email": "john.doe@example.com", - "phone": "987345765", - "addresses": [ - { - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - }, - { - "id": 234, - "name": "Jane Doe", - "contacts": { - "email": "jane.doe@example.com", - "skype": "jane.doe", - "addresses": [ - { - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } - } -] -{% endhighlight %} -
-
- -If you map the table as in the [previous example](#array-items), you will receive a `primary-address` table: - -|street|country|user-contact_pk| -|---|---|---| -|Blossom Avenue|United Kingdom|1c439a9a39548290f7b7a4513a9224e7| -|Whiteheaven Mansions|United Kingdom|605e865710f95dba665f6d0e8bc19f1a| - -Notice that the records link to the `user-contact` table. This may produce unnecessarily complicated -links between the tables because, from the response, it is obvious that each address is assigned to -a specific user. To avoid this, you can directly map a nested property: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "contacts": { - "type": "table", - "destination": "user-contact", - "tableMapping": { - "email": { - "type": "column", - "mapping": { - "destination": "email" - } - }, - "phone": { - "type": "column", - "mapping": { - "destination": "tel" - } - } - } - }, - "contacts.addresses.0": { - "type": "table", - "destination": "primary-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - } - } -} -{% endhighlight %} - -The mapping for the `primary-address` table is now **not nested** inside the mapping for the -`contacts` table. Therefore, it links directly to the `users` table. The content is the same because -the mapping still refers to the same property --- the first item of the `addresses` property of `contacts` -(`contacts.addresses.0`). The following table is produced: - -|street|country|users_pk| -|---|---|---| -|Blossom Avenue|United Kingdom|b5d72095c441b3a3d6f23ad8142c3f8b| -|Whiteheaven Mansions|United Kingdom|5f7f2ab65a680f1a9387a8fafe6b9050| - -The user table now contains an additional column --- `primary-address`: - -|id|name|user-contact|primary-address| -|---|---|---|---| -|123|John Doe|b5d72095c441b3a3d6f23ad8142c3f8b|b5d72095c441b3a3d6f23ad8142c3f8b| -|234|Jane Doe|5f7f2ab65a680f1a9387a8fafe6b9050|5f7f2ab65a680f1a9387a8fafe6b9050| - -See [example [EX069]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/069-mapping-tables-nested-direct). - -#### Using primary keys -In the above example, you can see that the `primary-address` table contains -an auto-generated key to link back to users. This is unnecessary because you can safely link to -the user ID. To do this, you only need to specify the primary key for the table: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id", - "primaryKey": true - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "contacts": { - "type": "table", - "destination": "user-contact", - "parentKey": { - "primaryKey": true, - "destination": "userId" - }, - "tableMapping": { - "email": { - "type": "column", - "mapping": { - "destination": "email" - } - }, - "phone": { - "type": "column", - "mapping": { - "destination": "phone" - } - } - } - }, - "contacts.addresses.0": { - "type": "table", - "destination": "primary-address", - "tableMapping": { - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - } - } -} -{% endhighlight %} - -The most important part in the above configuration is the `"primaryKey": true` setting for -the `id` column in the `users` table. Thanks to this, Generic Extractor is able to automatically link -all related records to this ID. In the `user-contact` and `primary-address` tables, the column -`users_pk` will be created, which will contain the user ID. The name is auto-generated as the -name of the parent table with the suffix `_pk`. - -To override this auto-generated name, the following configuration is used in the `user-contact` -table, renaming the `users_pk` column to `userId`. - -{% highlight json %} -"parentKey": { - "primaryKey": true, - "destination": "userId" -}, -{% endhighlight %} - -It also marks the `userId` column in the `user-contact` table as the primary key. The following tables -are produced by the above mapping configuration: - -users: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user-contact: - -|email|phone|userId| -|---|---|---| -|john.doe@example.com|987345765|123| -|jane.doe@example.com||234| - -primary-address: - -|street|country|users_pk| -|---|---|---| -|Blossom Avenue|United Kingdom|123| -|Whiteheaven Mansions|United Kingdom|234| - -See [example [EX070]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/070-mapping-tables-nested-direct-pk). - -#### Multiple primary key columns -Generic Extractor allows you to set only a single (primary) key for a table. This means that -if you set `primaryKey` on multiple columns, you will create a compound primary key. Let's say -that you have an API with the following response: - -{% highlight json %} -[ - { - "firstName": "John", - "lastName": "Doe", - "interests": [ - "girls", "cars", "flowers" - ] - }, - { - "firstName": "John", - "lastName": "Doe", - "interests": [ - "boys", "cars", "flowers" - ] - } -] -{% endhighlight %} - -Notice that the response does not contain a single unique property (id). You can create the -following configuration: - -{% highlight json %} -"mappings": { - "users": { - "firstName": { - "mapping": { - "destination": "first_name", - "primaryKey": true - } - }, - "lastName": { - "mapping": { - "destination": "last_name", - "primaryKey": true - } - }, - "interests": { - "type": "table", - "destination": "interests", - "tableMapping": { - ".": { - "type": "column", - "mapping": { - "destination": "interest" - } - } - } - } - } -} -{% endhighlight %} - -to extract the following tables: - -users: - -|first\_name|last\_name| -|---|---| -|John|Doe| -|Jane|Doe| - -interests: - -|interest|users\_pk| -|---|---| -|girls|John,Doe| -|cars|John,Doe| -|flowers|John,Doe| -|boys|Jane,Doe| -|cars|Jane,Doe| -|flowers|Jane,Doe| - -**Important:** If you set a column (or combination of columns) as a primary key that has duplicate -values, the rows will not be imported! - -See [example [EX071]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/071-mapping-multiple-pk). - -#### Multiple primary keys from nested columns -The [above example](#multiple-primary-key-columns) shows how to set a compound primary key. -It is also possible to create a compound key using a [parent column](#using-primary-keys). -Let's say that you have an API with the following response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "addresses": [ - { - "index": 1, - "street": "Blossom Avenue", - "country": "United Kingdom" - }, - { - "index": 2, - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - }, - { - "id": 234, - "name": "Jane Doe", - "addresses": [ - { - "index": 1, - "street": "Whiteheaven Mansions", - "city": "London", - "country": "United Kingdom" - } - ] - } -] -{% endhighlight %} - -Notice that the `addresses` response does not contain a single unique property, but there is an `index` -property which is unique within a specific user. The primary key for an address would, therefore, be the -combination of `id` and `index`. - -Create the following configuration: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "mapping": { - "destination": "id", - "primaryKey": true - } - }, - "name": { - "mapping": { - "destination": "name" - } - }, - "addresses": { - "type": "table", - "parentKey": { - "destination": "userId", - "primaryKey": true - }, - "destination": "user-address", - "tableMapping": { - "index": { - "type": "column", - "mapping": { - "destination": "index", - "primaryKey": true - } - }, - "street": { - "type": "column", - "mapping": { - "destination": "street" - } - }, - "country": { - "type": "column", - "mapping": { - "destination": "country" - } - } - } - } - } -} -{% endhighlight %} - -to extract the following tables: - -users: - -|id|name| -|---|---| -|123|John Doe| -|234|Jane Doe| - -user-address: - -|index|street|country|userId| -|1|Blossom Avenue|United Kingdom|123| -|2|Whiteheaven Mansions|United Kingdom|123| -|1|Whiteheaven Mansions|United Kingdom|234| - -When imported to Storage, the primary key for the `user-address` table will be set to -the combination of `index` and `userId`. The configuration has three important parts. - -The first part: - -{% highlight json %} -"id": { - "mapping": { - "destination": "id", - "primaryKey": true - } -} -{% endhighlight %} - -sets the `id` property from a user as the primary key for the resulting table. - -The second part: - -{% highlight json %} -"parentKey": { - "destination": "userId", - "primaryKey": true -} -{% endhighlight %} - -adds the primary key from users (i.e., the `id` property) to the child table `user-address` as a `userId` column. -It also sets it as the primary key for the `user-address` table. - -The third part: - -{% highlight json %} -"index": { - "type": "column", - "mapping": { - "destination": "index", - "primaryKey": true - } -} -{% endhighlight %} - -adds the `index` column from the `user-address` table to the list of the primary key columns in that table. - -**Important:** If you set a column (or a combination of columns) as a primary key that has duplicate -values, the rows will not be imported! - -See [example [EX115]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/115-multiple-pk-parent). - -#### Disabled parent key -It is also possible to entirely disable the relationships between parts of the response objects. -Consider, for example, this API response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe", - "children": [ - { - "id": 1234, - "name": "Jenny Doe", - "favoriteColors": "blue,pink" - }, - { - "id": 1235, - "name": "Jimmy Doe", - "favoriteColors": "red,green,blue" - } - ] - }, - { - "id": 234, - "name": "Jane Doe", - "children": [ - { - "id": 2345, - "name": "Janet Doe", - "favoriteColors": "black" - } - ] - } -] -{% endhighlight %} - -You may extract (by default) the `children` as a separate entity related to their parents. Another -option is to extract the `children` as an entity equal to their parents. This can be done by -disabling the relationship: - -{% highlight json %} -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "favoriteColors": { - "type": "column", - "mapping": { - "destination": "colors" - } - }, - "children": { - "type": "table", - "destination": "users", - "parentKey": { - "disable": true - } - } - } -} -{% endhighlight %} - -The important part is `parentKey.disable` set to `true` in the `children` mapping. Then, an already -existing mapping can be referenced --- `"destination": "users"` defines that the children are to be mapped using -the same configuration as their parents. - -Notice that the `children` mapping contains no `tableMapping` configuration. This is because the mapping of -the `users` data type is used both for users and their children. Setting `tableMapping` for `children` would have -no effect. This also means that the `favoriteColors` column configuration **must be** defined in the `users` -mapping (even though it is not used by the users in the API response). - -See [example [EX072]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/072-mapping-pk-disable). - -#### User data in mapping - -There are situations when you need to add custom columns to the output data. For this purpose, the -[`userData` functionality](/extend/generic-extractor/configuration/config/#user-data) can be used. - -Consider this API response: - -{% highlight json %} -[ - { - "id": 123, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -Let's say you want to add a `country` column to output data, but you want to use custom mapping. To -handle this situation, you have to define mapping also for the `userData`. - -{% highlight json %} -"userData": { - "country": "UK" -}, -"mappings": { - "users": { - "id": { - "type": "column", - "mapping": { - "destination": "id" - } - }, - "name": { - "type": "column", - "mapping": { - "destination": "name" - } - }, - "country": { - "type": "user", - "mapping": { - "destination": "country" - } - } - } -} -{% endhighlight %} - -The produced user table will look like this: - -|id|name|country| -|---|---|---| -|123|John Doe|UK| -|234|Jane Doe|UK| - -See [example [EX134]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/134-user-data-in-mapping). diff --git a/extend/generic-extractor/configuration/configuration.md b/extend/generic-extractor/configuration/configuration.md index 350f5fff..f95ff1d0 100644 --- a/extend/generic-extractor/configuration/configuration.md +++ b/extend/generic-extractor/configuration/configuration.md @@ -1,101 +1,5 @@ --- title: Generic Extractor Configuration permalink: /extend/generic-extractor/configuration/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/ --- - -* TOC -{:toc} - -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/).* - -To get an overall idea of what to expect when configuring Generic Extractor, look at the following **overview** of various configuration sections. - -Then review a [sample configuration](#configuration-map) featuring all configuration options and their -nesting. The **configuration map** is also available as a [separate article](/extend/generic-extractor/map/). - - -### User Interface - -{% include branches-beta-warning.html %} - -Recently, we created a convenient user interface that allows you to build a configuration for the Generic Extractor without writing JSON code. -You can set up and test the connection in a few clicks, just like you are used to in some other popular API development tools. - -Features such as cURL import, request tests, output mapping generator, or dynamic function templates and evaluation make the configuration process as easy as ever. - -You can switch between the JSON representation and the user interface in the upper right corner of the configuration editor. - -{: .image-popup} -![UI Switch](/extend/generic-extractor/configuration/ui_switch.png) - -#### Backward compatibility - -The new user interface is mostly backward compatible with the old JSON configuration. However, some features are not yet supported in the new UI. -In such cases, you will be notified in the UI what sections are not supported. - -***NOTE:** The new UI does not affect the functionality of old configurations. All configurations will continue to work. -However, in some cases, you might need to perform some manual adjustments in order to make the UI compatible.* - - -### JSON Configuration Sections -*Click on the section names if you want to learn more.* - -- **parameters** - - [**api**](/extend/generic-extractor/configuration/api/) --- sets the basic properties of the API. - - [**baseUrl**](/extend/generic-extractor/configuration/api/#base-url) --- defines the URL to which the - API requests should be sent. - - [**caCertificate**](/extend/generic-extractor/configuration/api/#ca-certificate) --- defines custom certificate authority bundle in `crt`/`pem` format. - - [**#clientCertificate**](/extend/generic-extractor/configuration/api/#client-certificate) --- defines client certificate and private key in `crt`/`pem` format. - - [**pagination**](/extend/generic-extractor/configuration/api/pagination/) --- breaks a result with many items into separate pages. - - [**authentication**](/extend/generic-extractor/configuration/api/authentication/) --- needs to be - configured for any API which is not public. - - [**retryConfig**](/extend/generic-extractor/configuration/api/#retry-configuration) --- automatically - and repeatedly, retries failed HTTP requests. - - [**http**](/extend/generic-extractor/configuration/api/#default-http-options) --- sets the timeouts, default - headers, and parameters sent with each API call. - - **aws** - - [**signature**](/extend/generic-extractor/configuration/aws-signature) --- defines AWS credentials for signature request - - [**config**](/extend/generic-extractor/configuration/config/) --- describes the actual extraction. - - [**debug**](/extend/generic-extractor/running/#debug-mode) --- shows all HTTP requests sent by - Generic Extractor. - - [**outputBucket**](/extend/generic-extractor/configuration/config/#output-bucket) --- defines the name - of a Storage Bucket in which the extracted tables will be stored. - - [**http**](/extend/generic-extractor/configuration/config/#http) --- sets the HTTP headers sent with - every request. - - [**jobs**](/extend/generic-extractor/configuration/config/jobs/) --- describes the API endpoints - (resources) to be extracted. - - [**mappings**](/extend/generic-extractor/configuration/config/#mappings) --- describes how the JSON - response is converted into CSV files that will be imported into Storage. - - [**incrementalOutput**](/extend/generic-extractor/incremental/) --- loads the extracted data into - Storage incrementally. - - [**userData**](/extend/generic-extractor/configuration/config/#user-data) --- adds arbitrary data to - extracted records. - - [**sshProxy**](/extend/generic-extractor/configuration/ssh-proxy/) --- securely access HTTP(s) endpoints inside your private Network. - - [**iterations**](/extend/generic-extractor/configuration/iterations/) --- executes a configuration multiple times, each time - with different values. -- [**authorization**](/extend/generic-extractor/configuration/api/authentication/#oauth) --- allows injecting OAuth authentication. - -There are also simple pre-defined [**functions**](/extend/generic-extractor/functions/) available, adding extra -flexibility when needed. - -Generic Extractor can be run from within the [**Keboola user interface**](/extend/generic-extractor/running/) (only -configuration [JSON](/extend/generic-extractor/tutorial/json/) is needed), or [**locally**](/extend/generic-extractor/running/#running-locally) -(Docker is needed). - -### Configuration Map -The following sample configuration shows various configuration options and their nesting. -You can use the map to navigate between them. The parameter map is also available -[separately](/extend/generic-extractor/map/), and we recommend pinning it to your toolbar for quick reference. - -{% highlight json %} -{% include config-map.json %} -{% endhighlight %} - - - diff --git a/extend/generic-extractor/configuration/iterations.md b/extend/generic-extractor/configuration/iterations.md index 532f09a5..91df4f13 100644 --- a/extend/generic-extractor/configuration/iterations.md +++ b/extend/generic-extractor/configuration/iterations.md @@ -1,261 +1,7 @@ ---- -title: Iterations -permalink: /extend/generic-extractor/configuration/iterations/ -redirect_from: - - /extend/generic-extractor/iterations/ ---- - -* TOC -{:toc} - -The `iterations` section allows you to **execute a configuration multiple times, each time with different -values**. The most typical use for `iterations` is extraction of the same data from multiple accounts. -Iterations can always be replaced by creating multiple complete configurations of Generic Extractor. - -Iterations are specified as an array of objects, where each object contains the same properties -as the [`config`](/extend/generic-extractor/configuration/config/) section. All properties of the object are optional. - -Consider the following example of an `iterations` configuration defining that the entire Generic Configuration -will be executed twice: the first time with the username `JohnDoe`, and the second time with the username -`DoeJohn`. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "basic" - } - }, - "config": { - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "users" - } - ] - }, - "iterations": [ - { - "username": "JohnDoe", - "#password": "TopSecret" - }, - { - "username": "DoeJohn", - "#password": "EvenMoreSecret" - } - ] - } -} -{% endhighlight %} - -Since **all `iterations` properties override the `config` properties**, they are accessible -as [configuration attributes](/extend/generic-extractor/functions/#configuration-attributes) -via the `attr` property. - -Keep in mind that `iterations` can refer directly only to the things specified in the `config` section. -For the `api` section, you must use [functions](/extend/generic-extractor/functions/). -Also, it is not possible to iterate over values returned in the response. -The number of iterations and their values must be defined in the configuration. - -## Configuration -Because the values defined in `iterations` override those in the `config` section, -everything that can be in the `config` section is allowed as well -(including arbitrary user attributes used in [functions](/extend/generic-extractor/functions/)). -Using `jobs` and `mappings` in iterations does not make much sense though. - -Also, if you use `userData` in iterations, they must result in the same columns; otherwise the resulting -table cannot be imported into Storage. If you use `incrementalOutput`, only the last value of `incrementalOutput` -is honoured. - -## Examples - -### Iterating Parameters -Suppose, you have an API which takes a URL parameter `account_id`, which restricts the returned data to a -certain account. The following configuration executes the entire configuration for two accounts --- `345` and `456`: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "outputBucket": "ge-tutorial", - "userData": { - "account": { - "attr": "accountId" - } - }, - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "account_id": { - "attr": "accountId" - } - } - } - ] - }, - "iterations": [ - { - "accountId": 345 - }, - { - "accountId": 456 - } - ] - } -} -{% endhighlight %} - -Since the `iterations` section overrides the values in the `config` section, the below configuration -yields the exact same results as the configuration above: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "outputBucket": "ge-tutorial", - "accountId": 123, - "userData": { - "account": { - "attr": "accountId" - } - }, - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "account_id": { - "attr": "accountId" - } - } - } - ] - }, - "iterations": [ - { - "accountId": 345 - }, - { - "accountId": 456 - } - ] - } -} -{% endhighlight %} - -It looks as if the first execution is with `account_id=123`, but it is not the case. The configuration -will be executed only twice: the first time with `account_id=345` and the second time with `account_id=456`. -See [example [EX112]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/112-iterations-params). - -### Iterating Headers -Suppose you have an API from which you want to extract data from two accounts (`JohnDoe` and `DoeJohn`). The -API uses the [HTTP Basic Authentication](/extend/generic-extractor/configuration/api/authentication/basic/) method, and in addition, -each user has their own API token, which must be provided in the `X-Api-Token` header. - -Even if the above parameters relate to the [`api` configuration](/extend/generic-extractor/configuration/api/), which cannot -be directly included in `iterations`, we can specify them as `http.headers` and therefore it is still possible to use them. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "basic" - } - }, - "config": { - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - }, - "iterations": [ - { - "http": { - "headers": { - "X-Api-Token": "1234abcd" - } - }, - "username": "JohnDoe", - "#password": "TopSecret" - }, - { - "http": { - "headers": { - "X-Api-Token": "zyxv9876" - } - }, - "username": "DoeJohn", - "#password": "EvenMoreSecret" - } - ] - } -} -{% endhighlight %} - -Next to `username` and `#password` from the `config` section, the above configuration overrides also -the `http.headers.X-Api-Token` setting. The configuration can be simplified by using -[functions and references](/extend/generic-extractor/functions/): - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "basic" - } - }, - "config": { - "http": { - "headers": { - "X-Api-Token": { - "attr": "apiToken" - } - } - }, - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - }, - "iterations": [ - { - "apiToken": "1234abcd", - "username": "JohnDoe", - "#password": "TopSecret" - }, - { - "apiToken": "zyxv9876", - "username": "DoeJohn", - "#password": "EvenMoreSecret" - } - ] - } -} -{% endhighlight %} - -Here, the `config` section specifies the part of the token authentication which is common to both iterations. -Each iteration then specifies only the token. By writing `"attr": "apiToken"` under `X-Api-Token` we say that -`X-Api-Token` will have the value of the `apiToken` property from the `config` section. That property is in -turn specified in the iteration objects. - - -See [example [EX113]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/113-iterations-headers). +--- +title: Iterations +permalink: /extend/generic-extractor/configuration/iterations/ +redirect_from: + - /extend/generic-extractor/iterations/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/iterations/ +--- diff --git a/extend/generic-extractor/configuration/ssh-proxy/index.md b/extend/generic-extractor/configuration/ssh-proxy/index.md index a0596d11..c16402c5 100644 --- a/extend/generic-extractor/configuration/ssh-proxy/index.md +++ b/extend/generic-extractor/configuration/ssh-proxy/index.md @@ -1,95 +1,5 @@ --- title: SSH Proxy Configuration permalink: /extend/generic-extractor/configuration/ssh-proxy/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/configuration/ssh-proxy/ --- - -* TOC -{:toc} - -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/).* -*Use [Parameter Map](/extend/generic-extractor/map/) to help you navigate among various -configuration options.* - -An SSH proxy for Generic Extractor allows you tu securely access HTTP(s) endpoints inside your private network. -It creates an SSH tunnel, and all traffic from Generic Extractor is forwarded through the tunnel to the destination server. - -A sample `config` configuration can look like this: - -{% highlight json %} -{ - ..., - "sshProxy": { - "host": "proxy.example.com", - "user": "proxy", - "port": 22, - "#privateKey": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----" - } -} -{% endhighlight %} - - -## Usage -Before using an SSH proxy, set up an **SSH proxy server** -to act as a gateway to your private network where your destination server resides. - -Complete the following steps to set up an SSH proxy for Generic Extractor: - -### 1. Set Up SSH Proxy Server -Here is a very basic [Dockerfile](https://docs.docker.com/engine/reference/builder/) example. -All it does is run an sshd daemon and expose port 22. You can, of course, set this up in your system in -a similar way without using Docker. - -{% highlight dockerfile %} -FROM ubuntu:14.04 - -RUN apt-get update - -RUN apt-get install -y openssh-server -RUN mkdir /var/run/sshd - -RUN echo 'root:root' |chpasswd - -RUN sed -ri 's/^PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config -RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config - -EXPOSE 22 - -CMD ["/usr/sbin/sshd", "-D"] -{% endhighlight %} - -This server should be in the same private network where your destination server resides. It should be accessible publicly from the internet via SSH. -The default port for SSH is 22, but you can choose a different port. - -We highly recommend to allow access only from the [Keboola IP address ranges](https://help.keboola.com/extractors/ip-addresses/). - -See the following pages for more information about setting up SSH on your server: - -- [OpenSSH configuration](https://help.ubuntu.com/community/SSH/OpenSSH/Configuring) -- [Dockerized SSH service](https://docs.docker.com/engine/examples/running_ssh_service/) - - -### 2. Generate SSH Key Pair -Generate an SSH key pair and copy the public key to your **SSH proxy server**. -Paste it to the **public.key** file, and then append it to the authorized_keys file. - -{% highlight bash %} -mkdir ~/.ssh -cat public.key >> ~/.ssh/authorized_keys -{% endhighlight %} - -### 3. Configure Generic Extractor SSH Proxy - -{% highlight json %} -{ - ..., - "sshProxy": { - "host": "your-ssh-proxy-host", - "user": "ssh-proxy-user", - "port": 22, - "#privateKey": "your-generated-private-key" - } -} -{% endhighlight %} - -See [example [EX131]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/131-ssh-tunnel). -and [example [EX133]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/133-ssh-tunnel-iterations-params). diff --git a/extend/generic-extractor/configuration/ui_switch.png b/extend/generic-extractor/configuration/ui_switch.png deleted file mode 100644 index 83152c49..00000000 Binary files a/extend/generic-extractor/configuration/ui_switch.png and /dev/null differ diff --git a/extend/generic-extractor/events.png b/extend/generic-extractor/events.png deleted file mode 100644 index 3609158c..00000000 Binary files a/extend/generic-extractor/events.png and /dev/null differ diff --git a/extend/generic-extractor/function_eval.gif b/extend/generic-extractor/function_eval.gif deleted file mode 100644 index 4513c09c..00000000 Binary files a/extend/generic-extractor/function_eval.gif and /dev/null differ diff --git a/extend/generic-extractor/functions.md b/extend/generic-extractor/functions.md index 7ee2b762..7ec31f56 100644 --- a/extend/generic-extractor/functions.md +++ b/extend/generic-extractor/functions.md @@ -1,1511 +1,5 @@ --- title: Functions permalink: /extend/generic-extractor/functions/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/functions/ --- - -* TOC -{:toc} - -Functions are simple pre-defined functions that - -- allow you to add extra flexibility when needed. -- can be used in several places in the Generic Extractor configuration to introduce dynamically generated values instead of -those provided statically. -- allow referencing the existing values in the configuration instead of copying them. -- are advantageous and sometimes necessary when [publishing your configuration as a new component](/extend/generic-extractor/publish/). - -## Configuration -A function is used instead of a simple value in specific parts of the Generic Extractor configuration (see [below](#function-contexts)). -A function configuration is an object with the properties `function` (one of the [available function names](#supported-functions) and `args` -(function arguments), for example: - -{% highlight json %} -{ - "function": "concat", - "args": [ - "John", - "Doe" - ] -} -{% endhighlight %} - -The argument of a function can be any of the following: - -- [Scalar](/extend/generic-extractor/tutorial/json/#data-values) (simple) value (as in the above example) -- Reference to a value from [function context (see below)](#function-contexts) -- Another function object - -Additionally, the function may be replaced by a plain reference to the function context. This means you can write (where permitted) -a configuration value in three possible ways: - -**A simple value:** - -{% highlight json %} -{ - ..., - "baseUrl": "http://example.com/ -} -{% endhighlight %} - -**A function call:** - -{% highlight json %} -{ - ..., - "baseUrl": { - "function": "concat", - "args": [ - "http://", - "example.com" - ] - } -} -{% endhighlight %} - -**A reference to a value from the function context:** -{% highlight json %} -{ - ..., - "baseUrl": { - "attr": "someUrl" - } -} -{% endhighlight %} - -These forms can be combined freely. They can be also nested in a virtually unlimited way. For instance: - -{% highlight json %} -{ - ..., - "baseUrl": { - "function": "concat", - "args": [ - "https://", - { - "attr": "domain" - } - ] - } -} -{% endhighlight %} - -### User Interface -You may create functions in the user interface's `User Parameters` or `User Data` sections. -You can also create the functions directly from other configuration contexts, e.g., when defining the query parameters on the endpoint. - -Aside from predefined functions, the UI also offers the most common templates that you can use. - -{: .image-popup} -![img.png](/extend/generic-extractor/functions.png) - -The UI also offers a convenient way to evaluate the function and see the results. - -{: .image-popup} -![img.png](/extend/generic-extractor/function_eval.gif) - - -## Supported Functions - -### md5 -The [`md5` function](https://www.php.net/manual/en/function.md5.php) calculates the [MD5 hash](https://en.wikipedia.org/wiki/MD5) of a -string. The function takes one argument, which is the string to hash. - -{% highlight json %} -{ - "function": "md5", - "args": [ - "NotSoSecret" - ] -} -{% endhighlight %} - -The above will produce `1228d3ff5089f27721f1e0403ad86e73`. - -See an [example](#job-parameters). - -### sha1 -The [`sha1` function](https://www.php.net/manual/en/function.sha1.php) calculates the [SHA-1 hash](https://en.wikipedia.org/wiki/SHA-1) of a -string. The function takes one argument which is the string to hash. - -{% highlight json %} -{ - "function": "sha1", - "args": [ - "NotSoSecret" - ] -} -{% endhighlight %} - -The above will produce `64d5d2977cc2573afbd187ff5e71d1529fd7f6d8`. - -See an [example](#job-parameters). - -### base64_encode -The [`base64_encode` function](https://www.php.net/manual/en/function.base64-encode.php) converts a -string to the [MIME Base64 encoding](https://en.wikipedia.org/wiki/Base64#MIME). The function -takes one argument which is the string to encode. - -{% highlight json %} -{ - "function": "base64_encode", - "args": [ - "TeaPot" - ] -} -{% endhighlight %} - -The above will produce `VGVhUG90`. - -See an [example](#nested-functions). - -### hash_hmac -The [`hash_hmac` function](https://www.php.net/manual/en/function.hash-hmac.php) creates -an [HMAC (Hash-based message authentication code)](https://en.wikipedia.org/wiki/Hash-based_message_authentication_code) -from a string. The function takes -three arguments: - -1. Name of a hashing algorithm (see the -[list of supported algorithms](https://www.php.net/manual/en/function.hash-algos.php#refsect1-function.hash-algos-examples)) -2. Value to hash -3. Secret key - -{% highlight json %} -{ - "function": "hash_hmac", - "args": [ - "sha256", - "12345abcd5678efgh90ijk", - "TeaPot" - ] -} -{% endhighlight %} - -The above will return `d868d581b2f2edd09e8e7ce12c00723b3fcffb6a5d74c40eae9d94181a0bf731`. - -See an [example](#api-default-parameters). - - -### hash -This function works similarly to the `hash_hmac` function but requires only two arguments (no secret key required): - -1. The name of a hashing algorithm (see the - [list of supported algorithms](https://www.php.net/manual/en/function.hash-algos.php#refsect1-function.hash-algos-examples)). -2. The value to hash. - -{% highlight json %} -{ - "function": "hash", - "args": [ - "sha256", - "12345abcd5678efgh90ijk" - ] -} -{% endhighlight %} - -### time -The [`time` function](https://www.php.net/manual/en/function.time.php) returns the current time as a -[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time). -To obtain the current time in a more readable format, use the -the [`date` function](#date). It takes no arguments. - -{% highlight json %} -{ - "function": "time" -} -{% endhighlight %} - -The above will produce something like `1492674974`. - -### date -The [`date` function](https://www.php.net/manual/en/function.date.php) formats the provided or the current -timestamp into a human readable format. The function takes either one or two arguments: - -1. [Formatting string](https://www.php.net/manual/en/function.date.php#refsect1-function.date-parameters) -2. Optional [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time); if not provided, the current time is used. - -{% highlight json %} -{ - "function": "date", - "args": [ - "Y-m-d" - ] -} -{% endhighlight %} - -The above will produce something like `2017-04-20`. - -{% highlight json %} -{ - "function": "date", - "args": [ - "Y-m-d H:i:s", - 1490000000 - ] -} -{% endhighlight %} - -The above will produce `2017-03-20 8:53:20`. - -See an [example](#user-data). - -### strtotime -The [`strtotime` function](https://www.php.net/manual/en/function.strtotime.php) converts a string date into a [Unix timestamp](https://en.wikipedia.org/wiki/Unix_time). The function takes -one or two arguments: - -1. String date -2. Base for relative dates (see below) - -{% highlight json %} -{ - "function": "strtotime", - "args": [ - "21 oct 2017 9:16pm" - ] -} -{% endhighlight %} - -The above will produce `1508620560`, which represents the date `2017-10-21 21:16:00`. However, the -[`strtotime` function](https://www.php.net/manual/en/function.strtotime.php) is most useful with relative dates which it also allows. For example, you can -write: - -{% highlight json %} -{ - "function": "strtotime", - "args": [ - "-7 days", - 1508620560 - ] -} -{% endhighlight %} - -The above will give `1508015760`, which represents the date `2017-10-14 21:16:00`. The second argument -specifies the base date (as a Unix timestamp) from which the relative date is computed. This is particularly -useful for [incremental extraction](/extend/generic-extractor/incremental/). Also note that -it is common to combine the `strtotime` and `date` functions to convert between string and timestamp -representation of a date. - -See an [example](#nested-strtotime). - -### sprintf -The `sprintf` function formats values and inserts them into a string. The `sprintf` function maps directly to -the [original PHP function](https://www.php.net/manual/en/function.sprintf.php), which is very versatile and has many -uses. The function accepts two or more arguments: - -1. String with [formatting directives](https://www.php.net/manual/en/function.sprintf.php) (marked with the percent character `%`) -2. Values inserted into the string: - -{% highlight json %} -{ - "function": "sprintf", - "args": [ - "Three %s are %.2f %s.", - "apples", - 0.5, - "plums" - ] -} -{% endhighlight %} - -The above will produce `Three apples are 0.50 plums.` - -See a [simple insert example](#api-base-url) or a [formatting example](#job-placeholders). - -### concat -The `concat` function concatenates an arbitrary number of strings into one. For example: - -{% highlight json %} -{ - "function": "concat", - "args": [ - "Hen", - "Or", - "Egg" - ] -} -{% endhighlight %} - -The above will produce `HenOrEgg` (see [example 1](#api-base-url), [example 2](#headers)). See also the -[`implode` function](#implode). - -### implode -The [`implode` function](https://www.php.net/manual/en/function.implode.php) concatenates an arbitrary number -of strings into one using a delimiter. The function takes -two arguments: - -1. Delimiter string which is used for the concatenation -2. Array of values to be concatenated - -For example: - -{% highlight json %} -{ - "function": "implode", - "args": [ - ",", - [ - "apples", - "oranges", - "plums" - ] - ] -} -{% endhighlight %} - -The above will produce `apples,oranges,plums` (see an [example](#headers)). -The delimiter can be empty, in which case the `implode` function is equivalent to the [`concat` function](#concat): - -{% highlight json %} -{ - "function": "implode", - "args": [ - "", - [ - "Hen", - "Or", - "Egg" - ] - ] -} -{% endhighlight %} - -### ifempty -The `ifempty` function can be useful for handling optional values. The function takes two arguments and -returns the first one if it is not empty. If the first argument is empty, it returns the second argument. - -{% highlight json %} -{ - "function": "ifempty", - "args": [ - "", - "Banzai" - ] -} -{% endhighlight %} - -The above will return `Banzai`. For the `ifempty` function, an empty string and the values `0` and `null` are -considered 'empty'. - -See an [example](#optional-job-parameters). - -## Function Contexts -Every place in the Generic Extractor configuration in which a function may be used may allow different arguments of the function. -This is referred to as a **function context**. Many contexts share access to **configuration attributes**. - -### Configuration Attributes -The configuration attributes are accessible in specific function contexts and they represent the entire [`config`](/extend/generic-extractor/configuration/config/) -section of the Generic Extractor configuration. There is some processing involved: - -- The [`jobs`](/extend/generic-extractor/configuration/config/jobs/) section is removed entirely. -- All other values are flattened (keys are concatenated using a dot `.`) into a one-level deep object. -- The result object is available in a property named `attr`. - -For example, the following configuration: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com" - }, - "config": { - "debug": true, - "outputBucket": "get-tutorial", - "server": "localhost:8888", - "incrementalOutput": false, - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ], - "http": { - "headers": { - "X-AppKey": "ThisIsSecret", - "X-Auth": { - "function": "concat", - "args": [ - "Tea", - "Pot" - ] - } - } - }, - "userData": { - "tag": "fullExtract", - "mode": "development" - }, - "mappings": { - "content": { - "whatever": "foobar" - } - } - } - } -} -{% endhighlight %} - -will be converted to the following function context: - -{% highlight json %} -{ - "attr": { - "debug": true, - "outputBucket": "mock-server", - "server": "localhost:8888", - "incrementalOutput": false, - "http.headers.X-AppKey": "ThisIsSecret", - "http.headers.X-Auth.function": "concat", - "http.headers.X-Auth.args.0": "Tea", - "http.headers.X-Auth.args.1": "Pot", - "userData.tag": "fullExtract", - "userData.mode": "development", - "mappings.content.whatever": "foobar" - } -} -{% endhighlight %} - -See [example [EX119]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/119-function-nested-config). - -### Base URL Context -The Base URL function context is used when setting the [`baseURL` for API](/extend/generic-extractor/configuration/api/#base-url), and it -contains [configuration attributes](/#function-contexts). - -See an [example](#api-base-url). - -### Headers Context -The Headers function context is used when setting the [`http.headers` for API](/extend/generic-extractor/configuration/api/#headers) -or the [`http.headers` in config](/extend/generic-extractor/configuration/config/#http), and it contains -[configuration attributes](/#function-contexts). - -See an [example](#headers). - -### Parameters Context -The Parameters function context is used when setting job [request parameters --- `params`](/extend/generic-extractor/configuration/config/jobs/#request-parameters). -It contains [configuration attributes](/#function-contexts) plus the times of the current -(`currentStart`) and previous (`previousStart`) run of Generic Extractor. -The times are [Unix timestamps](https://en.wikipedia.org/wiki/Unix_time). -If the extraction is run for the first time, `previousStart` is 0. - -With the following configuration: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com" - }, - "config": { - "debug": true, - "outputBucket": "get-tutorial", - "server": "localhost:8888", - "jobs": [ - ... - ] - } - } -} -{% endhighlight %} - -the parameters function context will contain: - -{% highlight json %} -{ - "attr": { - "debug": true, - "outputBucket": "mock-server", - "server": "localhost:8888" - }, - "time": { - "previousStart": 0, - "currentStart": 1492678268 - } -} -{% endhighlight %} - -See an [example of using parameters context](#job-parameters). - -The `time` values are used in [incremental processing](/extend/generic-extractor/incremental/). - -### Placeholder Context -The Placeholder function context refers to configuration of [placeholders in child jobs](/extend/generic-extractor/configuration/config/jobs/children/#placeholders). -When using function to process a placeholder value, the placeholder must be specified as an object with the `path` property. -Therefore instead of writing: - -{% highlight json %} -"placeholders": { - "user-id": "userId" -} -{% endhighlight %} - -write: - -{% highlight json %} -"placeholders": { - "user-id": { - "path": "userId", - "function": ... - } -} -{% endhighlight %} - -The placeholder function context contains the following structure: - -{% highlight json %} -{ - "placeholder": { - "value": "???" - } -} -{% endhighlight %} - -where `???` is the value obtained from the response JSON from the path provided in the `path` property -of the placeholder. - -See an [example](#job-placeholders). - -### User Data Context -The User Data function context is used when setting the [`userData`](/extend/generic-extractor/configuration/config/#user-data). -The parameters context contains [configuration attributes](/#function-contexts) plus the times of the current (`currentStart`) and -previous (`previousStart`) run of Generic Extractor. The User Data Context is therefore -same as the [Parameters Context](#parameters-context). - -See an [example](#user-data). - -### Login Authentication Context -The Login Authentication function context is used in the -[login authentication](/extend/generic-extractor/configuration/api/authentication/login/) method. -Functions are supported in both [`loginRequest`](/extend/generic-extractor/configuration/api/authentication/login/#configuration-parameters) -and [`apiRequest` ](/extend/generic-extractor/configuration/api/authentication/login/#configuration-parameters) configurations. -The `loginRequest` function context contains [configuration attributes](/#function-contexts). -In the `apiRequest` context, the flattened reponse of the login request is available additionally -to the [configuration attributes](/#function-contexts). -The login authentication context is the same for both `params` and `headers` -[login authentication configuration options](/extend/generic-extractor/configuration/api/authentication/login/#configuration-parameters). If the -login authentication request returns e.g.: - -{% highlight json %} -{ - "user": "John Doe", - "authorization": { - "token": "quiteSecret", - "validUntil": "2017-20-12 12:20:17" - } -} -{% endhighlight %} - -The following function context will be available in the API request headers and query: - -{% highlight json %} -{ - "attr": { - "outputBucket": "mock-server" - }, - "response": { - "user": "John Doe", - "authorization.token": "quiteSecret", - "authorization.validUntil": "2017-20-12 12:20:17" - } -} -{% endhighlight %} - -The login response is available in the `response` node. The `attr` node contains [configuration attributes](/extend/generic-extractor/functions/#configuration-attributes). -See an [example](/extend/generic-extractor/configuration/api/authentication/login/#login-authentication-with-functions) and a more -[complicated example](/extend/generic-extractor/configuration/api/authentication/login/#login-authentication-with-login-and-api-request) of using functions in -both login request and API request. - -### Query Authentication Context -The Query Authentication function context is used in the -[query authentication](/extend/generic-extractor/configuration/api/authentication/query/) method. -The Query Authentication Context contains [configuration attributes](/#function-contexts) plus -a representation of the complete HTTP request to be sent (`request`) plus a key -value list of query parameters of the HTTP request (`query`). - -The following configuration: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "http": { - "defaultOptions": { - "params": { - "account": "admin" - } - } - }, - "authentication": { - "type": "query", - "query": { - "signature": { - "function": "sha1", - "args": [ - "time", - { - "attr": "#api-key" - } - ] - } - } - } - }, - "config": { - "#api-key": "12345abcd5678efgh90ijk", - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "params": { - "showColumns": "all" - } - } - ] - } - } -} -{% endhighlight %} - -leads to the following function context: - -{% highlight json %} -{ - "query": { - "account": "admin", - "showColumns": "all" - }, - "request": { - "url": "http:\/\/example.com\/users?account=admin&showColumns=all", - "path": "\/users", - "queryString": "account=admin&showColumns=all", - "method": "GET", - "hostname": "example.com", - "port": 80, - "resource": "\/users?account=admin&showColumns=all" - }, - "attr": { - "#api-key": "12345abcd5678efgh90ijk", - "outputBucket": "mock-server" - } -} -{% endhighlight %} - -See the [basic example](#api-default-parameter) and a [more complicated example](#api-query-authentication). - -### OAuth 2.0 Authentication Context -The OAuth Authentication Context is used for the -[`oauth20`](/extend/generic-extractor/configuration/api/authentication/oauth20/) authentication method -(it is not applicable to `oauth10`) and contains the following: - -- Representation of the complete HTTP request to be sent (`request`) -- A key value list of query parameters of the HTTP request (`query`) -- An `authorization` section containing the response from the OAuth service provider - -This context is available for both the `headers` and `query` sections of the `oauth20` authentication methods. - -The following configuration: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "oauth20", - "format": "json", - "headers": { - "Authorization": { - "function": "concat", - "args": [ - "Bearer ", - { - "authorization": "#data.access_token" - } - ] - } - } - } - }, - "config": { - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"status\": \"ok\",\"access_token\": \"testToken\", \"foo\": {\"bar\": \"baz\"}}", - "appKey": "clientId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -leads to the following function context: - -{% highlight json %} -{ - "query": { - "showColumns": "all" - }, - "request": { - "url": "http:\/\/example.com\/users?showColumns=all", - "path": "\/users", - "queryString": "showColumns=all", - "method": "GET", - "hostname": "example.com", - "port": 80, - "resource": "\/users?showColumns=all" - }, - "authorization": { - "data.status": "ok", - "data.access_token": "testToken", - "data.foo.bar": "baz" - "timestamp": 1492949837, - "nonce": "99206d94a6846841", - "clientId": "clientId", - } -} -{% endhighlight %} - -The `authorization` section of the configuration contains the -[OAuth2 response](/extend/generic-extractor/configuration/api/authentication/oauth20/). The function context contains -the parsed and flattened response fields under the key `data`, provided that the response was sent in JSON format -and that [`"format": "json"`](/extend/generic-extractor/configuration/api/authentication/oauth20/#configuration) was set. - -In the response above, these are the keys `data.status`, `data.access_token`, `data.foo.bar`. This is defined -entirely by the behavior of the OAuth Service provider. If the response is a plaintext (usually directly a token), -then the entire response is available in the field `data`. - -Apart from that, the fields `timestamp` (Unix timestamp of the request), -`nonce` (cryptographic [nonce](https://en.wikipedia.org/wiki/Cryptographic_nonce) for -signing the request) and `clientId` (the value of `authorization.oauth_api.credentials.appKey`, which is obtained when -the application is published) are added to the `authorization` section. - -For usage, see [OAuth examples](/extend/generic-extractor/configuration/api/authentication/oauth20/). - -### OAuth 2.0 Login Authentication Context -The OAuth Login Authentication Context is used for the -[`oauth20.login`](/extend/generic-extractor/configuration/api/authentication/oauth20-login/) authentication method -(it is not applicable to `oauth20`). The OAuth Login Authentication context contains -OAuth information split into the properties `consumer` (response obtained from the service provider) and -`user` (data obtained from the user). This context is available for -both the `headers` and `params` sections of the `oauth20` authentication methods. -For the context available in the `apiRequest` configuration, see the [login authentication](/extend/generic-extractor/functions/#login-authentication-context). - -The following configuration: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com", - "authentication": { - "type": "oauth20.login", - ... - } - }, - "config": { - ... - } - }, - "authorization": { - "oauth_api": { - "credentials": { - "#data": "{\"status\": \"ok\",\"access_token\": \"testToken\", \"mac_secret\": \"iAreSoSecret123\", \"foo\": {\"bar\": \"baz\"}}", - "appKey": "clientId", - "#appSecret": "clientSecret" - } - } - } -} -{% endhighlight %} - -leads to the following function context: - -{% highlight json %} -{ - "consumer": { - "client_id": "clientId", - "client_secret": "clientSecret" - }, - "user": { - "status": "ok", - "access_token": "testToken", - "mac_secret": "iAreSoSecret123", - "foo.bar": "baz" - } -} -{% endhighlight %} - -The `authorization` section of the configuration contains the -[OAuth2 response](/extend/generic-extractor/configuration/api/authentication/oauth20/). The function context -contains the parsed and flattened response fields in the `user` property. The content of the -`user` property is fully dependent on the response of the OAuth service provider. The -`consumer` property contains the `client_id` and `client_secret` which contain values of -`authorization.oauth_api.credetials.appKey` and -`authorization.oauth_api.credetials.appSecret` respectively. -(These are obtained by Keboola when the application is published). - -For usage, see [OAuth Login examples](/extend/generic-extractor/configuration/api/authentication/oauth20-login/). - -## Examples - -### API Base URL -When [publishing your Generic Extractor configuration](/extend/generic-extractor/publish/), chances are -you want the end-user to provide a part of the API configuration. Due to the limitations of -[how templates work](/extend/generic-extractor/publish/#configuration-considerations), the parameter -obtained from the end-user configuration will be only available in the `config` section. - -Let's say that the end-user enters `www.example.com` as the API server and that values become -available as the `server` property of the `config` section, for instance: - -{% highlight json %} -"config": { - "outputBucket": "ge-tutorial", - "server": "www.example.com", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] -} -{% endhighlight %} - -This means that the [configuration attributes](#configuration-attributes) will be available as: - -{% highlight json %} -{ - "attr": { - "outputBucket": "ge-tutorial", - "server": "www.example.com" - } -} -{% endhighlight %} - -Then use the [`concat` function](#concat) to access that value and merge it with other parts to create the -final API URL (`http://example.com/api/1.0/`): - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": { - "function": "concat", - "args": [ - "http://", - { - "attr": "server" - }, - "/api/1.0/" - ] - } - } - } -} -{% endhighlight %} - -See [example [EX087] with concat](https://github.com/keboola/generic-extractor/tree/master/doc/examples/087-function-baseurl) -or an alternative [example [EX088] with sprintf](https://github.com/keboola/generic-extractor/tree/master/doc/examples/088-function-baseurl-sprintf). - -### API Default Parameters -Suppose you have an API which expects a `tokenHash` parameter to be sent with every request. The -token hash is supposed to be generated by the SHA-256 hashing algorithm from a token and secret -you obtain. - -Because the [`api.http.defaultOptions.params`](/extend/generic-extractor/configuration/api/#headers) option does not -support functions, either supply the parameters in the [`jobs.params`](/extend/generic-extractor/configuration/config/jobs/#request-parameters) -configuration, or use [API Query Authentication](/extend/generic-extractor/configuration/api/authentication/query/). -Using (or abusing) the API Query Authentication is possible if the default parameters represent authentication, or -if the API does not use any authentication method (two authentication methods are not possible): - -The below configuration reads the `#api-key` and `#secret-key` parameters from the `config` section, -computes SHA-256 hash and sends it as a `tokenHash` parameter with every request. - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "query", - "query": { - "tokenHash": { - "function": "hash_hmac", - "args": [ - "sha256", - { - "attr": "#api-key" - }, - { - "attr": "#secret-key" - } - ] - } - } - } - }, - "config": { - "#api-key": "12345abcd5678efgh90ijk", - "#secret-key": "TeaPot", - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users" - } - ] - } - } -} -{% endhighlight %} - -See [example [EX099]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/099-function-query-parameters). - -The solution with using the `jobs.params` configuration can look like this: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "#api-key": "12345abcd5678efgh90ijk", - "#secret-key": "TeaPot", - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "tokenHash": { - "function": "hash_hmac", - "args": [ - "sha256", - { - "attr": "#api-key" - }, - { - "attr": "#secret-key" - } - ] - } - } - } - ] - } - } -} -{% endhighlight %} - -The only practical difference is that the `tokenHash` parameter is going to be sent only with -the single `users` job. - -See [example [EX098]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/098-function-hmac). - -### API Query Authentication -Suppose you have an API with only a single endpoint `/items` to which you have to -pass a `type` parameter to list resources of a given type. On top of that, the API requires -an `apiToken` parameter and a `signature` parameter (a hash of the token and type) to be sent with every request. -The following configuration handles the situation: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://mock-server:80/101-function-query-auth/", - "authentication": { - "type": "query", - "query": { - "apiToken": { - "attr": "#token" - }, - "signature": { - "function": "sha1", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "#token" - }, - { - "query": "type" - } - ] - } - ] - } - }, - "apiRequest": { - "headers": { - "X-Api-Token": "token" - } - } - } - }, - "config": { - "#token": "1234abcd567efg890hij", - "debug": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "items", - "dataType": "users", - "params": { - "type": "users" - } - }, - { - "endpoint": "items", - "dataType": "orders", - "params": { - "type": "orders" - } - } - ] - } - } -} -{% endhighlight %} - -There are two jobs, both to the same endpoint (`items`), but with a different `type` parameter and `dataType`. -The authentication method `query` adds two more parameters to each request: `apiToken` (contain the value -of `config.#token`) and `signature`. The `signature` parameter is created as an SHA-1 hash of the -token and resource type (`"query": "type"` is taken from the `jobs.params.type` value). - -See [example [EX101]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/101-function-query-auth). - -### Job Placeholders -Let's say you have an API with an endpoint `/users`, returning a list of users, and an -endpoint `/user/{userId}`, returning details of a specific user with a given ID. The list response -looks like this: - -{% highlight json %} -[ - { - "id": 3, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -To obtain the details of the first user, the user-id has to be padded to five digits. The details API call for the -first user must be sent to `/user/00003`, and for the second user to `/user/00234`. To achieve this, use the -`sprintf` function, which allows [number padding](https://www.php.net/manual/en/function.sprintf.php#example-6129). - -The following `placeholders` configuration in the child job calls the function with the first argument set to -`%'.05d` (which is a sprintf [format](https://www.php.net/manual/en/function.sprintf.php) to pad with zero to five digits) -and the second argument set to the value of the `id` property found in the parent response. The placeholder path must -be specified in the `path` property. That means that the configuration: - -{% highlight json %} -"placeholders": { - "user-id": "id" -} -{% endhighlight %} - -has to be converted to: - -{% highlight json %} -"placeholders": { - "user-id": { - "path": "id", - "function": "sprintf", - "args": [ - "%'.05d", - { - "placeholder": "value" - } - ] - } -} -{% endhighlight %} - -The following `user-detail` table will be extracted: - -|id|name|address\_city|address\_country|address\_street|parent\_id| -|123|John Doe|London|UK|Whitehaven Mansions|00003| -|234|Jane Doe|St Mary Mead|UK|High Street|00234| - -Notice that the `parent_id` column contains the processed value and not the original one. - -See [example [EX085]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/085-function-job-placeholders), -or a not-so-useful [example [EX086]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/086-function-job-placeholders-reference) -(using reference). - -### Job Parameters -Let's say you have an API which requires you to send a hash of a certain value with every request. Specifically, -each request must be done with the [HTTP POST method](/extend/generic-extractor/tutorial/rest/#method) with content: - -{% highlight json %} -{ - "token": "someValue" -} -{% endhighlight %} - -The following configuration does exactly that. The value of the token is taken from the configuration -root (using the `attr` reference). This is useful in case the configuration is used as part of a -[template](/extend/generic-extractor/publish/). The actual hash will be generated of the `NotSoSecret` value. - - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "debug": true, - "outputBucket": "mock-server", - "tokenValue": "NotSoSecret", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "method": "POST", - "params": { - "token": { - "function": "md5", - "args": [ - { - "attr": "tokenValue" - } - ] - } - } - } - ] - } - } -} -{% endhighlight %} - -See [example [EX089]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/089-function-job-parameters-md5) -or an alternative [example [EX090] with SHA1 hash](https://github.com/keboola/generic-extractor/tree/master/doc/examples/090-function-job-parameters-sha1). -or an alternative [example [EX136]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/136-post-request-functions) with more deeply nested functions. - -### Optional Job Parameters -Let's say you have an API which allows you to send the list of columns to be contained in the API response. -For example, to list users and include their `id`, `name` and `login` properties, call -`/users?showColumns=id,name,login`. Also, you want to enter these values as an array in the `config` section because -the config is generated by a [template](/extend/generic-extractor/publish/). If the end-user -does not wish to filter the columns, they can -list all the columns (which would be annoying) or leave the column filter empty. In that case, the API -call would be `/users?showColumns=all`. - -The following configuration does exactly that: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "columns": "", - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "method": "GET", - "params": { - "showColumns": { - "function": "ifempty", - "args": [ - { - "attr": "columns" - }, - "all" - ] - } - } - } - ] - } - } -} -{% endhighlight %} - -See [example [EX097]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/097-function-ifempty). - -### User Data -Assume that you have an API returning a response that does not contain any time information. For example: - -{% highlight json %} -[ - { - "id": 3, - "name": "John Doe" - }, - { - "id": 234, - "name": "Jane Doe" - } -] -{% endhighlight %} - -Add the extraction time to each record so that you at least know when each record was obtained -(when the creation time is unknown). Add additional data to each record using -the [`userData` configuration](/extend/generic-extractor/configuration/config/#user-data): - -{% highlight json %} -"userData": { - "extractionDate": { - "function": "date", - "args": [ - "Y-m-d H:i:s", - { - "time": "currentStart" - } - ] - } -} -{% endhighlight %} - -The following table will be extracted: - -|id|name|extractionDate| -|3|John Doe|2017-04-20 10:17:20| -|234|Jane Doe|2017-04-20 10:17:20| - -Or, use an alternative configuration that also adds the current date: - -{% highlight json %} -"userData": { - "extractionDate": { - "function": "date", - "args": [ - "Y-m-d H:i:s" - ] - } -} -{% endhighlight %} - -But whereas the first one puts a single same date to each record, the alternative configuration will return different times for different records -as they are extracted. - -See [example [EX091]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/091-function-user-data) or -an alternative [example [EX092] with a set date](https://github.com/keboola/generic-extractor/tree/master/doc/examples/092-function-user-date-set-date). - -### Headers -Suppose you have an API which requires you to send a custom `X-Api-Auth` header with every request. -The header must contain a user name and password separated by a colon. For instance, `JohnDoe:TopSecret`. - -This can be done using the following `api` configuration: - -{% highlight json %} -"api": { - "baseUrl": "http://example.com/", - "http": { - "headers": { - "X-Api-Auth": { - "function": "concat", - "args": [ - { - "attr": "credentials.#username" - }, - ":", - { - "attr": "credentials.#password" - } - ] - } - } - } -} -{% endhighlight %} - -Alternatively, achieve the same result using the `implode` function: - -{% highlight json %} -"api": { - "baseUrl": "http://mock-server:80/093-function-api-http-headers/", - "http": { - "headers": { - "X-Api-Auth": { - "function": "implode", - "args": [ - ":", - [ - { - "attr": "credentials.#username" - }, - { - "attr": "credentials.#password" - } - ] - ] - } - } - } -} -{% endhighlight %} - -Both configurations rely on having the username and password parameters -in the [`config` section](/extend/generic-extractor/configuration/config/), in this case also nested in the `credentials` property: - -{% highlight json %} -"config": { - "credentials": { - "#username": "JohnDoe", - "#password": "TopSecret" - }, - "jobs": ... -} -{% endhighlight %} - -See [example [EX093]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/093-function-api-http-headers) or an -[alternative example [EX094] setting headers in the `config` section](https://github.com/keboola/generic-extractor/tree/master/doc/examples/094-function-config-headers). - -### Nested Functions -If the API in the [above example](#headers) tries to mimic the -[HTTP authentication](/extend/generic-extractor/configuration/api/authentication/basic/), -the header has to be sent as a [base64 encoded](https://en.wikipedia.org/wiki/Base64#MIME) value. -That is instead of sending a `JohnDoe:TopSecret`, you have to send `Sm9obkRvZTpUb3BTZWNyZXQ=`. To do this -you have to wrap the `concat` function which generates the header value in another function (`base64_encode`). - -{% highlight json %} -"api": { - "baseUrl": "http://example.com/", - "http": { - "headers": { - "X-Api-Auth": { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "#username" - }, - ":", - { - "attr": "#password" - } - ] - } - ] - } - } - } -} -{% endhighlight %} - -See [example [EX095]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/095-function-nested). - -### Nested StrToTime -Suppose you have an API which requires you to specify the `from` and `to` date parameters to obtain orders created -in that time interval. You want to specify only the `from` date and extract a week of data. -Enter (preferably in a [template](/extend/generic-extractor/publish/)) the -value `2017-10-04` and send an API request to -`/orders?from=2017-10-04&to=2017-10-11`. The following configuration can be used: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/" - }, - "config": { - "startDate": "2017-10-04", - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "method": "GET", - "params": { - "from": { - "attr": "startDate" - }, - "to": { - "function": "date", - "args": [ - "Y-m-d", - { - "function": "strtotime", - "args": [ - "+7 days", - { - "function": "strtotime", - "args": [ - { - "attr": "startDate" - } - ] - } - ] - } - ] - } - } - } - ] - } - } -} -{% endhighlight %} - -The configuration probably seems rather complicated, so taken apart -- the most innermost part: - -{% highlight json %} -{ - "function": "strtotime", - "args": [ - { - "attr": "startDate" - } - ] -} -{% endhighlight %} - -takes the value from the `config` property `startDate` (which is `2017-10-04`) and converts it to -a timestamp value (`???` below). - -Then there is an outer part: - -{% highlight json %} -{ - "function": "strtotime", - "args": [ - "+7 days", - ??? - ] -} -{% endhighlight %} - -that takes the timestamp representing `2017-10-04` and adds 7 days to it. This yields another -timestamp value (`???` below). - -Then there is another outer part: - -{% highlight json %} -{ - "function": "date", - "args": [ - "Y-m-d", - ??? - ] -} -{% endhighlight %} - -converting the timestamp back to a string format (`Y-m-d` format) which yields `2017-10-11`. -This value is assigned to the `to` parameter of the API call. - -See [example [EX096]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/096-function-nested-from-to). diff --git a/extend/generic-extractor/functions.png b/extend/generic-extractor/functions.png deleted file mode 100644 index 0a893a9f..00000000 Binary files a/extend/generic-extractor/functions.png and /dev/null differ diff --git a/extend/generic-extractor/generic-intro.png b/extend/generic-extractor/generic-intro.png deleted file mode 100644 index c50c3e4b..00000000 Binary files a/extend/generic-extractor/generic-intro.png and /dev/null differ diff --git a/extend/generic-extractor/incremental.md b/extend/generic-extractor/incremental.md index 8ce38991..054499ea 100644 --- a/extend/generic-extractor/incremental.md +++ b/extend/generic-extractor/incremental.md @@ -1,216 +1,5 @@ ---- -title: Incremental Loading -permalink: /extend/generic-extractor/incremental/ ---- - -* TOC -{:toc} - -Extracting data incrementally is universally beneficial --- it **speeds up the extraction** and **lowers the load** on both the API and -[Keboola Storage](https://help.keboola.com/storage/) (thus saving -[credits](https://help.keboola.com/management/limits/#project-power)). - -## Options -After you have incrementally extracted data from an API, the data must be -[incrementally loaded](https://help.keboola.com/storage/tables/#incremental-loading) -into Storage. To do that, simply set `"incrementalOutput": true` in the `config` section. - -There are, however, a number of implications in the incremental loads. It essentially boils downs to the following use cases, -depending on what kind of data you are importing (extracting from an API): - -- The imported data contains only **added entries**. When `incrementalOutput` is turned on, the data will be -simply appended to the target table in Storage. Turning `incrementalOutput` to false probably makes no sense -because the table will contain only the new entries. -- The imported data contains **added and modified entries**. When `incrementalOutput` is turned on, set a primary key on the table so that new rows are added and existing [rows are updated](https://help.keboola.com/storage/tables/#primary-key-deduplication). -If the primary key is not set, the modified entries will be duplicated in the target table. Turning -`incrementalOutput` to false probably makes no sense because the table will contain only the new entries. -- The imported data contains **all rows**. In this case, set a primary key for the table or turn -`incrementalOutput` to false. Turning `incrementalOutput` to true probably makes no sense because the table will -contain duplicate entries. If you set the primary key, new rows will be added and modified rows will be updated. -Note that in this case more [credits](https://help.keboola.com/management/limits/#project-power) are consumed. - -In neither of these situations will the missing rows get deleted. If you want to do so, the only way is -to turn `incrementalOutput` to false and do full loads. - -Using incremental loads obviously requires some support from the API. Generic Extractor supports incremental -loads by using [`previousStart`](/extend/generic-extractor/functions/#parameters-context) and the -[`time` function](/extend/generic-extractor/functions/#time). Setting the primary key is done using -[mappings](/extend/generic-extractor/configuration/config/mappings/). - -## Examples - -### Previous Start Example -Assume you have an API supporting a parameter `modified_since` which expects a -[Unix Timestamp](https://en.wikipedia.org/wiki/Unix_time). The response then contains only the -records that were modified after the specified date. The following configuration can be used: - -{% highlight json %} -{ - "config": { - "incrementalOutput": true, - "outputBucket": "mock-server", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "modified_since": { - "time": "previousStart" - } - } - } - ] - } -} -{% endhighlight %} - -The configuration adds the `modified_since` parameter as a reference to the internal -[`time.previousStart` value](/extend/generic-extractor/functions/#parameters-context), which contains the timestamp of the last -**successful start** of the extraction of the particular configuration. The request generated by this configuration is something like: - - GET /users?modified_since=1492606006 - -where `1492606006` is the variable timestamp of the last successful start. This introduces state into the -Generic Extractor configuration as it now remembers when it last successfully ran. This means -that if you run the above configuration every five minutes, it will extract the data modified within the last five minutes. -If you run it every hour, it will extract the data modified within the last hour. - -Should one of the runs fail or be skipped for any reason, the extraction will pick up where it ended the last time it was successful. -See [example [EX107]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/107-incremental-load). - -The last successful time is stored in the [configuration state](https://help.keboola.com/extend/common-interface/config-file/#state-file). -If for some reason you need to reset it, -[update the configuration via API](https://api.keboola.com/?service=storage#put-/v2/storage/branch/-branchId-/components/-componentId-/configs/-configurationId-). - -### Previous Start Date -If an API similar to the one in the [above example](#previous-start-example) requires the date to be -sent as a string, the following jobs configuration (which uses the [`date` function](/extend/generic-extractor/functions/#date)) -can be used: - -{% highlight json %} -{ - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "modified_since": { - "function": "date", - "args": [ - "Y-m-d H:i:s", - { - "time": "previousStart" - } - ] - } - } - } - ] -} -{% endhighlight %} - -This sends a request like: - - GET /users?modified_since=2017-04-19%2012%3A46%3A46 - -in a more readable [url-decoded](https://meyerweb.com/eric/tools/dencoder/) form: - - GET /users?modified_since=2017-04-19 12:46:46 - -Otherwise the configuration behaves the same way as the [previous example](#previous-start-example). - -See [example [EX108]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/108-incremental-load-date). - -### Incremental Load From To -Another option is an API which requires the `from` and `to` parameters. The following -configuration generates the `from` date as the date of the last extraction (using the [`time.previousStart` -value](/extend/generic-extractor/functions/#parameters-context)). It also generates the `to` date as the date -of the current extraction (using the [`time.currentStart` value](/extend/generic-extractor/functions/#parameters-context)): - -{% highlight json %} -{ - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "from": { - "function": "date", - "args": [ - "Y-m-d", - { - "time": "previousStart" - } - ] - }, - "to": { - "function": "date", - "args": [ - "Y-m-d", - { - "time": "currentStart" - } - ] - } - } - } - ] -} -{% endhighlight %} - -This configuration will send a request similar to this one: - - GET /109-incremental-load-from-to/users?from=2017-04-19&to=2017-04-24 - -See [example [EX109]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/109-incremental-load-from-to). - -### Incremental Relative Load -Suppose you have an API supporting the `from` and `to` parameters as in [the above example](#incremental-load-from-to) and -want to extract the last day data. It can be done using the following configuration: - -{% highlight json %} -{ - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "from": { - "function": "date", - "args": [ - "Y-m-d", - { - "function": "strtotime", - "args": [ - "-1 day", - { - "time": "currentStart" - } - ] - } - ] - }, - "to": { - "function": "date", - "args": [ - "Y-m-d", - { - "time": "currentStart" - } - ] - } - } - } - ] -} -{% endhighlight %} - -This configuration leads to a request similar to this one: - - GET /110-incremental-relative/users?from=2017-04-23&to=2017-04-24 - -Remember, this is not a truly reliable incremental load. If you put such configuration -into an orchestration, and the configuration does not run for some reason, you may miss some data. -However, this may still be a useful approach for obtaining samples of data for POCs. - -See [example [EX110]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/110-incremental-relative). +--- +title: Incremental Loading +permalink: /extend/generic-extractor/incremental/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/incremental/ +--- diff --git a/extend/generic-extractor/index.md b/extend/generic-extractor/index.md index 6b7d62ab..30691134 100644 --- a/extend/generic-extractor/index.md +++ b/extend/generic-extractor/index.md @@ -1,64 +1,5 @@ --- title: Generic Extractor permalink: /extend/generic-extractor/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/ --- - -* TOC -{:toc} - -Generic Extractor is a [Keboola component](/overview/) that acts like a customizable -[HTTP REST](/extend/generic-extractor/tutorial/rest/) client. It can be configured to extract data -from virtually any sane web API. - -Due to the versatility of different APIs running in the wild, Generic Extractor offers many [**configuration options**](/extend/generic-extractor/configuration/). - -You may opt to use the [**visual builder**](/extend/generic-extractor/configuration/#user-interface), which provides a very convenient way -of configuring and testing the configuration. With it, you can build -an entirely new extractor for Keboola in **less than an hour**. - -{: .image-popup} -![Generic Extractor - UI](/extend/generic-extractor/ui.png) - -To get started quickly, follow our [Generic Extractor tutorial](/extend/generic-extractor/tutorial). - -## Generic Extractor Requirements -Generic Extractor allows you to extract data from an API into Keboola only by configuring it. -No programming skills or additional tools are required. You just need to do two easy things before you start: - -- Become familiar with [JSON format](/extend/generic-extractor/tutorial/json/). -- Have the documentation of your chosen API at hand. The API should be [RESTful](/extend/generic-extractor/tutorial/rest/) -and, more or less, follow the HTTP specification. - -## Configuration & Development -Again, if you are new to Generic Extractor, we strongly suggest you go through the -[Generic Extractor tutorial](/extend/generic-extractor/tutorial/). It outlines the basic principles and the most important features. - -With the new convenient user interface, you can set up and test the connection in a few clicks, -just like you are used to in some other popular API development tools. - -Features such as cURL import, request tests, output mapping generator, or dynamic function templates and evaluation make the configuration process as easy as ever. - -If you intend to develop a more complicated configuration, check out how to [run Generic Extractor locally](/extend/generic-extractor/running/). -The documentation includes [several examples](https://github.com/keboola/generic-extractor/tree/master/doc) that [can also be run locally](/extend/generic-extractor/running/#running-examples). - -## Publishing Generic Extractor Configuration -Each Generic Extractor configuration can be [published](/extend/generic-extractor/publish/) as -a new standalone component. However, for registration, configurations must be -[converted to templates](/extend/generic-extractor/publish/#submission). - -Publishing your Generic Extractor configuration is **not required**. However, when published, -it can be easily used in multiple projects. A great advantage of using templates is that they -do not limit the configuration. You can always switch to JSON -[free-form configuration](/extend/generic-extractor/publish/#submission) when necessary. - -Also, templates can be used only with published components based on Generic Extractor configurations. - -## Generic Extractor Source -As with other Keboola components, the Generic Extractor connector is available on -[GitHub](https://github.com/keboola/generic-extractor/). Apart from the -main repository, it uses some vital libraries (which partially define its capabilities): - -- [Juicer](https://github.com/keboola/juicer) --- component responsible for processing HTTP JSON responses -- [CSV Map](https://github.com/keboola/php-csvmap) --- library that converts JSON data into CSV tables -- [Filter](https://github.com/keboola/php-filter) --- library that allows to match values together -- [JSON Parser](https://github.com/keboola/php-jsonparser) --- JSON parser which produces CSV tables while maintaining relations diff --git a/extend/generic-extractor/map.md b/extend/generic-extractor/map.md index 1c6aee47..c87207f6 100644 --- a/extend/generic-extractor/map.md +++ b/extend/generic-extractor/map.md @@ -1,20 +1,5 @@ ---- -title: Generic Extractor Parameter Map -permalink: /extend/generic-extractor/map/ ---- -*To configure your first Generic Extractor, follow our [tutorial](/extend/generic-extractor/tutorial/).* - -Use the following sample configuration to navigate among various **configuration options**: - -{% highlight json %} -{% include config-map.json %} -{% endhighlight %} - - - +--- +title: Generic Extractor Parameter Map +permalink: /extend/generic-extractor/map/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/map/ +--- diff --git a/extend/generic-extractor/publish.md b/extend/generic-extractor/publish.md index a1b3619f..1ee16dd5 100644 --- a/extend/generic-extractor/publish.md +++ b/extend/generic-extractor/publish.md @@ -1,351 +1,7 @@ ---- -title: Publish Generic Extractor -permalink: /extend/generic-extractor/publish/ -redirect_from: - - /extend/generic-extractor/registration/ ---- - -* TOC -{:toc} - -It is possible to publish a Generic Extractor configuration as a completely separate component. -This enables sharing the API extractor between various projects and simplifies its further configuration. - -## Configuration Considerations -Before converting your configuration to a universally available component, consider -what values in the configuration should be provided by the end-user (typically authentication values). -Then design a [configuration schema](https://help.keboola.com/extend/component/ui-options/configuration-schema/) for setting -those values. You can [test the schema online](http://jeremydorn.com/json-editor/) ([alternative](https://mozilla-services.github.io/react-jsonschema-form/)). -The values obtained from the end user will be stored in the [`config` property](/extend/generic-extractor/configuration/config/). -Modify your configuration to read those values from there. - -Do not forget that if you prefix a value with a hash `#`, it will be -[encrypted](/overview/encryption/) once the configuration is saved. -Also, try to make the extractor [work incrementally](/extend/generic-extractor/incremental/) -if possible. - -## Publishing -To publish your Generic Extractor configuration, you need to [create a new component](https://help.keboola.com/extend/component/tutorial/) in -the [Developer Portal](https://components.keboola.com/). Choose an appropriate name and the type `extractor`. Once you -have created the component, edit it, and fill in the following details: - -- **Repository** - - **Type** --- AWS ECR - - **Image Name** -- `147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/ex-generic-v2` - - **Tag** -- see the [Generic Extractor GitHub repository](https://github.com/keboola/generic-extractor/releases) - - **Region** -- leave empty -- **UI options** --- set to `genericTemplatesUI` - -For a list of available tags, see the [Generic Extractor GitHub repository](https://github.com/keboola/generic-extractor/). It is also possible to use the `latest` tag, which points to the highest available tag. However, -we recommend that you configure your component with a specific tag and update it manually to avoid problems with breaking changes -in future Generic Extractor releases. - -Because the UI is assumed to be `genericTemplatesUI`, provide a -[**configuration schema**](https://help.keboola.com/extend/component/ui-options/configuration-schema/) and -a **template** to be used in conjunction with the schema. Optionally, the template UI may also contain an interface to -negotiate [OAuth authentication](/extend/generic-extractor/configuration/api/authentication/#oauth). -An example of the template UI is shown in the picture below. - -{: .image-popup} -![Screenshot - Generic templates UI](/extend/generic-extractor/template-1.png) - -The `Config` section of the templates UI is defined by the configuration schema you provide. -The `Template` section contains at least one template. A template is simply a configuration of -Generic Extractor. - -For example, you might want to provide one configuration for incremental loading -and a different configuration for full loading. The template UI also has the option to -`Switch to JSON editor`, which displays the configuration JSON and allows the end user to modify it. -Notice that the JSON editor allows modification only to the [`config`](/extend/generic-extractor/configuration/config/) -section. Other sections, such as [`api`](/extend/generic-extractor/configuration/api/) or -[`authorization.oauth_api`](/extend/generic-extractor/configuration/api/authentication/#oauth), may not be modified by the end user. - -You can review existing templates in their [GitHub repository](https://github.com/keboola/kbc-ui-templates/tree/master/resources). -If you feel confident, you can send a pull request with your templates, otherwise submit it when requesting the -[publication of your component](https://help.keboola.com/extend/publish/). - -## Example -Let's say you have the following working API configuration -(see [example [EX111]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/111-templates-example)): - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "token", - "headers": { - "Authorization": { - "function": "base64_encode", - "args": [ - "JohnDoe:TopSecret" - ] - } - } - }, - "apiRequest": { - "headers": { - "X-Api-Auth": "auth.token" - } - } - }, - "default": { - "http": { - "params": { - "accountId": 123 - } - } - } - }, - "config": { - "incrementalOutput": true, - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "type": "active" - } - }, - { - "endpoint": "orders", - "dataType": "orders" - } - ] - } - } -} -{% endhighlight %} - -and you identify that four values of that configuration need to be specified by the end user: -`JohnDoe`, `TopSecret`, `123`, and `active`. - -For each of the values, create a parameter of the appropriate type: - -- `JohnDoe` --- a string parameter `login` -- `TopSecret` --- a string parameter `#password` (it will be encrypted) -- `123` --- a numeric parameter `accountId` -- `active` --- an enumeration parameter `userType` with values `active`, `inactive`, `all` - -The parameter names are completely arbitrary. However, they must not conflict with existing -configuration properties of [Generic Extractor](/extend/generic-extractor/configuration/config/) (e.g., `jobs`, `mappings`). -Now create a [configuration schema](https://help.keboola.com/extend/component/ui-options/configuration-schema/) for the four parameters. - -{% highlight json %} -{ - "title": "Person", - "type": "object", - "properties": { - "login": { - "type": "string", - "title": "Login:", - "description": "Your API user name", - "minLength": 4 - }, - "#password": { - "type": "string", - "title": "Password:", - "description": "Your API password", - "minLength": 4 - }, - "accountId": { - "type": "integer", - "title": "Account ID", - "description": "See in-app help for obtaining Account Id" - }, - "userType": { - "title": "User type:", - "type": "string", - "enum": [ - "active", - "inactive", - "all" - ], - "default": "active", - "description": "Specify which users to obtain" - } - }, - "required": [ - "login", "#password", "accountId", "userType" - ] -} -{% endhighlight %} - -When you test the [schema online](http://jeremydorn.com/json-editor/) ([alternative](https://mozilla-services.github.io/react-jsonschema-form/)), it will produce a -configuration JSON: - -{: .image-popup} -![Screenshot - Schema Test](/extend/generic-extractor/schema-test.png) - -{% highlight json %} -{ - "login": "JohnDoe", - "#password": "TopSecret", - "accountId": 123, - "userType": "inactive" -} -{% endhighlight %} - -The above properties will be merged into the [`config` section](/extend/generic-extractor/configuration/config/). Now -modify the configuration so that it reads them from there using [functions and references](/extend/generic-extractor/functions/). - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "token", - "headers": { - "Authorization": { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "username" - }, - ":", - { - "attr": "#password" - } - ] - } - ] - } - } - }, - "apiRequest": { - "headers": { - "X-Api-Auth": "auth.token" - } - } - } - }, - "config": { - "incrementalOutput": true, - "username": "JohnDoe", - "#password": "TopSecret", - "accountId": 123, - "userType": "active", - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "accountId": { - "attr": "accountId" - }, - "type": { - "attr": "userType" - } - } - }, - { - "endpoint": "orders", - "dataType": "orders", - "params": { - "accountId": { - "attr": "accountId" - } - } - } - ] - } - } -} -{% endhighlight %} - -The argument to the `base64_encode` function is now the -[`concat` function](/extend/generic-extractor/functions/#concat), which joins together the -values of the `username` and `#password` fields. The `accountId` parameter needs to be moved to the -`jobs` section because the `http.defaultOptions.params` section does not support function calls (yet!). -The `type` parameter was changed to a reference to the `userType` field -(see [example [EX111]](https://github.com/keboola/generic-extractor/tree/master/doc/examples/111-templates-example)). - -When you handled the configuration parameters, turn the configuration into a template. Place -the `api` section to a separate, individual `api.json` file: - -{% highlight json %} -{ - "baseUrl": "http://example.com/", - "authentication": { - "type": "login", - "loginRequest": { - "endpoint": "token", - "headers": { - "Authorization": { - "function": "base64_encode", - "args": [ - { - "function": "concat", - "args": [ - { - "attr": "username" - }, - ":", - { - "attr": "#password" - } - ] - } - ] - } - } - }, - "apiRequest": { - "headers": { - "X-Api-Auth": "auth.token" - } - } - } -} -{% endhighlight %} - -Once you make sure that the extractor works as it did before, -remove the user provided values (`username`, `#password`, `accountId`, `userType`) from -the `config` section, put it in a `data` section and add `name` and `description` to it. -Save the file into a separate `template.json` file. The template file therefore contains -`name`, `description` and `data` nodes. - -{% highlight json %} -{ - "name": "Basic", - "description": "Basic incremental template", - "data": { - "incrementalOutput": true, - "jobs": [ - { - "endpoint": "users", - "dataType": "users", - "params": { - "accountId": { - "attr": "accountId" - }, - "type": { - "attr": "userType" - } - } - }, - { - "endpoint": "orders", - "dataType": "orders", - "params": { - "accountId": { - "attr": "accountId" - } - } - } - ] - } -} -{% endhighlight %} - -Create as many `template.json` files as you wish. However, all of them need to share the same `api.json` -configuration. When you want to publish your component, attach the `api.json` and all `template.json` files. +--- +title: Publish Generic Extractor +permalink: /extend/generic-extractor/publish/ +redirect_from: + - /extend/generic-extractor/registration/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/publish/ +--- diff --git a/extend/generic-extractor/running.md b/extend/generic-extractor/running.md index c9a92724..620fbf69 100644 --- a/extend/generic-extractor/running.md +++ b/extend/generic-extractor/running.md @@ -1,159 +1,5 @@ ---- -title: Running Generic Extractor -permalink: /extend/generic-extractor/running/ ---- - -* TOC -{:toc} - -Generic Extractor is normally run from within the Keboola user interface. It can be found in the **Extractors** section -and all you need to do is provide its configuration JSON. No other settings are necessary. - -{: .image-popup} -![Screenshot - Generic Extractor Configuration](/extend/generic-extractor/configuration.png) - -Because creating the configuration JSON can be a non-trivial task, there are some things which can help -you in developing the configuration. - -## Debug Mode -Debug mode can be turned on by setting `"debug": true` in the `config` section of the configuration, e.g.: - -{% highlight json %} -{ - "api": { - ... - }, - "config": { - "debug": true, - ... - } -} -{% endhighlight %} - -In debug mode, the extractor displays all API requests it sends, helping you understand what is really happening, -why something is skipped, etc. - -{: .image-popup} -![Screenshot - Debug Logs](/extend/generic-extractor/events.png) - -**Warning:** If the API sends sensitive data (e.g. authorization token) in the URL, these may become -visible in the events. Also, debug mode considerably slows the extraction. Therefore it should never -be turned on in production configurations. - -## Running Locally -If you are working on a complicated configuration, or developing a new component based on -Generic Extractor, running every configuration from the Keboola UI may be slow and tedious. -You may run Generic Extractor locally, provided that you have access to Docker. -The following is **not necessary** to run or configure Generic Extractor in Keboola. - -### Run Built Version -Create an empty directory somewhere and in it create a `config.json` file with a -configuration you want to execute. For example: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://api.github.com", - "http": { - "Accept": "application/json", - "Content-Type": "application/json;charset=UTF-8" - } - }, - "config": { - "debug": true, - "jobs": [ - { - "endpoint": "/orgs/keboola/members", - "dataType": "members" - } - ] - } - } -} -{% endhighlight %} - -Then run Generic Extractor in the current directory by executing the following command on *nix systems: - - docker run -v $(pwd):/data 147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/ex-generic-v2:latest - -or on Windows: - - docker run -v %cd%:/data 147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/ex-generic-v2:latest - -You should see: - - DEBUG: Using NO Auth [] [] - DEBUG: Using automatic conversion of single values to arrays where required. [] [] - DEBUG: GET /orgs/keboola/members HTTP/1.1 Host: api.github.com User-Agent: Guzzle/5.3.1 curl/7.38.0 PHP/7.0.17 [] [] - DEBUG: Analyzing members {"rowsAnalyzed":[],"rowsToAnalyze":7} [] - DEBUG: Processing results for __kbc_default. [] [] - INFO: Extractor finished successfully. [] [] - -along with the output tables created in `/out/tables` sub-directory of the current directory. -It is recommended to remove the contents of the `out/tables` directory before running the extractor again. - -**Important:** Generic Extractor itself is not able to decrypt encrypted values. That means that when you -supply the configuration directly in the `config.json` file, you must always provide decrypted values --- e.g.: - -{% highlight json %} -{ - ..., - "config": { - "#username": "JohnDoe", - "#password": "TopSecret", - ... - } -} -{% endhighlight %} - -When you store such configuration in the Keboola UI, it will automatically be encrypted: - -{% highlight json %} -{ - ..., - "config": { - "#username": "JohnDoe", - "#password": "KBC::ComponentProjectEncrypted==r13Khq0lR4ycDNTujirz5/GMqNEVZ4tZ2OTmRcsNYqlP/a/STMelWtz9R8yEtr3ck6KiYA7XrL8pqIQv9S7Ro28KNZgmqtSNzKhFcEsItPnTDCQqvnU99q2a0ES+oN/v", - ... - } -} -{% endhighlight %} - -The above configuration then **cannot** be run locally. -Read more about [encryption](/overview/encryption/). - -### Building and Running the Image -To build the container from source: - -- Clone this repository: `git clone https://github.com/keboola/generic-extractor.git`. -- Switch to the created directory: `cd generic-extractor`. -- Build the container: `docker compose build`. -- Install dependencies locally: `docker compose run --rm extractor composer install`. -- Create a **data folder** for configuration: `mkdir data`. - -To run the built container: - -- Create a configuration file `config.json` in the **data folder**. -- Run the extraction: `docker compose run --rm extractor`. -- You will find the extracted data in the `out/tables` sub-directory of the **data folder**. - -Before running the extractor again, it is recommended to clear the `out` directory by -running `docker compose run --rm extractor rm -rf data/out`. - -## Running Examples -[All examples](https://github.com/keboola/generic-extractor/tree/master/doc) referenced in this documentation are actually runnable against the proper API. Because -it is difficult to find the specific API for the case (and gain access to it), you can test -these configurations against a [mock server](https://github.com/keboola/ex-generic-mock-server). -Each example contains a set of requests (`*.request` file) and responses (`*.response`) and -optionally their headers (`*.requestHeaders` and `*.responseHeaders`). - -To run the examples: - -- Clone Generic Extractor repository: `git clone https://github.com/keboola/generic-extractor.git`. -- Navigate to the documentation directory: `cd generic-extractor/doc`. -- Run a single example of your choice, e.g.: `docker compose run -e "KBC_EXAMPLE_NAME=001-simple-job" extractor`. -- The output will be available in `examples/001-simple-job/out/tables`. -- Or run all examples by executing `./run-samples.sh`. - -If you want to create your own example, follow the instructions in the [mock server repository](https://github.com/keboola/ex-generic-mock-server/blob/master/README.md#creating-examples). +--- +title: Running Generic Extractor +permalink: /extend/generic-extractor/running/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/running/ +--- diff --git a/extend/generic-extractor/schema-test.png b/extend/generic-extractor/schema-test.png deleted file mode 100644 index f141f29f..00000000 Binary files a/extend/generic-extractor/schema-test.png and /dev/null differ diff --git a/extend/generic-extractor/template-1.png b/extend/generic-extractor/template-1.png deleted file mode 100644 index bce5ad80..00000000 Binary files a/extend/generic-extractor/template-1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/2_child.png b/extend/generic-extractor/tutorial/2_child.png deleted file mode 100644 index 2678c3f8..00000000 Binary files a/extend/generic-extractor/tutorial/2_child.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/base_configuration.png b/extend/generic-extractor/tutorial/base_configuration.png deleted file mode 100644 index 2f5ca629..00000000 Binary files a/extend/generic-extractor/tutorial/base_configuration.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/basic.md b/extend/generic-extractor/tutorial/basic.md index 703be457..aebc1ab8 100644 --- a/extend/generic-extractor/tutorial/basic.md +++ b/extend/generic-extractor/tutorial/basic.md @@ -1,226 +1,5 @@ ---- -title: Basic Configuration -permalink: /extend/generic-extractor/tutorial/basic/ ---- - -* TOC -{:toc} - -Before configuring Generic Extractor, you should have a basic understanding -of [REST API](/extend/generic-extractor/tutorial/rest/) and -[JSON format](/extend/generic-extractor/tutorial/json/). This tutorial uses the -[MailChimp API](https://mailchimp.com/developer/reference/), so -have its documentation at hand. You also need the -[MailChimp API key](/extend/generic-extractor/tutorial/#prepare). - -## Configuration -Generic Extractor configuration is written in [JSON format](/extend/generic-extractor/tutorial/json/) -and comprises [several sections](/extend/generic-extractor/configuration/#configuration-sections) (a -[configuration map](/extend/generic-extractor/map/) for navigation is available). - -A [user interface](/extend/generic-extractor/configuration/#user-interface) is available that can help you with the configuration - and generate the JSON configuration for you. - -### Base Configuration -The first configuration part is a `Base Configuration` section where you can set the Base URL and Authentication method of the -API you connect to. - -In our case, we will use the MailChimp API, so the `Base URL` will be `https://us13.api.mailchimp.com/3.0/`, and the `Authentication` method will be `Basic Authentication`. - -**Important:** Make sure that the `baseUrl` URL ends with a slash! - -In the `Destination` section, you can set: -- The `Output Bucket` where the data will be stored. It will be set to the ID of the [Storage Bucket](https://help.keboola.com/storage/buckets/) -- `Incremental Output` option, which defines whether you want the result to overwrite the existing data or append to it. [See more](/extend/generic-extractor/incremental/) - - Note that when using Incremental Output, you should set up the mapping. - -{: .image-popup} -![Base Configuration](/extend/generic-extractor/tutorial/base_configuration.png) - -#### JSON -If you switch to the `JSON` mode, the created configuration will translate to the `api` section where you set the **basic properties** of the API. -In the most simple case, this is the `baseUrl` property and `authentication`, as shown in this JSON snippet: - -{% highlight json %} -{ - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - } - } -} -{% endhighlight %} - -**Important:** Make sure that the `baseUrl` URL ends with a slash! - -The `config` section describes the **actual extraction**. Its most important parts are the `outputBucket` and -`jobs` properties. `outputBucket` must be set to the ID of the [Storage Bucket](https://help.keboola.com/storage/buckets/) -where the data will be stored. If no bucket exists, it will be created. - -It also contains the authentication parameters, such as `username` and `password`. Start with this -configuration section: - -{% highlight json %} -"config": { - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13", - "outputBucket": "ge-tutorial", - "incrementalOutput": false -} -{% endhighlight %} - -The `password` property is prefixed with the hash mark `#`, meaning the value will be [encrypted](/overview/encryption/) once -you save the configuration. - -### Endpoint Section -Once you set up the Base Configuration, you can set up the actual endpoint to be queried. - -Start by clicking the **+ New Endpoint** button: - -{: .image-popup} -![New Endpoint](/extend/generic-extractor/tutorial/new_endpoint.png) - -You will be asked to provide the relative endpoint URL path. In our case, we will use the `campaigns` endpoint. - -{: .image-popup} -![New Endpoint modal](/extend/generic-extractor/tutorial/new_endpoint_modal.png) - -- In the URL section, you will see the resulting endpoint URL combined with the `Base URL` you set up in the `Base Configuration` section. - - **Important:** Do not start the URL with a slash. If you do so, the URL -will be absolute from the domain `https://us13.api.mailchimp.com/campaigns`, which is invalid -(it is missing the `3.0` part). An alternative would be to put `/3.0/campaigns` in the `endpoint` property. -- Alternatively, you may opt to create the endpoint using the **cURL command**, which is usually available in the API documentation. - -Now you are getting close to a runnable configuration, and you may proceed with testing the configuration by clicking the `TEST ENDPOINT` button: - -{: .image-popup} -![Test endpoint](/extend/generic-extractor/tutorial/test_endpoint.png) - -In the test endpoint popup, you will see the following sections: -- `Records`: The actual data that will be used for parsing. -- `Response`: The response from the API. It includes headers, status code, and response body in the `data` property. -- `Request`: The request that has been sent to the API. -- `Debug log`: A log outputted by the component for debugging purposes. - -In the `Records` section, you will now see the following: -``` -[ - "The root element of the response is not a list; please change your Data Selector path to list" -] -``` - -Also, if you try to run this configuration, you will get an error similar to this: - - The response contains more than one array! Use the 'dataField' parameter to specify a key to the data array. - (endpoint: campaigns, arrays in the response root: campaigns, _links) - -That means that the extractor got the response but cannot automatically process it. The `Data Selector` path doesn't point to an array. - -Examine the `data` attribute of the response, and you will see the following objects: `campaigns`, `total_items`, and `_links`: - -{% highlight json %} -{ - "campaigns": [ - { - "id": "42694e9e57", - "type": "regular", - ... - }, - { - "id": "f6276207cc", - "type": "regular", - ... - } - ], - "total_items": 2, - "_links": [ - { - "rel": "parent", - "href": "https://usX.api.mailchimp.com/3.0/", - "method": "GET", - "targetSchema": "https://api.mailchimp.com/schema/3.0/Root.json" - }, - { - "rel": "self", - "href": "https://usX.api.mailchimp.com/3.0/campaigns", - "method": "GET", - "targetSchema": "https://api.mailchimp.com/schema/3.0/Campaigns/Collection.json", - "schema": "https://api.mailchimp.com/schema/3.0/CollectionLinks/Campaigns.json" - } - ] -} -{% endhighlight %} - -Generic Extractor expects the response to be an array of items. If it receives an object, it -searches its properties to find an array. Finding multiple arrays will be confusing because it is unclear which array you want. -To fix this, change the `Data Selector` parameter (aka `dataField`) to value `campaigns` to point to the array of items you want to extract. - -{: .image-popup} -![Selector](/extend/generic-extractor/tutorial/data_selector.png) - -Now, run the configuration by clicking the **Run** button and go to the job details to see what happened: - -{: .image-popup} -![Screenshot - Generic Extractor job](/extend/generic-extractor/tutorial/job-1.png) - -The extraction produced two tables. The `in.c-ge-tutorial.campaigns` table contains all the -fields of a campaign and as many rows as you have campaigns. - -{: .image-popup} -![Screenshot - Campaigns Table](/extend/generic-extractor/tutorial/table-campaigns-sample.png) - -The table `in.c-ge-tutorial.campaigns__links` contains the contents of the `_links` property. -Because the `_links` property is a nested array within a single campaign object, it cannot be easily -represented in a single column of the `campaigns` table. Generic Extractor, therefore, replaces the column -value with a generated key, for example, `campaigns_75d5b14d79d034cd07a9d95d5f0ca5bd`, and automatically -creates a new table that has the column `JSON_parentId` with that value so that you can join the tables together. - -### Final JSON Configuration -The main parts of the configuration and their nesting are shown in the following schema: - -{: .image-popup} -![Schema - Generic Extractor configuration](/extend/generic-extractor/generic-intro.png) - -The resulting JSON configuration will look like this: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - } - } - "config": { - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13", - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "campaigns", - "dataField": { - "path": "campaigns", - "delimiter": "." - } - } - ] - } - } -} -{% endhighlight %} - -**Important:** It may seem confusing that the `endpoint` and `dataField` properties are set to `campaigns`. -This is just a coincidence; the `endpoint` property refers to the `campaigns` in the resource URL, and -the `dataField` refers to the `campaigns` property in the JSON retrieved as the API response. - -## Summary -The above tutorial demonstrates a very basic configuration of Generic Extractor. The extractor is capable -of doing much more; see other parts of this tutorial for an explanation of pagination, jobs and mapping: - -- [Pagination](/extend/generic-extractor/tutorial/pagination/) --- breaks a result with many items into separate pages. -- [Jobs](/extend/generic-extractor/tutorial/jobs/) --- describe the API endpoints - (resources) to be extracted. -- [Mapping](/extend/generic-extractor/tutorial/mapping/) --- describes how the JSON - response is converted into CSV files that will be imported into Storage. +--- +title: Basic Configuration +permalink: /extend/generic-extractor/tutorial/basic/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/basic/ +--- diff --git a/extend/generic-extractor/tutorial/child_debug.png b/extend/generic-extractor/tutorial/child_debug.png deleted file mode 100644 index 926488ff..00000000 Binary files a/extend/generic-extractor/tutorial/child_debug.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/child_endpoint.png b/extend/generic-extractor/tutorial/child_endpoint.png deleted file mode 100644 index b431a9b8..00000000 Binary files a/extend/generic-extractor/tutorial/child_endpoint.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/config-1.png b/extend/generic-extractor/tutorial/config-1.png deleted file mode 100644 index 55f407d9..00000000 Binary files a/extend/generic-extractor/tutorial/config-1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/configuration-schema.svg b/extend/generic-extractor/tutorial/configuration-schema.svg deleted file mode 100644 index 0c010c24..00000000 --- a/extend/generic-extractor/tutorial/configuration-schema.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/extend/generic-extractor/tutorial/create_endpoint_child.png b/extend/generic-extractor/tutorial/create_endpoint_child.png deleted file mode 100644 index 07dbf2f4..00000000 Binary files a/extend/generic-extractor/tutorial/create_endpoint_child.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/create_mapping.png b/extend/generic-extractor/tutorial/create_mapping.png deleted file mode 100644 index 8fa62ba3..00000000 Binary files a/extend/generic-extractor/tutorial/create_mapping.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/create_mapping_toggle.png b/extend/generic-extractor/tutorial/create_mapping_toggle.png deleted file mode 100644 index bf1c4d65..00000000 Binary files a/extend/generic-extractor/tutorial/create_mapping_toggle.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/data_selector.png b/extend/generic-extractor/tutorial/data_selector.png deleted file mode 100644 index 91083e09..00000000 Binary files a/extend/generic-extractor/tutorial/data_selector.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/img.png b/extend/generic-extractor/tutorial/img.png deleted file mode 100644 index 02755a43..00000000 Binary files a/extend/generic-extractor/tutorial/img.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/img_1.png b/extend/generic-extractor/tutorial/img_1.png deleted file mode 100644 index 70e32bd7..00000000 Binary files a/extend/generic-extractor/tutorial/img_1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/index.md b/extend/generic-extractor/tutorial/index.md index 81965cd3..d4736fbd 100644 --- a/extend/generic-extractor/tutorial/index.md +++ b/extend/generic-extractor/tutorial/index.md @@ -1,83 +1,5 @@ --- title: Generic Extractor Tutorial permalink: /extend/generic-extractor/tutorial/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/ --- - -* TOC -{:toc} - -In this tutorial, we will guide you through configuring Generic Extractor for a new API. -In our case, MailChimp --- an email marketing service. - -Even though there already is a MailChimp extractor available in Keboola as a -[component](/extend/generic-extractor/publish/) based on Generic Extractor, -the [MailChimp](https://mailchimp.com/) API is ideal for this tutorial because it is fairly -easy to understand and has excellent documentation. - -## Prepare -There are a few things you need to do before you start: - -1. Read our [quick introduction to REST](/extend/generic-extractor/tutorial/rest/) for a basic understanding of -**HTTP requests** and **REST API**. -2. Read our [quick introduction to JSON](/extend/generic-extractor/tutorial/json/) to learn how to write **JSON -configurations**. -3. [Create your MailChimp account](https://login.mailchimp.com/signup/), free of charge, if you do not have one -already. -4. Follow the MailChimp wizard or [help](https://us13.admin.mailchimp.com/campaigns/) and **fill the account with -data**: - - Create a new Campaign (choose the *regular type*). - - Create a new List and add some addresses to it (preferably yours). - - Go back to Campaigns, select your campaign and hit "Next" in the bottom right corner. - - Design a test email and send it. - - Check that you have received the email and read it. -5. To gain access to the MailChimp API, go to your Account detail and under Extras find the option to -[generate your API Key](https://mailchimp.com/help/about-api-keys/#Find-or-Generate-Your-API-Key). -It will look like this: `c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13`. - -## Get Started -Let's take a closer look at the [MailChimp API](https://mailchimp.com/developer/) now. -There are plenty of documentation guides available. To explore the API and review what information is in -each resource, use, for example, the [Playground](https://us1.api.mailchimp.com/playground/). - -The basic properties of the API are outlined in the -[Getting Started Guide](https://mailchimp.com/developer/guides/get-started-with-mailchimp-api-3/#resources). -The following are the crucial parts for our use-case: - -- The root API URL is `https://.api.mailchimp.com/3.0`, where `` refers to a data center for your -account. The data center is the last part of the API key; if the API key is -`c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13`, the root URL is `https://us13.api.mailchimp.com/3.0`. -- API Authentication can be done using **HTTP Basic Authentication** where you use **any string** (text) for -username and the API key for password. - -Now, go straight to the documentation of the -[**Campaign** resource](https://mailchimp.com/developer/reference/campaigns/). -Because you intend to extract data from MailChimp, the only part you are interested in is the **Read Method**. - -{: .image-popup} -![Screenshot - Read Campaign Documentation](/extend/generic-extractor/tutorial/mailchimp-api-docs-1.png) - -The documentation lists the URL (`/campaigns`) of the **Campaign Resource**, and the query string -parameters (these go into the URL), such as `fields`, `count`, etc. It also lists example -requests and responses. The response body is in [JSON](/extend/generic-extractor/tutorial/json) format and starts like this: - -{% highlight json %} -{ - "campaigns": [ - { - "id": "42694e9e57", - "type": "regular", - "create_time": "2015-09-15T14:40:36+00:00", - ... -{% endhighlight %} - -## Next Steps -Now you have everything you need to actually start extracting the data. Continue with your Generic Extractor -configuration here: - -- [Basic configuration](/extend/generic-extractor/tutorial/basic/) --- sets the basic properties of the API and describes the actual extraction. -- [Pagination](/extend/generic-extractor/tutorial/pagination/) --- breaks a result with a - large number of items into separate pages. -- [Jobs](/extend/generic-extractor/tutorial/jobs/) --- describes the API endpoints - (resources) to be extracted. -- [Mapping](/extend/generic-extractor/tutorial/mapping/) --- describes how the JSON - response is converted into CSV files that will be imported into Storage. \ No newline at end of file diff --git a/extend/generic-extractor/tutorial/job-1.png b/extend/generic-extractor/tutorial/job-1.png deleted file mode 100644 index 4b784849..00000000 Binary files a/extend/generic-extractor/tutorial/job-1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/job-2.png b/extend/generic-extractor/tutorial/job-2.png deleted file mode 100644 index 1b113919..00000000 Binary files a/extend/generic-extractor/tutorial/job-2.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/job-table-1.png b/extend/generic-extractor/tutorial/job-table-1.png deleted file mode 100644 index fedfc0f4..00000000 Binary files a/extend/generic-extractor/tutorial/job-table-1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/job-table-2.png b/extend/generic-extractor/tutorial/job-table-2.png deleted file mode 100644 index 49237514..00000000 Binary files a/extend/generic-extractor/tutorial/job-table-2.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/jobs.md b/extend/generic-extractor/tutorial/jobs.md index f3ba6b6e..0dfc98e0 100644 --- a/extend/generic-extractor/tutorial/jobs.md +++ b/extend/generic-extractor/tutorial/jobs.md @@ -1,251 +1,5 @@ ---- -title: Jobs Tutorial -permalink: /extend/generic-extractor/tutorial/jobs/ ---- - -* TOC -{:toc} - -On your way through the Generic Extractor tutorial, you have learned about - -- [Basic configuration](/extend/generic-extractor/tutorial/basic/) and -- [Configuration of pagination](/extend/generic-extractor/tutorial/pagination/). - -Now, we will show you how to use Generic Extractor's **sub-jobs**. - -Let's start this section with a closer examination of the `campaigns` resource of the MailChimp API. -Besides retrieving multiple campaigns using the `/campaigns` endpoint, it can also retrieve detailed -information about a single campaign using `/campaigns/{campaign_id}`. - -{: .image-popup} -![Screenshot - Mailchimp documentation](/extend/generic-extractor/tutorial/mailchimp-api-docs-1.png) - -Moreover, each campaign has three **sub-resources**: -`/campaigns/{campaign_id}/content`, `/campaigns/{campaign_id}/feedback` -and `/campaigns/{campaign_id}/send-checklist`. The `{campaign_id}` expression represents a placeholder -that a specific campaign ID should replace. To retrieve the sub-resource, use child jobs. - -## Child Jobs - -In the -[previous part](/extend/generic-extractor/tutorial/pagination/#running) of the tutorial, you created this job -property in the Generic Extractor configuration: - -{% highlight json %} -"jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns" - } -] -{% endhighlight %} - -All sub-resources are retrieved by configuring the `children` property in JSON; its structure is the same as the -structure of the `jobs` property, but it must additionally define `placeholders`. - -In the UI, you just create a new endpoint and mark it as a `Child Job` of the parent job of your choice. Any placeholders, -e.g., variables that will be set from the parent object, should be enclosed in curly braces, e.g., `{campaign_id}`. - -{: .image-popup} -![Create endpoint](/extend/generic-extractor/tutorial/create_endpoint_child.png) - -Once the endpoint is created, the `Placeholders section` will be prefilled for you. We will set the `Response Path` value to `id`, -since we want to use the `id` property from the parent response to replace the `{campaign_id}` placeholder in the child endpoint. - -{: .image-popup} -![Child endpoint](/extend/generic-extractor/tutorial/child_endpoint.png) - - -Now, you can test the endpoint as in previous examples. -The `Mapping.Data Selector` (aka `dataField`) property must refer to an array, i.e., `items` or `_links` in our case -(see the [documentation](https://mailchimp.com/developer/reference/campaigns/campaign-checklist/)). - -When you look at the debug log, you will also see that the connector is making all the parent requests: - -{: .image-popup} -![child_debug](child_debug.png) - -**The resulting underlying JSON will look like this:** - -{% highlight json %} -"jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns", - "children": [ - { - "endpoint": "campaigns/{campaign_id}/send-checklist", - "dataField": "items", - "placeholders": { - "campaign_id": "id" - } - } - ] - } -] -{% endhighlight %} - - -The `children` are executed for each element retrieved from the parent endpoint, i.e., for each campaign. -The `placeholders` setting connects the placeholders used in the `endpoint` property with -the data in the actual parent response. -That means that the `campaign_id` placeholder in the `campaigns/{campaign_id}/send-checklist` endpoint -will be replaced by the `id` property of the JSON [response](https://mailchimp.com/developer/reference/campaigns/): - -{: .image-popup} -![Screenshot - Mailchimp docs](/extend/generic-extractor/tutorial/mailchimp-api-docs-2.png) - -Also, note that the placeholder name is completely arbitrary (i.e., it is just a coincidence that -it is also named `campaign_id` in the Mailchimp documentation). Therefore, the following configuration is -also valid: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "offset", - "offsetParam": "offset", - "limitParam": "count", - "limit": 1 - } - }, - "config": { - "debug": true, - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13", - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns", - "children": [ - { - "endpoint": "campaigns/{cid}/send-checklist", - "dataField": "items", - "placeholders": { - "cid": "id" - } - } - ] - } - ] - } - } -} -{% endhighlight %} - -Running the above configuration gives you a new table named, for example, -`in.c-ge-tutorial.campaigns__campaign_id__send-checklist`. The table -contains messages from campaign checking. You will see something like this: - -{: .image-popup} -![Screenshot - Job Table](/extend/generic-extractor/tutorial/job-table-1.png) - -Note that apart from the API response properties `type`, `heading`, and `details`, an additional field, -`parent_id`, was added. It contains the value of the placeholder (`campaign_id`) for the particular -request. So, to join the two tables together in SQL, you would use the join condition: - - campaigns.id=campaigns__campaign_id__send-checklist.parent_id - -However, you have to remember what table the `parent_id` column refers to. - -## Multiple Jobs -You have probably noticed that the `jobs` and `children` properties are arrays. It means that you can retrieve multiple -endpoints in a single configuration. Let's pick the campaign `content` sub-resource too: - -{: .image-popup} -![second child](/extend/generic-extractor/tutorial/2_child.png) - - -The placeholder configuration is the same, however, -the question is what to put in the `Data Selector` (`dataField`). If you examine the sample [response](https://mailchimp.com/developer/reference/campaigns/campaign-content/) -after running the test endpoint, it looks like this: - -{% highlight json %} - -{ - "plain_text": "** Designing...*|END:IF|*", - "html": "", - "_links": [ - { - "rel": "parent", - "href": "https://usX.api.mailchimp.com/3.0/campaigns/42694e9e57", - "method": "GET", - "targetSchema": "https://api.mailchimp.com/schema/3.0/Campaigns/Instance.json" - }, - ... - ] -} - -{% endhighlight %} - -If you use JSON configuration with no `dataField` like in the above configuration and run it, you will obtain a table like this: - -{: .image-popup} -![Screenshot - Job Table](/extend/generic-extractor/tutorial/job-table-2.png) - -This is not what you expected. Instead of obtaining the campaign content, you -got the `_links` property from the response because Generic Extractor automatically -picks an array in the response. To get the entire response as a **single table record**, set `dataField` -to the [path](/extend/generic-extractor/tutorial/json/#references) in the object. Because you want to use the -**entire response**, set `dataField` to `.` to start in the root. - -***Note:** If you use the UI editor, the `Data Selector` (`dataField`) is automatically set to `.` by default.* - -**The resulting JSON:** - -{% highlight json %} -"jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns", - "children": [ - { - "endpoint": "campaigns/{campaign_id}/send-checklist", - "dataField": { - "path": "items", - "delimiter": "." - }, - "placeholders": { - "campaign_id": "id" - } - }, - { - "endpoint": "campaigns/{campaign_id}/content", - "dataField": { - "path": ".", - "delimiter": "." - }, - "placeholders": { - "campaign_id": "id" - } - } - ] - } -] -{% endhighlight %} - -Running the above configuration will get you the table `in.c-ge-tutorial.campaigns__campaign_id__content` -with columns like `plain_text`, `html`, and others. - -You will also get the table `in.c-ge-tutorial.campaigns__campaign_id__content__links`. It -represents the `links` property of the `content` resource. The links table contains the -`JSON_parentId` column, which includes a generated hash, such as -`campaigns/{campaign_id}/content_1c3b951ece2a05c1239b06e99cf804c2`, whose value is inserted into -the `links` column of the campaign content table. This is done automatically because once -you say that the entire response is supposed to be a single table row, the array `_links` -property will not fit into a single value of a table. - -## Summary -Now that you know how to extract sub-resources using child jobs, as well as resources composed directly of -properties (without an array of items), you probably think that the `_links` property, found all over the -MailChimp API and giving us a lot of trouble, is best to be ignored. The answer to this is -**mapping**, described in the tutorial's [next part](/extend/generic-extractor/tutorial/mapping/). - -You might also have noticed some duplicate records in the table `in.c-ge-tutorial.campaigns__campaign_id__content` -along the way. You'll look into this as well. +--- +title: Jobs Tutorial +permalink: /extend/generic-extractor/tutorial/jobs/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/jobs/ +--- diff --git a/extend/generic-extractor/tutorial/json.md b/extend/generic-extractor/tutorial/json.md index b477308b..bd97c1f8 100644 --- a/extend/generic-extractor/tutorial/json.md +++ b/extend/generic-extractor/tutorial/json.md @@ -1,147 +1,5 @@ ---- -title: JSON Introduction -permalink: /extend/generic-extractor/tutorial/json/ ---- - -* TOC -{:toc} - -[JSON (JavaScript Object Notation)](http://www.json.org/) is an easy-to-work-with format for describing structured -data. Before you start working with JSON, familiarize yourself with basic programming jargon. It is also recommended -to have a text editor with JSON support (you can also use an [online editor](http://www.jsoneditoronline.org/)). - -## Object Representation -To describe structured data, JSON uses **objects** and **arrays**. - -### Objects -Objects consist of **properties** and their **values**. Because the values in an object are identified by names -(property names), they are not kept in a particular order. - -The following object describes *John Doe* using two properties : `firstName` and `lastName`. - -{% highlight json %} -{ - "firstName": "John", - "lastName": "Doe" -} -{% endhighlight %} - -*Notice that the object is enclosed in `{}`. The properties and values are both in double quotes and are separated -by a colon. The individual properties are separated from each other using commas.* - -### Arrays -As objects collect named values, **arrays** are ordered lists of values that do not have a property -name but are identified by their numeric position. - -Let's go on to describing John Doe's family using an **array** (marked by `[]`) of three **objects**: - -{% highlight json %} -[ - { - "firstName": "John", - "lastName": "Doe", - "role": "father" - }, - { - "firstName": "Jenny", - "lastName": "Doe", - "role": "mother" - }, - { - "firstName": "Jimmy", - "lastName": "Doe", - "role": "son" - } -] -{% endhighlight %} - -*Objects are also separated from each other by commas. Notice that the last item (property or object) is not -followed by a comma.* - -### Terminology -The terminology varies a lot and other expressions are also commonly used: - -- Object --- also a record / structure / dictionary / hash table / keyed list / key value / associative array -- Property --- also a field / key / index -- Array --- also a collection / list / vector / ordinal array / sequence - -## Data Values -Each property value always has one of the following data types: - -- String --- text -- Number --- number -- Integer --- whole number (without decimal part) -- Boolean --- value which is either `true` or `false` -- Array --- ordered list of values -- Object --- collection of named values - -The types `string`, `number`, `integer` and `boolean` represent **scalar values** (simple). The types `array` and -`object` represent **structured values** (they are composed of other values). For example: - -{% highlight json %} -{ - "stringProperty": "someText", - "numberProperty": 12.45, - "integerProperty": 42, - "booleanProperty": false, - "arrayProperty": ["first", "second"], - "objectProperty": { - "name": "John", - "surname": "Doe" - } -} -{% endhighlight %} - -*Notice that only strings and property names are enclosed in double quotes. The boolean value is `false` without -double quotes because `false`, `true` and `null` (no or an unknown value) are **keywords**, not strings.* - -## References -There are multiple ways to refer to particular properties in a JSON document (for instance, [JSONPath](http://jsonpath.com/). -For the purpose of this documentation, we will use simple *dot notation*. Let's consider this JSON describing the -Doe's family: - -{% highlight json %} -{ - "address": { - "city": "Fresno", - "street": "Main Street" - }, - "members": [ - { - "firstName": "John", - "age": 42, - "shoeSize": 42.5, - "lastName": "Doe", - "interests": ["cars", "girls", "lego"], - "adult": true - }, - { - "firstName": "Jenny", - "adult": true, - "shoeSize": 24.5, - "lastName": "Doe", - "age": 42, - "interests": ["cars", "boys", "painting"] - }, - { - "adult": false, - "firstName": "Jimmy", - "lastName": "Doe", - "shoeSize": null, - "age": 1, - "interests": ["cars", "lego", "painting"] - } - ] -} -{% endhighlight %} - -To refer to John's city, we would write `address.city`. To refer to little Jimmy's shoe size, we -would write `members[2].shoeSize`. Array items indexes are *zero-based*, so the third item has -index `2`. - -The order of items in an object is not important. It is also worth noting that `[]` represents an empty array and -`{}` represents an empty object. - -## Summary -This page contains a little introduction to JSON documents. We intentionally avoided many details, -but you should now understand what JSON is, and how to write some stuff in it. +--- +title: JSON Introduction +permalink: /extend/generic-extractor/tutorial/json/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/json/ +--- diff --git a/extend/generic-extractor/tutorial/mailchimp-api-docs-1.png b/extend/generic-extractor/tutorial/mailchimp-api-docs-1.png deleted file mode 100644 index 8e61aaf8..00000000 Binary files a/extend/generic-extractor/tutorial/mailchimp-api-docs-1.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/mailchimp-api-docs-2.png b/extend/generic-extractor/tutorial/mailchimp-api-docs-2.png deleted file mode 100644 index 1ae66b62..00000000 Binary files a/extend/generic-extractor/tutorial/mailchimp-api-docs-2.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/mapping.md b/extend/generic-extractor/tutorial/mapping.md index d9d869ce..03c4b9f9 100644 --- a/extend/generic-extractor/tutorial/mapping.md +++ b/extend/generic-extractor/tutorial/mapping.md @@ -1,357 +1,5 @@ ---- -title: Mapping Tutorial -permalink: /extend/generic-extractor/tutorial/mapping/ ---- - -* TOC -{:toc} - -In the previous part of the tutorial, you [extracted the content of a MailChimp campaign](/extend/generic-extractor/tutorial/jobs/). -Now, it's time to clean up the response. - -This is the initial configuration: - -{: .image-popup} -![Whole cfg](/extend/generic-extractor/tutorial/mapping_all.png) - -**In JSON:** - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "offset", - "offsetParam": "offset", - "limitParam": "count", - "limit": 1 - } - }, - "config": { - "debug": true, - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13", - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns", - "children": [ - { - "endpoint": "campaigns/{campaign_id}/send-checklist", - "dataField": { - "path": "items", - "delimiter": "." - }, - "placeholders": { - "campaign_id": "id" - } - }, - { - "endpoint": "campaigns/{campaign_id}/content", - "dataField": { - "path": ".", - "delimiter": "." - }, - "placeholders": { - "campaign_id": "id" - } - } - ] - } - ] - } - } -} -{% endhighlight %} - -It extracts MailChimp campaigns with the `send-checklist` items and campaign `content`. -However, you are probably not interested in some parts of the content resource. Also, the table contains duplicates. - -***Technical note on duplicates:** If you examine the job events, you will see -the request `GET /3.0/campaigns/f7ed43aaea/content?count=1&offset=0` sent. That is, -pagination applies to **all API requests**. Generic Extractor tries to page the -unpaged `/content` resource. This may ultimately lead to duplicates because the extraction of that -resource is only terminated after the resource returns the same response twice.* - -## Mapping - -A mapping defines the shape of Generic Extractor outputs. It is stored -in the `config.mappings` property and is identified by the resource data type. -When a resource is assigned an internal `Result Name` (`dataType`), a mapping can be created -for it. To use a mapping, first define a `Result Name` (`dataType`) in the job property. - -### UI - -The mapping can be created in the `Mapping section` in the UI by clicking the `Create Mapping` toggle. - -{: .image-popup} -![Create mapping](/extend/generic-extractor/tutorial/create_mapping_toggle.png) - -You may generate the mapping automatically by clicking the**Infer Mapping** button in the top right corner. - -This operation will generate a mapping based on the sample response of the endpoint. - -{: .image-popup} -![Create mapping](/extend/generic-extractor/tutorial/create_mapping.png) - - -#### Primary key -To create a primary key, you can specify a `.` separated path of the elements in the response. ***Note:** If you are mapping child jobs, -the parent keys will automatically be included.* - -#### Nesting level -Currently, the automatic detection outputs only single table mapping. You can control the nesting level by specifying -the `Nesting Level` property. For example, a depth of 1 transforms `{"address": {"street": "Main", "details": {"postcode": "170 00"}}}` into two columns: -`address_street` and `address_details`. -All elements that have ambiguous types or are beyond the specified depth are stored in a single column as JSON, e.g., with the [`force_type`](https://developers.keboola.com/extend/generic-extractor/configuration/config/mappings/#mapping-without-processing) option. - -For example, if you click to generate mapping on the `Campaigns` endpoint with level 2 and primary key `id`, you will get this result -(note the link between the `Result Name` (`dataType`) and mappings key): - -``` -"mappings": {"campaigns": { - "id": { - "mapping": { - "destination": "id", - "primaryKey": true - } - }, - "web_id": "web_id", - "type": "type", - "create_time": "create_time", - "archive_url": "archive_url", - "long_archive_url": "long_archive_url", - "status": "status", - "emails_sent": "emails_sent", - "send_time": "send_time", - "content_type": "content_type", - "needs_block_refresh": "needs_block_refresh", - "resendable": "resendable", - "recipients.list_id": "recipients_list_id", - "recipients.list_is_active": "recipients_list_is_active", - "recipients.list_name": "recipients_list_name", - "recipients.segment_text": "recipients_segment_text", - "recipients.recipient_count": "recipients_recipient_count", - "settings.subject_line": "settings_subject_line", - "settings.title": "settings_title", - "settings.from_name": "settings_from_name", - "settings.reply_to": "settings_reply_to", - "settings.use_conversation": "settings_use_conversation", - "settings.to_name": "settings_to_name", - "settings.folder_id": "settings_folder_id", - "settings.authenticate": "settings_authenticate", - "settings.auto_footer": "settings_auto_footer", - "settings.inline_css": "settings_inline_css", - "settings.auto_tweet": "settings_auto_tweet", - "settings.fb_comments": "settings_fb_comments", - "settings.timewarp": "settings_timewarp", - "settings.template_id": "settings_template_id", - "settings.drag_and_drop": "settings_drag_and_drop", - "tracking.opens": "tracking_opens", - "tracking.html_clicks": "tracking_html_clicks", - "tracking.text_clicks": "tracking_text_clicks", - "tracking.goal_tracking": "tracking_goal_tracking", - "tracking.ecomm360": "tracking_ecomm360", - "tracking.google_analytics": "tracking_google_analytics", - "tracking.clicktale": "tracking_clicktale", - "delivery_status.enabled": "delivery_status_enabled", - "_links": { - "type": "column", - "mapping": { - "destination": "links" - }, - "forceType": true - } - -}} -``` - -### JSON - -The value of the `Result Name` (`dataType`) property is an arbitrary name. Apart from identifying -the resource type, it is also used as the **output table name**. If you run -the job, the content will be stored in `in.c-ge-tutorial.content`. - -Each mapping item is identified by the property name of the resource and must contain -`mapping.destination` with the target column name in the output table. For example: - -{% highlight json %} -"mappings": { - "content": { - "plain_text": { - "mapping": { - "destination": "text" - } - } -{% endhighlight %} - -The above mapping setting defines that the -resource property `plain_text` will be stored in the table column `text` for the `content` data type. No other -properties of the content resource will be imported. In other words, the mapping defines -all columns of the output table. - -To give an example, if you are interested in having the `plain_text` and `html` versions of the -campaign content, use a mapping like this: - -{% highlight json %} -"mappings": { - "content": { - "plain_text": { - "mapping": { - "destination": "text" - } - }, - "html": { - "mapping": { - "destination": "html" - } - } - } -} -{% endhighlight %} - -Note that the `destination` value is arbitrary but must be a valid column name. -The data type name (`content`) must match the value of the `dataType` property -as defined in some jobs. - -## Parent Reference -The above mapping works, but it is missing the campaign ID, and you cannot -match the content to some campaign records. Therefore, you must extract the campaign ID -from the context (i.e., the job parameter). This can be done using a special `user` mapping. - -When the mapping `type` is set to `user`, use the special prefix `parent_` to refer to -a `placeholder` defined in the job. You can create the following mapping: - -{% highlight json %} -"mappings": { - "content": { - "parent_id": { - "type": "user", - "mapping": { - "destination": "campaign_id" - } - } - } -} -{% endhighlight %} - -The above configuration defines a mapping for the `content` data type. -In the result table named `content`, the column `campaign_id` will be created. -Its content will be the value of the `id` placeholder -(`parent_id` minus the `parent_` prefix) in the respective job. - -Apart from specifying what columns should be in the output table, the -mapping allows you to set a column as part of a primary key. The entire configuration would -then look like this: - -{% highlight json %} -{ - "parameters": { - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "offset", - "offsetParam": "offset", - "limitParam": "count", - "limit": 10 - } - }, - "config": { - "debug": true, - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13", - "outputBucket": "ge-tutorial", - "jobs": [ - { - "endpoint": "campaigns", - "dataField": "campaigns", - "children": [ - { - "endpoint": "campaigns/{campaign_id}/send-checklist", - "dataField": "items", - "placeholders": { - "campaign_id": "id" - } - }, - { - "endpoint": "campaigns/{campaign_id}/content", - "dataField": ".", - "dataType": "content", - "placeholders": { - "campaign_id": "id" - } - } - ] - } - ], - "mappings": { - "content": { - "parent_id": { - "type": "user", - "mapping": { - "destination": "campaign_id", - "primaryKey": true - } - }, - "plain_text": { - "mapping": { - "destination": "text" - } - }, - "html": { - "mapping": { - "destination": "html" - } - } - } - } - } - } -} -{% endhighlight %} - -## Review -Now, let's review what parts are connected and how. Note that the values in blue -have been chosen arbitrarily when the configuration was created: - -{: .image-popup} -![Configuration Schema](/extend/generic-extractor/tutorial/configuration-schema.svg) - -## Summary -Mapping lets you precisely define what the extraction output will look like; it also -defines primary keys. - -If you do a one-time ad-hoc extraction, you may skip setting up the mapping and clean -the extracted data later in [Transformations](https://help.keboola.com/transformations/). -However, if you intend to use your configuration regularly or want to make it into a component, -setting up a mapping is recommended. - -## Tips and Tricks - -### Key Containing a Dot Character - -The key of the mapping supports dot notation to traverse into children. So, if the key contains a dot, you need to change the delimiter. See the following example: - -{% highlight json %} -"mappings": { - "content": { - "created.date": { - "delimiter": "/", - "type": "column", - "mapping": { - "destination": "createdDate" - } - } - } -} -{% endhighlight %} - -As you changed the delimiter from the default `.` to `/`, it's no longer parsed as two separate keys `created` and `date`, but rather just a single key `created.date`. +--- +title: Mapping Tutorial +permalink: /extend/generic-extractor/tutorial/mapping/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/mapping/ +--- diff --git a/extend/generic-extractor/tutorial/mapping_all.png b/extend/generic-extractor/tutorial/mapping_all.png deleted file mode 100644 index f7b207ed..00000000 Binary files a/extend/generic-extractor/tutorial/mapping_all.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/new_endpoint.png b/extend/generic-extractor/tutorial/new_endpoint.png deleted file mode 100644 index aeec1bfb..00000000 Binary files a/extend/generic-extractor/tutorial/new_endpoint.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/new_endpoint_modal.png b/extend/generic-extractor/tutorial/new_endpoint_modal.png deleted file mode 100644 index 6425aaa5..00000000 Binary files a/extend/generic-extractor/tutorial/new_endpoint_modal.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/pagination.md b/extend/generic-extractor/tutorial/pagination.md index 42359257..ffb67a16 100644 --- a/extend/generic-extractor/tutorial/pagination.md +++ b/extend/generic-extractor/tutorial/pagination.md @@ -1,169 +1,5 @@ ---- -title: Pagination Tutorial -permalink: /extend/generic-extractor/tutorial/pagination/ ---- - -* TOC -{:toc} - -Pagination breaks a result with a large number of items into separate pages and is used very commonly in -many API calls. - -In the previous part of the tutorial, you [fetched campaigns from -the MailChimp API](/extend/generic-extractor/tutorial/). If you created a new account, chances are that you probably have -only one campaign. You should now create some more campaigns (you do not have to configure them anyhow). - -If the API has consistent pagination for all resources (which the -[MailChimp API has](https://mailchimp.com/developer/guides/get-started-with-mailchimp-api-3/#Parameters)), -then the pagination is defined in the `Pagination` section of the endpoint configuration (or in the `api` section in the underlying JSON). - -## Preparation -The MailChimp API uses the [`offset` pagination method](https://mailchimp.com/developer/guides/get-started-with-mailchimp-api-3/#Parameters), -which means that each page has a fixed `limit` (by default 10 items), and you need to use the offset to move -that fixed-size page over the next set of results. For the first page, the `offset` is 0, for the second -page, the `offset` is 10. This is the same kind of pagination as in SQL. - -The offset pagination method is configured with the following basic properties: - -- `method` --- for MailChimp, set this property to `offset`. -- `offsetParam` --- name of the API parameter which defines the [page offset](https://mailchimp.com/developer/guides/get-started-with-mailchimp-api-3/#Parameters) -- `limitParam` -- name of the API parameters which define the [page size (limit)](https://mailchimp.com/developer/guides/get-started-with-mailchimp-api-3/#Parameters) - -So, for MailChimp, configure the pagination this way: - -- Click `Create New Pagination` in the Endpoint's Pagination section: - -{: .image-popup} -![Create pagination.png](/extend/generic-extractor/tutorial/img.png) - -- Name your pagination and select the Offset method: - -![Pagination](/extend/generic-extractor/tutorial/pagination.png) - -### JSON - -The resulting JSON configuration will look like this: - -{% highlight json %} -"api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "multiple", - "scrollers": { - "default": { - "method": "offset", - "limit": 100, - "limitParam": "count", - "offsetParam": "offset", - "firstPageParams": true, - "offsetFromJob": false - } - } - } -} - -{% endhighlight %} - -Alternatively, you can use a single pagination method instead of a scroller when configuring manually: - -{% highlight json %} -"api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "offset", - "offsetParam": "offset", - "limitParam": "count" - } -}, -{% endhighlight %} - -The entire Generic Extractor configuration will look like this: - -{% highlight json %} -{ - "api": { - "baseUrl": "https://us13.api.mailchimp.com/3.0/", - "authentication": { - "type": "basic" - }, - "pagination": { - "method": "multiple", - "scrollers": { - "default": { - "method": "offset", - "limit": 100, - "limitParam": "count", - "offsetParam": "offset", - "firstPageParams": true, - "offsetFromJob": false - } - } - } - }, - "config": { - "outputBucket": "ge-tutorial", - "incrementalOutput": false, - "jobs": [ - { - "__NAME": "campaigns", - "endpoint": "campaigns", - "method": "GET", - "dataType": "campaigns", - "dataField": { - "path": ".", - "delimiter": "." - } - } - ], - "__AUTH_METHOD": "basic", - "username": "dummy", - "#password": "c40xxxxxxxxxxxxxxxxxxxxxxxxxxxxx-us13" - } -} -{% endhighlight %} - -***Note:** The `__` prefixed parameters are for internal use by the UI and should not be modified. -Also, they have no effect on component functionality.* - -## Running - -Now, make sure that you have more than one campaign in your account. - -## Testing -Because you probably have fewer than ten (the default page size) campaigns in your MailChimp account, -there is no way to tell whether the pagination works. Let's make sure by setting the `limit` -to 1 and turning the `debug` mode on so that you can see all the requests sent by Generic Extractor. - -Run the configuration and review the events produced by the job. You should see something like this: - -{: .image-popup} -![Screenshot - Debug Events](/extend/generic-extractor/tutorial/job-2.png) - -The oldest events are at the bottom, so you can see that the extractor started by sending an HTTP request: - - GET /3.0/campaigns/?count=1&offset=0 - -Then, it continued with - - GET /3.0/campaigns/?count=1&offset=1 - GET /3.0/campaigns/?count=1&offset=2 - -and so on. You should also see a warning that the `dataField 'campaigns' contains no data`. -This is expected because Generic Extractor tries bigger offsets until the number of returned items is -less than the page size. With the page size set to 1, this means that the last page will contain no data. - -## Summary -In this part of the tutorial, you learned how to set up simple pagination. This is very important -because most APIs use some sort of pagination and without proper setting you would be -getting incomplete data. The next two parts of our tutorial deal with setting up jobs and mapping: - -- [Jobs](/extend/generic-extractor/tutorial/jobs/) --- describe the API endpoints - (resources) to be extracted. -- [Mapping](/extend/generic-extractor/tutorial/mapping/) --- describes how the JSON - response is converted into CSV files that will be imported into Storage. +--- +title: Pagination Tutorial +permalink: /extend/generic-extractor/tutorial/pagination/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/pagination/ +--- diff --git a/extend/generic-extractor/tutorial/pagination.png b/extend/generic-extractor/tutorial/pagination.png deleted file mode 100644 index 3b892a17..00000000 Binary files a/extend/generic-extractor/tutorial/pagination.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/rest.md b/extend/generic-extractor/tutorial/rest.md index 20f1409e..04efb3d6 100644 --- a/extend/generic-extractor/tutorial/rest.md +++ b/extend/generic-extractor/tutorial/rest.md @@ -1,136 +1,5 @@ ---- -title: REST HTTP API Introduction -permalink: /extend/generic-extractor/tutorial/rest/ ---- - -* TOC -{:toc} - -An [API (Application Programming Interface)](https://en.wikipedia.org/wiki/Application_programming_interface) is -an [interface](https://en.wikipedia.org/wiki/Interface_(computing)) to an application, or a **service** -designed for machine access. It can be seen as the UI (User Interface) of an application designed -for machines (other applications). - -So that another application can be programmed to consume the API, it has to have some sort of specification. -A common specification for communicating on the web is the [HTTP protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol). -Used by web browsers and other API clients, it defines how two parties (client and server) ought to communicate: - -- Client creates an HTTP **request** and sends it to the server over the network. -- Server processes the request, creates a **response**, and sends it to the client over the network. - -## HTTP Request -An HTTP request is composed of: - -- URL -- HTTP Method -- HTTP Headers -- Optional Body - -### URL -A [URL (Uniform Resource Locator)](https://en.wikipedia.org/wiki/URL) is the address you see in your web browser -address bar. It allows you to **locate a resource**. Each URL has several parts, and it is important to know them. -For example, the address - - https://www.example.com:8080/customers/acme/order/?show=deleted&fields=all - -is composed of: - -- `https` --- **Protocol** (HTTP or HTTPS), -- `www.example.com` --- **Host** --- network address of the HTTP server, -- `8080` --- **port** --- Optional network identifier within the target server; its default value is `80`. -- `/customers/acme/order/` --- Optional **path** to a **resource** we wish to obtain; its default value is `\`. -- `show=deleted&fields=all` --- Optional **request parameters** (also called **query string** or **query -string parameters**), separated by the character `&` (ampersand); the actual parameters are: - - `show` with the value `deleted`, and - - `fields` with the value `all`. - -Because the URL contains a number of special characters (`?`, `&`, `/` and many others), when these parameters -need to be part of the URL, they must be encoded (URL encoded, urlencoded, escaped). Therefore a URL: - - http://example.com/this address & special - -will be actually sent to the server as: - - http%3A%2F%2Fexample.com%2Fthis+address+%26+special - -The web browser (and Generic Extractor too) will normally do this conversion for you. However, you might run into -the encoded format in Generic Extractor events. There are plenty of [online tools to decode](https://urldecode.org/) -this encoded format. - -Sometimes, you may also encounter the term [URI (Uniform Resource Identifier)](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier). -It is used when a single **Resource** may be accessed through multiple URLs. For example, the web page -`http://example.com` may display the same content as `http://example.com/home`. In such case one of the URLs -(probably the second one) is chosen as an identifier, and becomes URI. For our use, there is no important -difference between URI and URL. - -An API **end-point** is identified by its URL, or URI, and should represent a distinct **resource** (users, -invoices etc.). ***Important:** The terms end-point, resource, URL and URI are used interchangeably throughout the -tutorial because they ultimately refer to the same thing.* - -### Method -An HTTP **Method** describes a type of the request to make. It also called an **HTTP Verb** because it -describes what to do with the **resource**. Common HTTP verbs are: - -- `GET` --- Obtain a resource. -- `POST` and `PATCH` --- Update a resource. -- `PUT` --- Create a resource. -- `DELETE` --- Delete a resource. - -Since Generic Extractor only reads data from another API, you will mostly use the `GET` method (and sometimes the -`POST` method). The other HTTP methods are not important for us. - -### Headers -An HTTP request can contain [**headers**](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Request_Headers), -which include additional information about the request and response. A typical example of a header is -`Content-type`. For instance, for a web page, `Content-Type: text/html` would be used because an -[HTML page](https://en.wikipedia.org/wiki/HTML) is being transferred. For an API request, it is commonly set -to `Content-type: application/json` because we are transferring [JSON data](http://www.json.org/). - -Apart from standard headers, there are also non-standard headers; these are marked with the prefix `X-`. An -example is the `X-StorageAPIToken` header used with Keboola [Storage API](/integrate/storage/api/). - -### Body -The `POST`, `PUT` and `PATCH` requests can send parameters the same way as the `GET` requests in the URL. -But they can also send them in the request **body**. These are sometimes called **POST data/postdata**. - -## HTTP Response -An HTTP response is composed of: - -- Response Headers --- same as the request headers (only sent by the server) -- Response Body --- actual content of the resource -- Status Code --- status of the request - -#### HTTP Status -The HTTP Status and [status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) represent -a standardized way of describing the response state. For example, the status `200 OK` (200 is the status code) -is associated with a successful response. There are many HTTP Statuses, but the following rules apply: - -- Status codes `2xx` (e.g., 200) represent success. -- Status codes `3xx` represent [redirection](https://en.wikipedia.org/wiki/URL_redirection). -- Status codes `4xx` represent a client error (the request is wrong). -- Status codes `5xx` represent a server error (the server failed to create the response). - -## REST API -[REST (Representational state transfer)](https://www.restapitutorial.com/lessons/whatisrest.html) (or RESTful) -is an API which follows a set of [loosely defined](http://restcookbook.com/Miscellaneous/rest-and-http/) principles: - -- The API URLs (or URIs) represent individual **resources**. Each API endpoint should represent a resource of -a *single type*. For example, it represents a list of users, and not a list of users and their invoices. -- Each resource is **represented** in a structured format ([JSON](http://www.json.org/) or -[XML](https://en.wikipedia.org/wiki/XML)). The data is not transferred, for instance, as ordinary text or a web -page. -- **Messages** (request and response) are transferred using various HTTP methods (`GET`, `POST`, etc.). -For example, for obtaining data, the `GET` method should be used. Also the `GET` method -should not cause any modifications of data. -- The entire communication is **stateless**. This means that multiple requests can be called in an -arbitrary order and must yield the same results. It is not correct for an API to have endpoints such as -`setFilter` and`getFilteredResult` because they imply that any state (a filter) is retained between those API -endpoints. - -## Summary -The above describes the basic concepts of an API, HTTP protocol and HTTP REST API. When you -understand these concepts (and the associated jargon), you can use Generic Extractor -to get responses from virtually any HTTP REST API. Since the REST rules are not rigidly specified, it -is not possible to ensure that Generic Extractor will be capable of reading 100% of APIs, -even when declared as RESTful by someone. - +--- +title: REST HTTP API Introduction +permalink: /extend/generic-extractor/tutorial/rest/ +redirect_to: https://help.keboola.com/components/extractors/generic-extractor/tutorial/rest/ +--- diff --git a/extend/generic-extractor/tutorial/sub-resources-docs.png b/extend/generic-extractor/tutorial/sub-resources-docs.png deleted file mode 100644 index 2ec67618..00000000 Binary files a/extend/generic-extractor/tutorial/sub-resources-docs.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/table-campaigns-sample.png b/extend/generic-extractor/tutorial/table-campaigns-sample.png deleted file mode 100644 index fd906395..00000000 Binary files a/extend/generic-extractor/tutorial/table-campaigns-sample.png and /dev/null differ diff --git a/extend/generic-extractor/tutorial/test_endpoint.png b/extend/generic-extractor/tutorial/test_endpoint.png deleted file mode 100644 index 64cd4388..00000000 Binary files a/extend/generic-extractor/tutorial/test_endpoint.png and /dev/null differ diff --git a/extend/generic-extractor/ui.png b/extend/generic-extractor/ui.png deleted file mode 100644 index 6cb30b30..00000000 Binary files a/extend/generic-extractor/ui.png and /dev/null differ diff --git a/extend/index.md b/extend/index.md index 5c005ab8..ab89d563 100644 --- a/extend/index.md +++ b/extend/index.md @@ -55,12 +55,12 @@ They can be implemented in virtually any programming language and be fully custo They also support OAuth authorization. To get started with building a component, see our [**tutorial**](https://help.keboola.com/extend/component/tutorial/). ## Generic Extractor -[Generic Extractor](/extend/generic-extractor/) is a Keboola component acting like a -customizable [HTTP REST client](/extend/generic-extractor/tutorial/rest/). It can be configured to extract data +[Generic Extractor](https://help.keboola.com/components/extractors/generic-extractor/) is a Keboola component acting like a +customizable [HTTP REST client](https://help.keboola.com/components/extractors/generic-extractor/tutorial/rest/). It can be configured to extract data from virtually any API and offers a vast amount of configuration options. With Generic Extractor, you can build an entirely new extractor for Keboola in less than an hour. -Components based on Generic Extractor are built using [JSON configuration](/extend/generic-extractor/tutorial/) and a -[published template](/extend/generic-extractor/publish/). They have a predefined UI, require no knowledge of Docker or +Components based on Generic Extractor are built using [JSON configuration](https://help.keboola.com/components/extractors/generic-extractor/tutorial/) and a +[published template](https://help.keboola.com/components/extractors/generic-extractor/publish/). They have a predefined UI, require no knowledge of Docker or other tools, and they use a Keboola owned [repository](https://github.com/keboola/kbc-ui-templates/). To get -started with Generic Extractor, see our [**tutorial**](/extend/generic-extractor/tutorial/). +started with Generic Extractor, see our [**tutorial**](https://help.keboola.com/components/extractors/generic-extractor/tutorial/). diff --git a/index.md b/index.md index e2267094..f6ed3864 100644 --- a/index.md +++ b/index.md @@ -27,7 +27,7 @@ In this documentation, we will show you how to - Leverage natural language to build data pipelines and interact with your Keboola project using the [Model Context Protocol (MCP)](https://help.keboola.com/ai/mcp-server/). - [**Extend Keboola by building your own components**](/extend/) for your own use or for other Keboola users and customers. - [Extend Keboola with arbitrary Docker images](https://help.keboola.com/extend/component/). - - Build your own [extractors](/extend/generic-extractor/) for services we do not support yet. + - Build your own [extractors](https://help.keboola.com/components/extractors/generic-extractor/) for services we do not support yet. - [**Automate your processes**](/automate/) to run any component in specified intervals or at specified times of the day. - Control any component of Keboola [programmatically](/integrate/jobs/) (for example, you can trigger data load when something happens in your system).