[Feature][CustomizePlugin] Data extraction 'rawDataParams' does not support wild cards

[riskpeep]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Use case

As a DevOps engineer, I would like to be able to easily configure data extraction into custom fields using the Customize plugin data extraction feature so I can add custom data fields.

Description

The Customize plugin supports the extraction of data from raw data using a data extraction feature. The current implementation allows configuration of data extraction using an array of transformation rules that look like this:

"transformationRules":[
          {
            "table":"issues", 
            "rawDataTable":"_raw_jira_api_issues", 
            "rawDataParams":"{\"ConnectionId\":1,\"BoardId\":8}", 
            "mapping":{
              "x_test":"fields.status.name" 
            }
          }
        ]

In a transformation rule the rawDataParams field is used as a filter on the source data to identify source data to pull from. Currently the rawDataParams field requires an exact match on the source data. This means that DevOps engineers must configure a transformation rule for EACH board from which data should be extracted. For even moderately complex devlake installations, this can mean 10s of transformation rules to capture the issues associated with multiple boards. This also means that if additional sources are configured, the customize plugin must also be configured to include the new source board id.

It would be much more desirable if the 'rawDataParams' supported wildcards or other mechanisms to allow one transformation rule to match on more than one board.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct