Skip to content

Support pass command as args in k8s executor - #10481

Closed
pingzh wants to merge 1 commit into
apache:masterfrom
pingzh:support-override_container_entrypoint-in-k8s
Closed

Support pass command as args in k8s executor#10481
pingzh wants to merge 1 commit into
apache:masterfrom
pingzh:support-override_container_entrypoint-in-k8s

Conversation

@pingzh

@pingzh pingzh commented Aug 22, 2020

Copy link
Copy Markdown
Contributor

The current implementation overrides the image's entrypoint, which is less ideal. Users may want to use their own entrypoint to do additional setup and clean up, before and after the airflow tasks.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg Bot added area:Scheduler including HA (high availability) scheduler k8s labels Aug 22, 2020
@pingzh

pingzh commented Aug 24, 2020

Copy link
Copy Markdown
Contributor Author

@dimberman could you please take a look? thanks

@dimberman

Copy link
Copy Markdown
Contributor

Hi ping, going forward we're actually going to be deleting all of the k8sexecutor airflow.cfg configs involved in pod creation. This is because as of 1.10.12 we allow users to generate pod_templates using yaml. Please see the PR here for simplification of k8sexecutor #10393

@dimberman dimberman closed this Aug 24, 2020
@pingzh

pingzh commented Aug 25, 2020

Copy link
Copy Markdown
Contributor Author

Hi ping, going forward we're actually going to be deleting all of the k8sexecutor airflow.cfg configs involved in pod creation. This is because as of 1.10.12 we allow users to generate pod_templates using yaml. Please see the PR here for simplification of k8sexecutor #10393

Thanks, will keep this in mind.

@dszakallas

dszakallas commented Sep 14, 2020

Copy link
Copy Markdown

@dimberman the command and args are dynamic, i.e. they could be different for each run, so the only way to customize this behavior from a pod template is to interpolate these parameters using a jinja template during runtime. I don't know if it is possible currently, but would cover more use cases more cleanly than this approach. E.g. a common use case not covered by this when the entrypoint relays the args to airflow but sets up some environment variables first, in which case only airflow's arguments should be passed to the script.
If we had two separate variables for the command (['airflow']) and args (['run', 'tasks', 'abc', '...']) interpolated into the pod template, we could support that behavior too, by using a static command in the pod spec, and using the dynamic args.

@kaxil kaxil added provider:cncf-kubernetes Kubernetes (k8s) provider related issues and removed area:k8s labels Nov 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants