Detect Spark driver completion by container state when tracking via k8s API - #68048
Detect Spark driver completion by container state when tracking via k8s API#68048karenbraganz wants to merge 12 commits into
Conversation
|
I still need to test this out and write unit tests. |
|
This has passed all unit tests as well as a manual test that I ran. |
| for container in pod.spec.containers: | ||
| if "spark" in container.name.lower() or "driver" in container.name.lower(): | ||
| driver_container = container | ||
| break | ||
| if len(pod.spec.containers) == 1: | ||
| driver_container = container |
There was a problem hiding this comment.
Say with a pod whose containers are [spark-metrics-exporter, spark-kubernetes-driver], where the exporter has terminated 0 and the driver is still Running. The loop matches the exporter, sets container_completed=True, and returns Succeeded. The task is now terminal, but the Spark application is still executing. The lookup loop needs more work.
There was a problem hiding this comment.
Also, we should probably make the names configurable as suggested in the original issue.
| terminal_phase = phase | ||
| break | ||
| if phase == "Failed": | ||
| if phase == "Failed" and not container_completed: |
There was a problem hiding this comment.
The added clause means a pod in Failed whose driver exited 0 (e.g. an Istio sidecar crashed) now falls through to line 1277 and returns Succeeded. We should at least log a warning.
track_driver_via_k8s_api, detect driver completion by container status rather than pod phase #67934This PR tracks Spark job completion by container state instead of pod phase when
track_driver_via_k8s_api=True. Sometimes the pod continues to run even after the driver container completes due to other sidecar containers. This PR makes driver completion detection more accurate by examining the container itself.