Since Spark 2.4.6 a passed exception can be a wrapper on top of an Error.
https://issues.apache.org/jira/browse/SPARK-31144
Problem
In order to maintain consistent behavior of the QueryExecutionListener.onFailure() method in different Spark versions, and to avoid unintentional fatal errors processing additional check should be placed in all custom listener implementations.
Solution
Create an adapter trait that can be mixed-in with a QueryExecutionListener filtering out calls that carry wrapped fatal exceptions (either directly or wrapped ones).
Since Spark 2.4.6 a passed exception can be a wrapper on top of an Error.
https://issues.apache.org/jira/browse/SPARK-31144
Problem
In order to maintain consistent behavior of the
QueryExecutionListener.onFailure()method in different Spark versions, and to avoid unintentional fatal errors processing additional check should be placed in all custom listener implementations.Solution
Create an adapter trait that can be mixed-in with a
QueryExecutionListenerfiltering out calls that carry wrapped fatal exceptions (either directly or wrapped ones).