How to get the actual duration of the job，the appLaunchedTime have value while the job is still in waiting.

[hellocandy]

Organization Name: Advantech

Short summary about the issue/question: We want to calculate the actual duration time of a job, from the start of the job to the end of the job. We need to exclude the waiting time. At first we use the createdTime, and later we changed to use the appLaunchedTime. But when the job is still in waiting status, the appLaunchedTime appears.

Brief what process you are following:

How to reproduce it:

OpenPAI Environment: Cluster

• OpenPAI version: 0.10.1
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

[Binyang2014]

@yqwang-ms rest-server retrieve the status and appLaunchedTime from launcher, can you take a look?

[yqwang-ms]

@xianfeng appLaunched just means the app is launched, but it may still wait for resources to really run.
We have not yet record the start timestamp for running state, we may support it in future.
Sorry for this inconvenience.

[yqwang-ms]

@hellocandy Your feature request is just supported in frameworkcontroller (so will be supported in Pure K8S PAI soon).

For more details, please check:
microsoft/frameworkcontroller#35

For a job attempt:
WholeDuration = CompletionTime - StartTime
RunningDuration = CompletionTime - RunTime
WaitingDuration = WholeDuration - RunningDuration

For the whole job which may have multiple attempts:
WholeDuration = CompletionTime - StartTime
RunningTime = Sum_all_attempts (CompletionTime - RunTime)
WaitingDuration = WholeDuration - RunningDuration

[yqwang-ms]

@abuccts Could you please help to expose the RunTime field through PAI RestServer (and WebPortal)?