[Feature Request] Make dotnet-trace trace from startup more useable for applications that have output, or are interactive

[davidwrighton]

In the current implementation, the current dotnet-trace collect -- hides all details of the executed application from the user of dotnet-trace. This is problematic in several ways, and while the --diagnostic-port switch can be used to workaround this limitation, I believe the current model is much less useable than it could be.

Background and Motivation

I am researching ways to build a profile guided optimization experience based on trace data. The basic approach is to build an application, measure and run a training scenario, and then rebuild the application with the training data. To make this approach efficient to work on, I need to build a set of scripts that will perform the various build and analysis operations. I've reached for the dotnet-trace tool as part of this workflow, in order to capture information about the training process.

I ran into several issues with using dotnet-trace as a tool in this process.

1. Documentation for the -- switch is only present in docs.microsoft.com, and is not present from the application command line help.
2. When I attempted to run my training application, I was unable to tell that it was actually crashing every time it ran it without quite a lot of effort. (I needed to use child process debugging with windbg to find this). While the actual crash was not the fault of dotnet-trace, the UX of dotnet-trace for child process work made it impractical to diagnose this issue efficiently.
3. Furthermore, when I attempted to use this tool on an ASP.NET process, I was unable to, as the tool captured the Ctrl-C handler which I intended to use to extra the ASP.NET process in a clean fashion. (This is less of a concern, now that I am aware of the --diagnostic-port switch.)

In addition to the above, the expected use of the profile guided optimization feature is that application developers will build a CI system uses dotnet-trace in the midst of every build to profile the product as part of their build process.

The suggested approach will look like the following...

1. Build the product with dotnet build
2. Run the newly built product under a series of training scenarios. These executions would be done via something like dotnet-trace --.

• For console applications this is a very simple process
• For ASP.NET applications, this is more complicated. I suspect we could probably structure the tests to be unittests runnable from the command line, but this may be an unreasonable expectation.

3. After generating traces, take them and feed them through the dotnet-pgo tool to produce a set of instrumentation data.
4. Rebuild the product with a special build that uses the instrumentation data.

In these cases, it is important that the training scenario test case output and behavior are straightforward to dump into the output.

Proposed Feature

Add a switch to dotnet-trace which can be used in combination with -- to generate a full set of output (and possibly) input from the child process. This is a better approach for scripting and some interactive uses as it allows the process to avoid the need for multiple windows, and complex parallel process orchestration. In addition to not hiding the output from the process, it would reduce/eliminate the periodic progress reports from the tool.

• My proposed name for this switch is --redirect-child-output:false

In addition, the exit code from the process is valuable, and should become the exit code from the dotnet-trace tool. This allows scripts to easily detect failure, and mark a CI leg as failed.

Usage Examples

See Background...

[davidwrighton]

I built an initial implementation of this with one of my earlier attempted changes for pr #1945, but as this feature is not universally agreed upon, that PR has been reduced to just fixing the bug I found.

[davidwrighton]

@josalem also expressed interest here. I'd like to hear his opinion on how best to address these issues.

[noahfalk]

Thanks David! Adding some configurability on this seems fine in principle and not unexpected based on earlier discussion

I’d rather case 1 be as simple as possible and we leave case 2 for anyone with more advanced requirements. If we get feedback that case 1 needs more options or different behavior, then we can course correct at that point 😄

In terms of exactly what to add a few possibilities off the cuff:

1. a boolean switch that controls output redirection only
2. a boolean switch that controls input and output redirection
3. separate switches for input/output
4. multi-valued switch that could toggle between the console, a file path or ignored?

(2) seems like the most intuitive to me, glad to get other opinions : ) In terms of names maybe "--show-child-io" or "--show-child-output"? Even if we used the name 'output' I am still thinking it would apply to input and output, but output might be more immediately relatable term than IO? The redirect terminology with a false argument felt a little like a double negative to me, ie "don't not show me the output" but I'm just one opinion. If others think it is clearer I don't have any strong objection to it.

it would reduce/eliminate the periodic progress reports from the tool.

I'd be in favor of the 'eliminate' option. I would strive to have IO connected to a single process at a time and make it clear for the user which process that is. My first thought would be let dotnet-trace print its opening diagnostic info about which providers are enabled and where the trace is going, then some message that says launching "fooo.exe bar bla command line", all the child IO after that, and then some closing message from dotnet-trace saying the child process exited with whatever the exit code was.

I assume dotnet-trace would stop the trace and exit based on whenever the child process exits and there is no option to stop the trace early or disconnect? Not suggesting we should have such an option, just confirming everyone is thinking the same thing.

[ayeltsov]

My scenario:

• I didn't want to miss early events in process startup => so attaching to PID is not a good option. It's a bit messy too in terms of manual steps and coordination. I would rather start collection, then process.
• My process's console is interactive => so using the "--" to have dotnet-trace start the process is not viable

So far usability is a step back compared to fusion logging in .net FW where one can set up logging, then do things related to process.

I'd love two things from usability perspective:

• have an option to leave console to the child process, for both input and output
• have ability to set up tracing before the app is ran (without changes or extra parameters - these things are often unavailable, for example when user does not have direct control on how exactly the app is started)
  • ability to filter by process name/path is a big bonus in this case

To pitch for the 2nd point above - I have a case right now where assembly resolution is doing something weird on customer's machine, I can't repro. I would love to give them very simple instructions to capture the fusion log, something like "run this thing; then run the app as usual; then send me this file". For them hunting for PIDs is not a friendly option.

[davidwrighton]

@ayeltsov I think your request here is for a somewhat different, but related feature. In particular if you don't believe that the -- option can be used, there isn't a child process at all in this situation. I think your request is for a model more similar to the --diagnostic-port option, where a parallel process can be listened to by a listener process. Unfortunately, my understanding is that dotnet-trace can only listen to a single process at a time, and even that requires a managing an environment variable. OTOH, while dotnet-trace does not support that model at this time, the model IS supported by the eventing infrastructure of CoreCLR. To access that feature you will need to use with ETW based collection(on Windows) or LTTNG collection (on Linux). These collection techniques capture operations from all processes on the server, and produce a data file that the trace examination tools can use. Offhand, I don't know where the documentation is for how to use LTTNG for coreclr events, but for ETW collection, I personally tend to use the collect feature of PerfView which can be found at https://github.com/microsoft/perfview . For use on customer machines, you may prefer using a tool such as xperf to capture the data, or even writing a bespoke tool to access the Windows apis for controlling etw events (if you need a really polished solution for your customers).

[davidwrighton]

@noahfalk I like option #2 (which is basically I implemented as a first stab at this...) Your suggestion of --show-child-io is my favorite of the names for this options I've seen so far.

A Launching "fooo.exe bar balh command line message seems good, and if we don't already have that printing out by default, I think we should do it unconditionally.

I agree, dotnet-trace would stop the trace and exit when the application exits; however, there is already an option to stop after X seconds, and I believe that option should remain. It should use the existing process kill mechanism to kill the process after the time command ends. I do not believe that any I/O should be allowed to stop tracing. (This includes Ctrl-C).

Printing out the return code would be a nice touch in all scenarios, whether or not we show/hide the I/O. We also need to return the error code of the launched process from the dotnet-trace tool to make these unattended uses of dotnet-trace easier to manage.

Elimination of progress reports. Yes. I think elimination is a fine choice. It was my preference, but it isn't strictly necessary for the scenarios I'm working with.

[noahfalk]

@ayeltsov - thanks for the feedback! Do you need tracing support for multiple processes or just a single one is sufficient? If it is just a single one I think this workflow would work for you now:

In one console or running as a background process:

dotnet-trace collect --diagnostic-port /foo/bar.socket --providers ...

In a second console or running in the foreground:

export DOTNET_DiagnosticPorts=/foo/bar.socket
// run the dotnet process as usual

The diagnostic port environment variable tells the newly spawned process to connect via the port as soon as it launches and from there dotnet-trace will enable the providers you requested and capture the trace.

If you need support for multiple processes David's suggestion of using ETW or Lttng could work. Alternately I think we should open a separate feature request for dotnet-trace to support multiple-process tracing. That isn't something it can do today and adding that capability would be out-of-scope for this current request. Hope that helps!

[noahfalk]

A Launching "fooo.exe bar balh command line message seems good, and if we don't already have that printing out by default, I think we should do it unconditionally.

Unconditional sounds good to me

however, there is already an option to stop after X seconds, and I believe that option should remain. It should use the existing process kill mechanism to kill the process after the time command ends. I do not believe that any I/O should be allowed to stop tracing. (This includes Ctrl-C).

Agreed. Sorry, when I said there would be no way to stop early I was saying it imprecisely : ) My intent was what you wrote here - there would be no I/O that stops tracing on the fly.

Printing out the return code would be a nice touch in all scenarios, whether or not we show/hide the I/O. We also need to return the error code of the launched process from the dotnet-trace tool to make these unattended uses of dotnet-trace easier to manage.

Sounds good to me.

[ayeltsov]

Thanks, and I apologize for taking this a bit off original topic. The original request to be able to interact with console is very valid and I'm +1 on that.

@noahfalk , your suggestion will definitely work in my current case.
I think what's missing in general (and again, this is out of scope for this request) is how to deal with tracing when user does not have easy ability to inject DOTNET_DiagnosticPorts variable just before executing the target process. For example, it could be started by something that user has no easy control of (another application, a service, etc)

[noahfalk]

Oh no worries @ayeltsov, that happens all the time and its no problem at all : ) I'd much rather get the feedback and we find the right issue for it than not hear from you. I opened #1958 which I think matches what you were interested in? We can always adjust the title or summary text if I didn't get it right. Thanks!