<LMON BE API> (ERROR): read_lmonp return a neg value

[antonl321]

Hi,

I did some tests with the latest stat-cl.

It works file on runs with 45 nodes x 16 MPI
but it fails with the error from bellow on a run with 880 nodes x 8 MPI

[atosla@ac1-1001 ~]$ stat-cl  `pgrep -o srun` 2>~/stat-cl.err
STAT started at 2022-07-08-13:44:15
Attaching to job launcher (null):690121 and launching tool daemons...
<Jul 08 13:44:15> <Launchmon> (INFO): Just continued the RM process out of the first trap
<Jul 08 13:44:18> <LMON BE API> (ERROR): read_lmonp return a neg value

In stat-cl.err I see the following

graph is not connected, found 0 potential roots
<Jul 08 13:44:18> <STAT_FrontEnd.C: 642> STAT returned error type STAT_MRNET_ERROR: MRNet reported a Network error with message: MRNet: Topology error: file has incorrect format
<Jul 08 13:44:18> <STAT.C: 181> STAT returned error type STAT_MRNET_ERROR: Failed to launch MRNet tree()
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-2041.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-2041.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-1087.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-1087.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac1-4064.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac1-4064.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-2012.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-2012.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac1-4008.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac1-4008.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-2013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-2013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-3013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-3013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
<Jul 08 13:44:18> <STAT_lmonBackEnd.C:146> ac3-2033.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
<Jul 08 13:44:18> <STATD.C:230> ac3-2033.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
...

[lee218llnl]

Can you attach the .top file in your stat_results directory?

[antonl321]

I got the following, obviously an incomplete list of hosts

$ cat ~/stat_results/ifsMASTER.sp.x.0016/*top
ac1-1001.bullx:0 =>
	ac1-1001:0
	ac1-1002:0
	ac1-1003:0
	ac1-1006:0
	ac1-1007:0
	ac1-1008:0
	ac1-1009:0
	ac1-1012:0
	ac1-1013:0
	ac1-1014:0
	ac1-1016:0
	ac1-1017:0
	ac1-1018:0
	ac1-1019:0
	ac1-1020:0
	ac1-1021:0
	ac1-1022:0
	ac1-1023:0
	ac1-1030:0
	ac1-1031:0
	ac1-1032:0
	ac1-1033:0
	ac1-1034:0
	ac1-1035:0
	ac1-1036:0
	ac1-1037:0
	ac1-1038:0
	ac1-1039:0
	ac1-1040:0
	ac1-1041:0;

[lee218llnl]

as a workaround, you can set the topology to "depth" with a value of "1". This will connect the STAT FE directly to all the STAT BEs. The topology file is actually just MRNet communication processes that STAT uses to make it's communication more scalable. The number of hosts you see in the topology file is probably the square root of the number of nodes of your job.

That aside, I will have to do some debugging to see why it doesn't like the topology file. It might not like that your hostnames have "-" characters in it.

Anyway, let me know if the depth 1 workaround suffices until I can fix this.

[lee218llnl]

I'm actually thinking that MRNet does not like the fact that there is the ".bullx" in your hostname. It may be the case that when it sees ".something" it thinks "something" is your domain name and then tries to resolve the network address using that. I'm guessing that in your case, ".bullx" is not the domain, but rather just part of the node name. Is this accurate?

[antonl321]

I think so. We have discussed this a while back in #33 (comment)

[antonl321]

depth 1 option works (no crash) but the stat-cl stays for ever in "Sampling traces..." when I try to sample a 880 nodes x 8 MPI/node job.
I have a suggestion here, if it were easy to implement it would be useful to see the progress of sampling every second or so. In this way one can decide quickly if it worth to wait or to cancel

[lee218llnl]

how long did you wait? It could take a few minutes when running with this flat topology rather than using a scalable tree.

Regarding progress sampling, there is no easy way to poll the daemons for progress, so unfortunately you suggestion cannot be implemented.

Ultimately, I'm hoping that we can get a fix in MRNet so you can use a scalable tree and then in theory the operation should be fairly quick.

[antonl321]

Up to 30 minutes or so.
If I use a very simple test code (a few lines) I can get a stack trace in a couple of minutes even at a larger scale.
As I said in my previous message stat-cl gets stuck in "Sampling traces...". The launch of the monitors goes ok. The app I'm interested in is rather complex with deep stack trace. I wonder if some timeouts block the whole process.

[lee218llnl]

can you try running the same complex app with the deep stack at a smaller scale? One potential issue is contention on the file system since all the STATD's will be trying to parse you executable's debug information.

[antonl321]

I have run it with 300 nodes, but still it takes a long time. I think that you are right about STATD being stuck waiting for the debug info from the file system. I looked on one node with bpf profile and offcpu tools and most of the time STATD is off cpu when stack traces are being collected. Also la large number of lustre rpc are around. Can this be improved? I need to think and ask around.

[antonl321]

I have run stac-cl with -l BE -l SW -l CP on the case that hangs ( 880 nodes x 8 MPI/node)
I have noticed that after some progress reported all log files stop being updated at about the same time (to the second).
The last line in *.STATD.log is
<STAT_FilterDefinitions.C:661> File request down stream filter end.
EXCEPT for one file which has
<STAT_BackEnd.C:1268> sending ackowledgement with tag 103 (first = 100) and return value 0

I wonder if a hang appears in STAT. Can one run stat on stat ? :)

One more note:
after I kill the hanging job I found in stat_results subfolder 880 files with this name pattern
ifsMASTER.sp.x.0050.BE__.dot

[lee218llnl]

Yes, you can run STAT on STAT, but the latter instance is likely to hang for the same reason. I think the problem is that without the MRNet tree, the STAT FE is trying to connect to 880 BEs and send data. This is clearly not scalable and will either take too long or hang. I'm hoping we can get the fix in MRNet to tolerate the "." in your system's nodes hostnames so we can let STAT/MRNet deploy a more scalable tree.

[antonl321]

Perhaps this helps, internally in our cluster the short node names are enough for comms. So if you can get them with something like hostname -s it should be a quick fix.

[lee218llnl]

I looked at the STAT source code and found that we already have a STAT_FE_HOSTNAME environment variable. @antonl321 in your case from above, can you set this variable to "ac1-1001" or whatever the short hostname is from where you invoke STAT? If you look at the .top file that it produces, the value that you specify should be on the first line, rather than the hostname with ".bullx". You might want to test at a smaller scale, say 4 nodes and then when you run stat-cl, also add "-d 2" to the stat-cl args.

[antonl321]

I'm a bit confused, what exactly should be in STAT_FE_HOSTNAME? The name of the head node, or hostname pattern, or ...?