Add maestro framework

[x41lakazam]

Maestro is a Pytorch-based benchmarking framework for overlapping communication/compute operations in distributed environments.

This presents a more accurate way to measure performance of AI patterns compared to the micro-benchmarking which measures performance of standalone operations.
It allows a user to define a workload pattern, then benchmark performance of every running element.

Example use-case of an overlapping benchmarking suite

As an example, one user would want to see how well AllGather and ReduceScatter overlap given a certain fabric and algorithms, maestro allow him to easily define it as a pattern running the two and measure the performance of each while they are overlapping, he would get such a result:

Block            | Size (B) | Avg lat (ms) | Min lat | Max lat | P99 lat | Avg BW (GB/s)
allgather        | 4M       | 1.4          | 0.2     | 3.0     | 3.0     | 10.8
reduce_scatter   | 4M       | 1.5          | 0.3     | 3.1     | 3.1     | 6.0

And therefore be able to see the impact of tuning each part of the system on the collectives, and optimize it.

Quick description of maestro capabilities

Maestro benchmark patterns, which are composed of blocks, i.e reusable unit of works that enqueue a GPU activity, and axes, i.e a collection of GPU teams, the user defines which blocks should run on which axis, maestro run the full pattern and report the performance of each block, separately.

An axis is representing the way of splitting the ranks in a classic parallelized AI workload, for example:

Who benefits from maestro

	Communication Engr.	AI researcher	Marketing teams
Goal	Maximize collective performance on realistic end-to-end workloads	Identify the best overlap opportunities (compute/communication and comm./comm.)	Quantify and communicate some added value on realistic overlap patterns
Means	Analyze collective behavior under complex overlap schedules	Benchmark experimental overlap patterns and report performance/efficiency	Compare “classic” patterns across ecosystems (ex: different algorithms) and summarize the impact
Stack used	Pre-configured kernels; customize the YAML file	Custom experimental kernels	No development - uses YAML provided by AI experts
Knowledge required	Communication libraries internals	AI kernels and execution/overlap behavior	None (ability to interpret benchmark results)

Use cases

Below is a few use cases that leveraged maestro for optimization:

• Overlapping SHARP ReduceScatter + AllGather: SHARP team used maestro to better tune both algorithm and reach a full 2 x SOL bandwidth, classical benchmark didn't offer the possibility to run both collectives at the same time properly
• QoS Priorization of collectives: In a highly parallelized AI workload, priorization is key, as such collectives are blocking entire layers while some others are less critical, maestro helped correctly leveraging QoS for priorizing some collectives over other ones
• Congestion control algorithms: Overlapping communications introduced new congestion patterns that didn't manifest in the standalone execution of basic collectives, maestro helped identifying the pain points and optimizing CC algorithms

[meta-cla]

Hi @x41lakazam!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[x41lakazam]

Hey @jansel @williamwen42 @bobrenjc93, is there any missing step to move this PR to review ?

[kwen2501]

Hi @d4l3k @dcci @bobrenjc93
This is a communication-compute overlap benchmark from NV.
Would it help evaluate collectives towards realistic settings?
Please guide us if there is a better place to contribute it. Thanks!

[lappazos]

@d4l3k @dcci @bobrenjc93 could you please review?