[BUG] the `benchmarks` folder is included upon installation

[hadim]

I noticed that while inspecting the conda package during my attempt to create a conda forge build.

The fix is likely as simple as adding benchmarks to packages=find_packages(exclude=[....]) in the setup.py file.

[jeffra]

Thank you @hadim, this is a good catch. I'll create a PR to fix this shortly.

[hadim]

benchmarks is still included in 0.8.0 but only the subfolder benchmarks/communications/ and its content.

Maybe in packages=find_packages(exclude= you need to specify benchmarks.communications or benchmarks.*?

You can double-check this by building a wheel locally and opening it to check which folders have been included.

---

It's not a critical issue but it's best to remove it to prevent any conflicts with other libraries.

[jeffra]

Thank you @hadim for bringing this up. I think now that i am looking at all of this closer we might just want to change this to be something like packages=find_packages(include=['deepspeed', 'deepspeed.*']),. But I am doing some local testing first to make sure this works as expected.

[jeffra]

Can you confirm this fixes the issue from your perspective?

#2714

[hadim]

It seems ok to me but I haven't setup deepspeed locally to confirm.

If you build a wheel locally with pip wheel you can then open the wheel and check what has been included. Also any CI installing deepspeed in non editable mode will confirm all the required deepspeed modules have been included correctly.

[jeffra]

I installed locally with the wheel we upload to pypi. We do build via sdist, which is a bit different: https://github.com/microsoft/DeepSpeed/blob/3f210c971590d5d23acc57004330f9ae38001748/release/release.sh#L34

Our CI will install and run all the tests on my PR to check that everything is all still good. My local tests before pushing the PR all looked good though. I installed in a fresh virtualenv and inspected the lib/python3.8/site-packages for that virtualenv to see what was installed. I repro'd your issue with the current master where you see that benchmarks/communication still exists.

[hadim]

Yes sdist and wheel even if different will both highlight what has been correctly or incorrectly included.

If it works for you locally then I think it's all good!