[BUG]

[Rishabh-Choudhry]

Describe the bug

When using deepspeed accelerator with pytorch lightning, I get this error. The issue seems to be the wrong path calling in the deepspeed library for UtilsBuilder class.

File "deepspeed/runtime/engine.py", line 392, in init
util_ops = UtilsBuilder().load()
TypeError: 'NoneType' object is not callable

The issue gets resolved after updating the path from from deepspeed.ops.op_builder import UtilsBuilder to from deepspeed.ops.op_builder.utils import UtilsBuilder.

To Reproduce
Use strategy=DeepSpeedStrategy(stage=2) with torch lightning trainer.

Expected behavior
The deepspeed accelerator should run as normal with pytorch lightning.

System info (please complete the following information):

• OS: Ubuntu
• GPU count and types : 1 x NVIDIA T4
• Python version : 3.8.12

[tjruwase]

@Rishabh-Choudhry, thanks for reporting this issue. Do you mean submitting a PR?

[jeffra]

@Rishabh-Choudhry and @yasyf, I can't seem to reproduce this issue. Can you share what version of deepspeed you are using? I tried latest master and also v0.8.1.

I am trying to reproduce via:

from deepspeed.ops.op_builder import UtilsBuilder
util_ops = UtilsBuilder().load()

I am glad to hear the suggested fix from @Rishabh-Choudhry works, but i worry other ops might also have the same problem but not being triggered in your setting. Maybe I need to try w. pytorch lightning directly though.

[yasyf]

@jeffra I am using master (with pytorch nightly)

[jeffra]

@yasyf, does the above snippet reproduce the stack trace or is it only reproducible from lightning?

Update: I was able to repro this, I will push a fix very soon.

[jeffra]

Hi @yasyf and @Rishabh-Choudhry, I've pushed a PR (#2963) can you give the related branch a try on your side to confirm if it fixes your issue or not?

[yasyf]

@jeffra fixed! totally didn't realize I still had colossal installed :)

[jeffra]

Excellent! Glad to hear :) @Rishabh-Choudhry please reopen if this is not fixed for you

[SamieeS]

Hi @jeffra ,

I get the same error when I try to use deep speed with Huggingface (without lightning). Suggested solution by @Rishabh-Choudhry helped to manually fixing it.

The problem is coming from these two files:

accelerate/utils/deepspeed.py
deepspeed/runtime/zero/stage_1_and_2.py