Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

RNNOp regression on CPU when using a GPU-enabled build #16628

Description

@leezu

Description

Using a master build (27f7082), I observe the following error when invoking RNNOp on a CPU machine. The line causing the error was added by @DickJC123 in #16391.

@DickJC123 should all CUDA_CALLs and CUDNN_CALLs be put in an extra if ctx_.dev_-ype == kGPU) check? It seems they are currently run even if the RNNOp is executed on CPU.

Error Message

E           mxnet.base.MXNetError: [18:23:30] /opt/brazil-pkg-cache/packages/DeepMXNet/DeepMXNet-master.322.0/AL2012/generic-flavor/src/src/operator/./rnn-inl.h:496: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: CUDA driver version is insufficient for CUDA runtime version
E           Stack trace:
[bt] E              (0) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x24) [0x7fb186c7e144]
[bt] E              (1) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::op::RNNOp<mshadow::cpu, float, float>::RNNOp(mxnet::op::RNNParam, mxnet::Context)+0xff6) [0x7fb18a684e26]
[bt] E              (2) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::OpStatePtr mxnet::OpStatePtr::Create<mxnet::op::RNNOp<mshadow::cpu, float, float>, mxnet::op::RNNParam const&, mxnet::Context const&>(mxnet::op::RNNParam const&, mxnet::Context const&)+0xd3) [0x7fb18a6858b3]
[bt] E              (3) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(+0x4f2f786) [0x7fb18a58f786]
[bt] E              (4) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(std::_Function_handler<mxnet::OpStatePtr (nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&), mxnet::OpStatePtr (*)(nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&)>::_M_invoke(std::_Any_data const&, nnvm::NodeAttrs const&, mxnet::Context, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> > const&, std::vector<int, std::allocator<int> > const&)+0x15) [0x7fb189352e65]
[bt] E              (5) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(+0x3e17f7d) [0x7fb189477f7d]
[bt] E              (6) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::imperative::RunGraph(bool, nnvm::IndexedGraph const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, unsigned long, unsigned long, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> >&&, std::vector<unsigned int, std::allocator<unsigned int> >&&, std::vector<mxnet::OpStatePtr, std::allocator<mxnet::OpStatePtr> >*, std::vector<mxnet::DispatchMode, std::allocator<mxnet::DispatchMode> > const&, bool, std::vector<mxnet::TShape, std::allocator<mxnet::TShape> >*, std::function<void (char const*, char const*, void*)> const&, bool)+0x1fd) [0x7fb18947846d]
[bt] E              (7) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::CachedOp::DynamicForward(mxnet::Context const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, bool)+0xc55) [0x7fb18943ed95]
[bt] E              (8) /local/p4clients/pkgbuild-VvHTh/workspace/build/Python-gluonnlp/Python-gluonnlp-0.9.x.38.0/AL2012/DEV.STD.PTHREAD/build/private/tmp/brazil-path/testrun.runtimefarm_excluderoot/lib/libmxnet.so(mxnet::CachedOp::Forward(std::shared_ptr<mxnet::CachedOp> const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0xcee) [0x7fb18944723e]
/opt/brazil-pkg-cache/packages/DeepMXNet/DeepMXNet-master.322.0/AL2012/DEV.STD.PTHREAD/build/lib/python3.6/site-packages/mxnet/base.py:254: MXNetError 

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions