Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

MXNet horovod examples result in a seg fault  #15259

Description

@apeforest

Description

Running mnist example under example/distributed_training-horovod caused segmentation fault. Cross linked an issue filed in horovod repo: horovod/horovod#1148

Environment info (Required)

Framework: MxNet

Framework version:
----------MXNet Info-----------
Version : 1.5.0
Directory : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet
Commit Hash : 62a85f3

Horovod version: 0.16.4

MPI version: 3.1.0

CUDA version: 10.0

NCCL version:

Python version:
----------Python Info----------
Version : 3.6.5
Compiler : GCC 7.2.0
Build : ('default', 'Apr 29 2018 16:14:56')
Arch : ('64bit', '')

OS and version:
--------System Info----------
Platform : Linux-4.4.0-1081-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-63-186
release : 4.4.0-1081-aws
version : #91-Ubuntu SMP Tue Apr 16 08:21:03 UTC 2019

GCC version: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609

Error Message:

(Paste the complete error message, including stack trace.)

Minimum reproducible example

horovodrun -np 4 -H localhost:4 python ./resnet.py or mnist.py

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions