large numpy array to mxnet ndarray conversion

[drmasud]

Description

Conversion fails on large matrices.

Error Message

Traceback (most recent call last):
File "", line 1, in
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/utils.py", line 146, in array
return _array(source_array, ctx=ctx, dtype=dtype)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2505, in array
arr[:] = source_array
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 449, in setitem
self._set_nd_basic_indexing(key, value)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 715, in _set_nd_basic_indexing
self._sync_copyfrom(value)
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 881, in _sync_copyfrom
ctypes.c_size_t(source_array.size)))
File "/home/ec2-user/.local/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [19:09:04] src/ndarray/ndarray_function.cc:51: Check failed: size == to->Size() (-294967296 vs. 4000000000) : copying size mismatch, from: 18446744072529682432 bytes, to: 16000000000 bytes.
Stack trace:
[bt] (0) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2795cb) [0x7efff4d2c5cb]
[bt] (1) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x259399b) [0x7efff704699b]
[bt] (2) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::NDArray::SyncCopyFromCPU(void const*, unsigned long) const+0x284) [0x7efff6fe5934]
[bt] (3) /home/ec2-user/.local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXNDArraySyncCopyFromCPU+0x2b) [0x7efff6d8ebcb]
[bt] (4) /usr/lib64/libffi.so.6(ffi_call_unix64+0x4c) [0x7f0059bbacec]
[bt] (5) /usr/lib64/libffi.so.6(ffi_call+0x1f5) [0x7f0059bba615]
[bt] (6) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2a0) [0x7f0059dcd290]
[bt] (7) /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9586) [0x7f0059dc6586]
[bt] (8) /usr/lib64/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x90) [0x7f0061bce7e0]

Steps to reproduce

T= nd.array(np.random.randn(5000000,800))
(Paste the commands you ran that produced the error.)

What have you tried to solve it?

1. workaround: If you convert in smaller chunks and concatenate the ndarray to create the final mxnet

[stephenrawls]

Looks like 5M > 2^32, i.e. you have more elements than a single int32 can represent.

This PR adds support for int64 number of elements: #14570

But it does it via a build-time flag USE_INT64_TENSOR_SIZE, which defaults to false.

Looks like you need to build a custom version of MXNet with this flag turned on: USE_INT64_TENSOR_SIZE

[samskalicky]

@apeforest assign @access2rohit

[access2rohit]

@drmasud can you paste the output of

python -c "from mxnet.runtime import Features; print(Features())"

[drmasud]

Here it is:

[✖ CUDA, ✖ CUDNN, ✖ NCCL, ✖ CUDA_RTC, ✖ TENSORRT, ✔ CPU_SSE, ✔ CPU_SSE2, ✔ CPU_SSE3, ✔ CPU_SSE4_1, ✔ CPU_SSE4_2, ✖ CPU_SSE4A, ✔ CPU_AVX, ✖ CPU_AVX2, ✖ OPENMP, ✖ SSE, ✔ F16C, ✖ JEMALLOC, ✖ BLAS_OPEN, ✖ BLAS_ATLAS, ✖ BLAS_MKL, ✖ BLAS_APPLE, ✔ LAPACK, ✖ MKLDNN, ✔ OPENCV, ✖ CAFFE, ✖ PROFILER, ✔ DIST_KVSTORE, ✖ CXX14, ✖ INT64_TENSOR_SIZE, ✔ SIGNAL_HANDLER, ✖ DEBUG]

[access2rohit]

@drivanov in your make file add this at the bottom

USE_INT64_TENSOR_SIZE = 1

and build again.

Let us know if that works or not.