When predicting, does mxnet provide thread-safe interface?

[gold-mango]

When deploying online, multi-threads is usually required. If each thread load a model, the memory consume costly, so is there a thread-safe interface that share model parameters?

[zihaolucky]

I have same question.

[piiswrong]

The engine is not thread safe so there is no way to use multiple threads for pushing computation to the engine.

However the engine already does threading and scheduling for you so you really shouldn't need to.
Instead consider a multi thread producer, single thread consumer model. You use multiple thread to accept request and push them to a queue, a backstage thread then read the queue and push it to mxnet and send the result back to the producer.
You can bind multiple executors on the same set of weight ndarrays and its ok as long as you don't modify them and always run them from the same thread. Although it probably won't bring you any benefits if you are doing it on the same card

[piiswrong]

BTW, python multi threading doesn't really work due to GIL. Try multi processing instead

[gold-mango]

@piiswrong On the same card, if I bind multiple executors on the same set of weight ndarrays, and run them from different thread simultaneously, is that ok?
Does it benefit? because more than one model are running simultaneously, then increase throughput rate?

[phoenixbai]

I have the same problem. Is it possible to provide multi-threaded prediction service using mxnet? As in caffe case, we have to copy the whole net as many times as the thread number. Is mxnet any better?

my trained model size is normally around 1GB, so memory consumption is a big issue.

[fasahath]

Any progress here?

[everwind]

I have the same question. Any progress here?

[everwind]

I think we should separate "data" and the model weights . when predicting, each thread can share the model weights and use different data . In this way , we have mush less memory consumption .
In practice, we often train the model on GPUS, but use the model on the cpu machine. so it is very important to provide the effective multi-thread prediction of the model

[leopd]

Tl;dr: if you use a high performance python web-server like gunicorn, you'll get what you want. You'll have lots of cores running in parallel working on different requests, and you won't need a copy in memory of the model for each worker.

Getting a complex codebase to be threadsafe is no small task, so this won't be "resolved" any time soon. Fortunately it's not necessary here for most of what you want. Your answer lies in the Unix fork() command. If you want to understand, go read: https://en.wikipedia.org/wiki/Fork_(system_call)

The magic of fork is copy-on-write memory semantics whereby each forked worker has its own virtual memory address space, but they all share the same physical memory. (Until one of them writes to the shared memory, in which case a private copy of that memory block is made in that process's virtual address space -- thus "copy-on-write".) So even though it's not multi-threaded, fork() & pre-fork worker servers like gunicorn let you accomplish almost the same thing with multiple processes instead of threads. Forked processes are somewhat more heavyweight than threads, but they're nowhere near as expensive as you running the same command multiple times.

[bhavinthaker]

Good points made by piiswrong@ and leopd@.

Based on my experience of using fork() over many years, I would say that calling fork() from a Multi-Threaded process is typically NOT recommended. If you do, you will need to understand the details in how it works to make correct use of it. A few caveats on the use of fork() are:

1. The behavior of fork() can vary across operating systems. For example, Solaris has fork-one and fork-all model, whereas Linux has fork-one model. The fork-one model duplicates only the calling thread whereas the fork-all model duplicates all the calling threads.

2. The fork-one model can leave mutexes in bad state in the called thread. See excellent description of the problems in the classic book on Linux Programming Interface [2].

3. A good summary of the problem is here: [3]

4. More problems are described here: [4]. For example, you need to ensure that Close-on-exec flag is set for file-descriptors [depends on whether the programming language does it for you, C does not], otherwise two processes will have reference to the same file and close() from one process will NOT actually close the file since the file is also referenced from the other process.

5. Long discussion on why fork without exec is dangerous in large programs [5]. I always used exec() after a fork() and set the Close-on-exec() for file-descriptions when I have called fork() from a MT process.

References:
[1] fork-one and fork-all model: https://docs.oracle.com/cd/E19683-01/806-6867/gen-1/index.html
[2] The Linux Programming Interface: A Linux and UNIX System Programming Handbook
By Michael Kerrisk: https://books.google.com/books?id=Ps2SH727eCIC&pg=PA686&lpg=PA686&dq=fork+but+no+exec+mutex&source=bl&ots=kMBgx2BVwb&sig=PI74pt3fvO8gHKLLUaIUrr_shy8&hl=en&sa=X&ved=0ahUKEwjTu8qn75_WAhVJ7GMKHb46BQIQ6AEIOjAH#v=onepage&q=fork%20but%20no%20exec%20mutex&f=false
[3] https://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/
[4] http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
[5] https://news.ycombinator.com/item?id=12302539

[bhavinthaker]

phoenixbai> As in caffe case, we have to copy the whole net as many times as the thread number. Is mxnet any better? my trained model size is normally around 1GB, so memory consumption is a big issue.

I would suggest saving the model in a file and accessing the file using memory mapped files from multiple threads/processes. This will significantly reduce the memory requirements of your solution.

Some good references:
[1] https://www.codeproject.com/Tips/683614/Things-to-Know-about-Memory-Mapped-File-in-Java
[2] https://howtodoinjava.com/java-7/nio/java-nio-2-0-memory-mapped-files-mappedbytebuffer-tutorial/
[3] http://javarevisited.blogspot.com/2012/01/memorymapped-file-and-io-in-java.html

[nswamy]

@piiswrong can you expand on why you suggesting that the executors should accessed from the same thread ? This means that for the lifetime of the process only thread will be responsible for interacting with MXNet.