NIF separation

[dvv]

Hi!

I wonder whether I can (and under what conditions) extract msgpack_jsx.erl and portion of msgpack_gen.erl from here to reuse in my pure Erlang lib? The rationale is that NIF is meant to be built for this package to be built, and this presents difficulties for non-unix users.

TIA,
--Vladimir

[Licenser]

I double that,
having the NIF separately would be a improvement! Perhaps leave jsx and jiffy in the msgpack-erlang and make msgpack-erlang-nif separate since it's a different pair of shoes.

Yes, please move the NIF stuff out of the package! 👍

'matthias'.

[kuenishi]

Does rebar work even in windows? Sorry I don't have windows environment. Google told me NIF works even in windows (maybe in some ugly way?). I know msgpack can be built in windows. I'd be happy if any of you send PR which enables building in Windows.
Why do you need any other environments? Or I miss an OS other than windows?

[Licenser]

Generally NIF's are slightly frowned upon by a good part of the Erlang community when used in places where they are not required.
This is not supposed to mean that having a msgpack nif is a bad idea, it's a very good one and cool to have, but perhaps better to separate the pure Erlang implementation from the one that involves C.

Aside from feeling more clean, having a NIF in the same repo requires users to set up the whole build toolchaing of compilers ETC which is not required normally for erlang projects and as in the windows case complicates things a lot.

I personally don't use windows but still would see them seperated if alone for the reason that I don't have to set up gcc and possibly linkers and things for a extensions I don't use (I use the pure erlang jsx version).

That all said I gladly voulounteer to split the two and send a pull request for msgpack-erlang that removes the NIF. So I can't make a pull request for a non existing repository, if you could offer to set up my own msgpack-erlang-nif repo and you can clone that or you set one up with the same code and I fork it and remove the non NIF implementations from that.

Cheers,
Heinz

Yes, rebar works in Windows. And of course you can also compile NIFs, but it is more complicated than on other platforms.

Besides all the good points made by @Licenser, here are my own thoughts.

If I run make bench in msgpack-erlang, I ask myself why you even bother with having a NIF implementation, as jsx is about three times faster when serializing, and only 0.3 times slower when deserializing:

test/bench_tests.erl:36:<0.120.0>: serialize: 0.514 s
test/bench_tests.erl:37:<0.120.0>: deserialize: 0.745 s
test/bench_tests.erl:38:<0.120.0>: for 2011 KB test data(msgpack_jiffy).
test/bench_tests.erl:42:<0.120.0>: serialize: 0.515 s
test/bench_tests.erl:43:<0.120.0>: deserialize: 0.743 s
test/bench_tests.erl:44:<0.120.0>: for 2011 KB test data(msgpack_jsx).
test/bench_tests.erl:48:<0.120.0>: serialize: 0.038 s
test/bench_tests.erl:49:<0.120.0>: deserialize: 0.085 s
test/bench_tests.erl:50:<0.120.0>: for 1884 KB test data(msgpack_nif).
test/bench_tests.erl:54:<0.120.0>: serialize: 0.107 s
test/bench_tests.erl:55:<0.120.0>: deserialize: 0.063 s
test/bench_tests.erl:56:<0.120.0>: for 3828 KB test data(t2b/b2t).
test/bench_tests.erl:75:<0.120.0>: serialize: 2.211 s
test/bench_tests.erl:87:<0.120.0>: deserialize: 3.706 s
test/bench_tests.erl:88:<0.120.0>: for 2011 KB test data(msgpack_jiffy x 5).
test/bench_tests.erl:75:<0.120.0>: serialize: 1.976 s
test/bench_tests.erl:87:<0.120.0>: deserialize: 3.792 s
test/bench_tests.erl:88:<0.120.0>: for 2011 KB test data(msgpack_jsx x 5).
test/bench_tests.erl:75:<0.120.0>: serialize: 0.038 s
test/bench_tests.erl:87:<0.120.0>: deserialize: 0.086 s
test/bench_tests.erl:88:<0.120.0>: for 1884 KB test data(msgpack_nif x 5).
test/bench_tests.erl:75:<0.120.0>: serialize: 0.121 s
test/bench_tests.erl:87:<0.120.0>: deserialize: 0.060 s
test/bench_tests.erl:88:<0.120.0>: for 3828 KB test data(t2b/b2t x 5).

Anyways, there's nothing wrong to have a NIF implementation, but from looking at the benchmarks, I would tend to not use it. And of course it would be nice, when it is not used, to have the option to not include it.

I'd be glad to help, if it's needed. Anyways, I think msgpack-erlang is a great tool, just let us make it even more awesome.

'matthias'.

[Licenser]

I think you're reading the results wrong, the NIF is is about 10 times faster at the moment.

Action      | JSX   | Jiffy | NIF   | term <> binary
serialize   | 0.515 | 0.514 | 0.038 | 0.107
deserailize | 0.743 | 0.745 | 0.085 | 0.063

You are absolutely right! So then the NIF implementation does make sense, when you need the speed.

[Licenser]

It comes with the usual NIF tradeoff of blocking the whole VM,if you have very small pieces of data it's all good but once the NIF takes longer it means that everything else halts - in the example 0.08s for decoding would be something I'd try to avoide, rather I wait 10 times longer and but have the sceduler be able to handle other tasks in the meantime.

[kuenishi]

Ah @Licenser I understand your idea. From my point of view, I don't want bother anyone by pinning versions in the dependency hell like this and this. Single repository reduces a lot cost for maintainer at small project like this, because keeping different repositories consistent is pain (even the number is two).
I might suggest separating rebar_no_nif.config from rebar.config and introduce some -ifndef(?MSGPCK_NIF) to hide out nif-dependent part from compiler.

To speak of scheduler blocking, I'll introduce bumping ticks later, or switch module from nif to others in case of larger term/binary.

@Licenser you made me actually read the man page about NIFs. After reading I'd say, I would like to avoid the use of NIFs as much as possible. Even if you need to integrate some functionality, there are better ways like c-nodes. Which of course comes with an overhead, but won't crash or hang your VM.

[Licenser]

Well anyone interested here is a NIF free version: https://github.com/project-fifo/msgpack-erlang/tree/niffree

[dvv]

I extracted relevant portions here, with atoms as hash keys enabled.