MessagePack: Fast and Compact Serialization | Hacker News Reader

MessagePack: Fast and Compact Serialization

msgpack.org

50 pointstilt15 comments

For perl it's fast as long as you have a small payload. Here's my benchmarks using progressively bigger data structures.

    perl: 5.008008
    Storable: 2.21
    JSON::XS: 2.25
    Data::MessagePack: 0.34

    ==== Size ====

    .-----------------+-----------+---------+---------.
    | src             | storable  | json    | msgpack |
    +-----------------+-----------+---------+---------+
    | 1               | 4         | fail    | 1       |
    | 3.14            | 8         | fail    | 5       |
    | {}              | 7         | 2       | 1       |
    | []              | 7         | 2       | 1       |
    | [('a')x10]      | 37        | 41      | 21      |
    | {('a')x10}      | 14        | fail    | 11      |
    | +{1,+{1,+{}}}   | 29        | 14      | 7       |
    | +[+[+[]]]       | 19        | 6       | 3       |
    '-----------------+-----------+---------+---------'
    ==========================

    Testing: theirs:HASH (4995 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.14 usr +  0.00 sys =  1.14 CPU) @ 26947.37/s (n=30720)
            mp:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 70446.85/s (n=78196)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 18442.45/s (n=19549)
                Rate storable     json       mp
    storable 18442/s       --     -32%     -74%
    json     26947/s      46%       --     -62%
    mp       70447/s     282%     161%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 23424.51/s (n=23893)
            mp:  1 wallclock secs ( 1.02 usr +  0.00 sys =  1.02 CPU) @ 35136.27/s (n=35839)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 25357.55/s (n=26879)
                Rate     json storable       mp
    json     23425/s       --      -8%     -33%
    storable 25358/s       8%       --     -28%
    mp       35136/s      50%      39%       --


    ==========================

    Testing: xdbi:HASH (24138 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 2414.15/s (n=2559)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 8373.83/s (n=8960)
      storable:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 7244.34/s (n=7679)
               Rate     json storable       mp
    json     2414/s       --     -67%     -71%
    storable 7244/s     200%       --     -13%
    mp       8374/s     247%      16%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.11 usr +  0.00 sys =  1.11 CPU) @ 4842.34/s (n=5375)
            mp:  2 wallclock secs ( 1.13 usr +  0.00 sys =  1.13 CPU) @ 4324.78/s (n=4887)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 4654.29/s (n=4887)
               Rate       mp storable     json
    mp       4325/s       --      -7%     -11%
    storable 4654/s       8%       --      -4%
    json     4842/s      12%       4%       --


    ==========================

    Testing: pdf:HASH (35239 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.04 usr +  0.00 sys =  1.04 CPU) @ 9845.19/s (n=10239)
            mp:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 5119.05/s (n=5375)
      storable:  1 wallclock secs ( 1.10 usr +  0.00 sys =  1.10 CPU) @ 7518.18/s (n=8270)
               Rate       mp storable     json
    mp       5119/s       --     -32%     -48%
    storable 7518/s      47%       --     -24%
    json     9845/s      92%      31%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3110.19/s (n=3359)
            mp:  1 wallclock secs ( 1.07 usr +  0.00 sys =  1.07 CPU) @ 2790.65/s (n=2986)
      storable:  1 wallclock secs ( 1.08 usr +  0.00 sys =  1.08 CPU) @ 3554.63/s (n=3839)
               Rate       mp     json storable
    mp       2791/s       --     -10%     -21%
    json     3110/s      11%       --     -13%
    storable 3555/s      27%      14%       --


    ==========================

    Testing: globallib:HASH (197319 bytes)

    ==== Serialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.09 usr +  0.00 sys =  1.09 CPU) @ 1760.55/s (n=1919)
            mp:  1 wallclock secs ( 1.06 usr +  0.00 sys =  1.06 CPU) @ 791.51/s (n=839)
      storable:  1 wallclock secs ( 1.05 usr +  0.00 sys =  1.05 CPU) @ 1279.05/s (n=1343)
               Rate       mp storable     json
    mp        792/s       --     -38%     -55%
    storable 1279/s      62%       --     -27%
    json     1761/s     122%      38%       --

    ==== Deserialize ====

    Benchmark: running json, mp, storable for at least 1 CPU seconds...
          json:  1 wallclock secs ( 1.71 usr +  0.00 sys =  1.71 CPU) @ 301.75/s (n=516)
            mp:  1 wallclock secs ( 1.03 usr +  0.00 sys =  1.03 CPU) @ 404.85/s (n=417)
      storable:  1 wallclock secs ( 1.12 usr +  0.00 sys =  1.12 CPU) @ 544.64/s (n=610)
              Rate     json       mp storable
    json     302/s       --     -25%     -45%
    mp       405/s      34%       --     -26%
    storable 545/s      80%      35%       --

In node.js, JSON is actually faster than msgpack: https://github.com/aikar/wormhole/issues/3

Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

> Also, due to V8's poor C++ API, msgpack doesn't support deserialization of 64 bit integers.

JavaScript _does_not_ have a number type wide enough to hold 64-bit integer. V8 API has nothing to do with it.

This sounds an awful lot like yet another ProtocolBuffers. What, if anything, new does it bring to the table?

This is schemaless. Protobufs require a .proto file that describes the data being serialized. A .proto file serves a similar purpose as the `struct` keyword in C.

MessagePack uses type prefixes to describe the data. For instance an unsigned 8-bit int starts with a `0xcc` followed by a byte.

Schemaless (dynamic typing) is the most obvious difference.

I wrote about the pros/cons of schema here: http://news.ycombinator.com/item?id=2836242

Compatibility with JSON is a characteristic of MessagePack. If you're already using JSON, you can use MessagePack as a faster/smaller substitution of JSON.

Since Protocol Buffers brings their original semantics, you must change lots of codes.

We use this as data exchange format for our RPC services at Shopify. Incredibly fast, stable, easy to use.

Is this actually better than gzipped JSON?

Depends on how you define 'better', I guess - at the very least it has to be several magnitudes faster, with no overhead for compression, parsing, etc.

You can use type-conversion APIs with MessagePack if you use C++, Java or C#. It converts deserialized objects (=dynamically-typed) into statically-typed objects using templates (C++) or reflection + dynamic code generation (Java and C#). You don't have to write boring type-checking codes.

Note that this is a characteristic of the implementation, not of the data format. Some libraries for JSON like JSONIC (Java) provides these APIs.

This appears to be mmap-friendly. Since arrays have length values it's possible to jump around in a large messagepack-serialized structure without having to scan it. In JSON if you want to get to element 2 of an array you have to scan to the end of element 1. If there's a length value at the beginning of element 1 (as it looks like there is with msgpack) then you can seek straight past it.

Does it have it suitably low memory/CPU requirements to run on embedded devices?

How well does it work on ARM/MIPS?

It works very well at least on ARM.

I've used MessagePack on iPhone 3GS for the format of a dictionary file (> 100MB). I utilized mmap and zero-copy deserialization feature of MessagePack. And I received (and merged) some patches sent by those who use it on iPhone4/iPad2.

I've never tried to run it on MIPS. But it probably works if you use gcc.

(I'm author of MessagePack)

Would be nice if suuport for RPC (and generally) for Erlang was better. It's "currently" in alpha.

Reply on news.ycombinator.com