Repository navigation
dpack data much bigger than json data #2
Description
Hi,
I am searching for a compact representation of javascript objects. It seems dpack is actually much larger than json for my use case. My data consists of one object which contains a lot of arrays and small objects with loads of Number primitives.
e.g.
[ [1, 2, 3, 4, { prop1: 5, prop2: 6}], [7, 8, 9, 10, { prop1: 11, prop2: 12}] ... ]
except that these numbers are not integers but doubles, so each number in json is around 16 to 18 bytes.
It turns out, however, if i look at the generated dpack data, that each of those numbers is still stored as ASCII using also 16 to 18 bytes each. The outcome is that a 1.9MB json test file is 3.7MB stored as dpack.
I would assume that a Number should be represented as 8 bytes max (or preferably 4 bytes if one would have an option to reduce doubles to floats).
Confusing is also that apart from storing Numbers similarly as json the dpack data is much larger than the json data.
What is going on here, is this a bug?
Activity
kriszyp commentedon Feb 23, 2021
Yes, dpack does store numbers with decimal ASCII representation, which can be large for floating point numbers. I am not sure why your particular case is larger in dpack, the example above is 41 bytes in dpack and 68 bytes in JSON. Also, it is worth noting that most of the efficiency/compactness gains from dpack come from reusing properties (and property names).
If you are looking for a more compact representation of floating point numbers, I would suggest that MessagePack or CBOR is much more efficient. I have shifted to focusing more developing a couple of libraries for these. You might want to check out https://github.com/kriszyp/msgpackr which has really good performance and compactness. It also includes options for specifically storing floating point numbers in 32-bit format, which may be useful for compact numeric storage.