So, I couldn't see it in the readme, apologies if I missed it but why?
It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
Almost double the decompression speed, *and* a higher compression ratio than LZ4? That's very promising.
I don't see much in your README that talks about how a developer would integrate misa into their code. I might suggest some basic code samples to help a developer integrate misa decode into their project.
> It offers particularly high decompression throughput on highly compressible files.
You know, you would expect that this is commonly observed. You'd expect, say, simple RLE (run length encoding) to be like this. It's more expensive to produce output requiring the processing of a large number of RLE opcodes that produce short sequences than one opcode that produces one long sequence.
Interesting, but if you are not robust to corrupted/malicious data, it is really in a different class of algorithm and it is hard to compare speeds directly.
From memory, 2505 MB/sec also sounds on the low side for LZ4 on a modern CPU?
In short, my decompressor is very simple, and a naive safe version of the decompressor is only about 5% slower than the current unsafe one (and I will add this safe version in v0.3.0).
As for the raw throughput numbers being low here, it's because Intel Turbo (frequency boost) was disabled for stability, and the CPU was running at a fixed frequency of 2.1 GHz (I've confirmed that the relative performance scales similarly even with Turbo enabled).
this is super interesting! im excited to give this a look this afternoon, since I specifically have wanted faster throughout for decompressing maps in a game engine.
You can look at the LZ4 decompression implementation in ClickHouse: https://presentations.clickhouse.com/2018-highload-siberia/ (the presentation is quite old, but the implementation was updated recently). It uses the same LZ4 format, just decompresses faster.
misa77 primarily targets textual data (ie. byte-aligned data formats where each byte corresponds to a symbol), so I hadn't tested it on game assets much until now.
After seeing your comment, I pulled some random assets from Pathfinder WoTR (in fact, Unity compresses them with lz4hc) and DOS2. The gains are much more modest here due to asset data being mostly floats, but level 0 performs decently nevertheless.
Just tested a few minutes ago on the ClickBench dataset. Overall quite good, but, depending on particular columns, most of the time slower - e.g., ~2000 vs ~2800 GB/sec on Graviton 4 machine in AWS.
It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
- misa77's format may change unexpectedly as it's still v0.x.y.
- The decoder assumes that the input is a valid misa77 stream. Invalid input is UB and I offer no guarantees for whatever misa77 does in this case.
- It's been through some local fuzzing but is not hardened, so treat it as experimental.
I don't see much in your README that talks about how a developer would integrate misa into their code. I might suggest some basic code samples to help a developer integrate misa decode into their project.
Congrats, looks like a cool project.
You know, you would expect that this is commonly observed. You'd expect, say, simple RLE (run length encoding) to be like this. It's more expensive to produce output requiring the processing of a large number of RLE opcodes that produce short sequences than one opcode that produces one long sequence.
From memory, 2505 MB/sec also sounds on the low side for LZ4 on a modern CPU?
In short, my decompressor is very simple, and a naive safe version of the decompressor is only about 5% slower than the current unsafe one (and I will add this safe version in v0.3.0).
As for the raw throughput numbers being low here, it's because Intel Turbo (frequency boost) was disabled for stability, and the CPU was running at a fixed frequency of 2.1 GHz (I've confirmed that the relative performance scales similarly even with Turbo enabled).
misa77 primarily targets textual data (ie. byte-aligned data formats where each byte corresponds to a symbol), so I hadn't tested it on game assets much until now.
After seeing your comment, I pulled some random assets from Pathfinder WoTR (in fact, Unity compresses them with lz4hc) and DOS2. The gains are much more modest here due to asset data being mostly floats, but level 0 performs decently nevertheless.
Results on a map asset (WoTR):
Results on equipment asset (WoTR): Results on texture asset (DOS2): Note: the benchmarking setup is identical to the intel x86-64 one described in the readme.For reference, to test ARM64, I tested v0.1.0 on an M3 mac with this fork of lzbench: https://github.com/welcome-to-the-sunny-side/lzbench/tree/ad...
Here, lz4's decompression speed was far slower than misa77 and zxc. Results are here: https://github.com/welcome-to-the-sunny-side/misa77/blob/mai...