jeffro256 if you want better compression of headers in a batch, look into byte entropies of each byte in the header, sort bytes from each header in the order from lowest entropy to the highest (excluding the merkle root hash obviously because it's obviously better to transmit it unchanged). And then send (byte 0, message 0), (byte 0, message 1),

..., (byte 0, message N), (byte 1, message 1), (byte 1, message 2), ... with proper RLE compression. RLE compression is very simple to implement and can be a self-contained small piece of code, easy to check for bugs. It will save you more than 1 byte. You can even experiment with bit-level reordering, it will probably give better results.

This is the "2-3 bytes smaller" I talked about

Colllect entropy data from historical headers, we have millions of them at this point

and you can also decide at which byte index exactly to cut the RLE compression and switch to the plain data

*for each specific batch

you'll have to convert all varints into regular integers before doing this, to normalize header sizes. It won't increase the compressed size thanks to RLE and the preprocessing technique I described

Preliminary numbers: 37.577 bytes per header is possible in theory. 36.733 bytes without the tx count field (it's roughly ~7 bits when compressed)

I'm doing a deep research on the topic now

~37.7 bytes per header in practice with my idea + some refinements

I spent ~1 hour writing a detailed tech doc, then ~1 more hour talking back and forth with Claude Fable 5 to polish it and fix all uncertainties. It's implementing and testing it now on the real Monero mainnet db.

Result: "Implementation complete and validated end to end. Final result: 37,697 bytes mean per 1000-header chunk (37.70 B/header) through the production API on real

mainnet data, with every chunk round-tripped byte-exact and every reconstructed block id matching the database."

Measured order-0 entropy of the best variant is 300.92 bits/header → a floor of ~37,578 payload bytes + ~100 container overhead ≈ 37,680 per 1000 headers. The codec achieves 37,697 — within 0.05% of the floor, so the pipeline is essentially lossless relative to its model class.7

tevador jeffro256 read the above ^

zigzag encoding for timestamps is most efficient and doesn't require monotonic timestamps

interesting that my initial instinct about 37-38 bytes per header was spot-on

<yiannisbot:matrix.org> @rucknium:monero.social: FYI that our proposal has been edited according to feedback on Wednesday to remove unnecessary edits: repo.getmonero.org/monero-project/ccs-proposals/-/merge_requests/667 - let me know if anything extra is needed.

<boog900> they are back :( > <@boog900> We seem to have lost some spy nodes recently: xmrnetscan.redteam.cash

<ravfx:xmr.mx> Oh, it was just maintainance!

<boog900> it looks like we have less known spies and more "normal" nodes

<boog900> I think they updated the digital ocean nodes and maybe added some more

wow, it turns out the header compressor created from my idea beats the strongest general-purpose compressor out there: p2pool.io/u/8505d637cefa4c7a/image.png

noice