02:35:01 jeffro256 if you want better compression of headers in a batch, look into byte entropies of each byte in the header, sort bytes from each header in the order from lowest entropy to the highest (excluding the merkle root hash obviously because it's obviously better to transmit it unchanged). And then send (byte 0, message 0), (byte 0, message 1), 02:35:01 ..., (byte 0, message N), (byte 1, message 1), (byte 1, message 2), ... with proper RLE compression. RLE compression is very simple to implement and can be a self-contained small piece of code, easy to check for bugs. It will save you more than 1 byte. You can even experiment with bit-level reordering, it will probably give better results. 02:35:24 This is the "2-3 bytes smaller" I talked about 02:35:47 Colllect entropy data from historical headers, we have millions of them at this point 02:36:49 and you can also decide at which byte index exactly to cut the RLE compression and switch to the plain data 02:40:36 *for each specific batch 02:45:48 you'll have to convert all varints into regular integers before doing this, to normalize header sizes. It won't increase the compressed size thanks to RLE and the preprocessing technique I described 08:49:07 Preliminary numbers: 37.577 bytes per header is possible in theory. 36.733 bytes without the tx count field (it's roughly ~7 bits when compressed) 08:49:17 I'm doing a deep research on the topic now 08:50:37 ~37.7 bytes per header in practice with my idea + some refinements 08:54:35 I spent ~1 hour writing a detailed tech doc, then ~1 more hour talking back and forth with Claude Fable 5 to polish it and fix all uncertainties. It's implementing and testing it now on the real Monero mainnet db. 08:55:08 Result: "Implementation complete and validated end to end. Final result: 37,697 bytes mean per 1000-header chunk (37.70 B/header) through the production API on real 08:55:08 mainnet data, with every chunk round-tripped byte-exact and every reconstructed block id matching the database." 08:56:07 Measured order-0 entropy of the best variant is 300.92 bits/header → a floor of ~37,578 payload bytes + ~100 container overhead ≈ 37,680 per 1000 headers. The codec achieves 37,697 — within 0.05% of the floor, so the pipeline is essentially lossless relative to its model class.7 09:18:15 https://gist.github.com/SChernykh/194de94faeb701cc46b928a4fb02922d 10:20:23 tevador jeffro256 read the above ^ 10:20:39 zigzag encoding for timestamps is most efficient and doesn't require monotonic timestamps 10:26:55 interesting that my initial instinct about 37-38 bytes per header was spot-on 14:00:30 @rucknium:monero.social: FYI that our proposal has been edited according to feedback on Wednesday to remove unnecessary edits: https://repo.getmonero.org/monero-project/ccs-proposals/-/merge_requests/667 - let me know if anything extra is needed. 16:08:46 they are back :( > <@boog900> We seem to have lost some spy nodes recently: https://xmrnetscan.redteam.cash/ 16:10:54 Oh, it was just maintainance! 16:13:22 it looks like we have less known spies and more "normal" nodes 16:14:29 I think they updated the digital ocean nodes and maybe added some more 18:14:11 wow, it turns out the header compressor created from my idea beats the strongest general-purpose compressor out there: https://p2pool.io/u/8505d637cefa4c7a/image.png 18:14:13 https://gist.github.com/SChernykh/194de94faeb701cc46b928a4fb02922d#comparison-with-general-purpose-compressors-informative 22:00:09 noice