-
hyc
I don't feel like he's focusing on worthwhile targets. dunno
-
hyc
I also haven't timed builds recently to see how much his improvements have actually done to speed up builds
-
hyc
anyone else?
-
hyc
on a completely separate topic... our P2P protocol still seems pretty slow.
-
hyc
was wondering why we don't just rip it out and use bittorrent protocol instead. or something similar.
-
hyc
bittorrent has an advantage in that it can receive chunks in any order. it knows where to put them because it has an initial .torrent file with hashes of every chunk.
-
hyc
we could introduce the notion of "superblocks" that are just sets of hashes of N blocks
-
hyc
so for IBD first you grab as many superblocks as possible. and each can also contain the hash of the preceding superblock, for sanity check
-
hyc
once you have those, you can download the actual blocks in arbitrary order, and you'll always know if you're getting valid ones or not
-
selsta
the p2p ccs should have a larger PR soon
-
hyc
even so, I think this idea will improve IBD significantly
-
moneromooo
The build speedups are really just for fun AFAICT. They don't speed up the common case, except maybe the precompiled headers one.
-
selsta
I'm not sure because I don't think networking is the bottleneck.
-
selsta
Or not significantly.
-
hyc
considering how much faster sync runs with two daemons on the same host, I'm quite sure it is
-
moneromooo
About syncing, monero needs to have the data in order anyway, so you don't havr that much leeway in ordering.
-
selsta
When syncing did you manually increase out_peers and down_limit?
-
moneromooo
If you sync and run "sync_info", you can see whether it's waitoing on the network.
-
selsta
moneromooo has a PR open to increase out peers during initial sync
-
hyc
selsta: on localhost, all default params. just add-exclusive-node 127.0.0.1:xxxx
-
moneromooo
The "moooo" line shows the current state of the blocks queue.
-
moneromooo
"m" means matches what we need next. o means we got that data. . means we requested that data.
-
moneromooo
So if it starts with . you're waiting on network. If it starts with m you're not.
-
moneromooo
It usually starts downloading ahead fairly quickly.
-
moneromooo
What might help is the rw lock on the blockchain class.
-
hyc
if we also changed the on-the-wire txn format to reference outputs by hash instead of 64bit ID, we wouldn't have that block ordering issue
-
hyc
if the concept of 64bit IDs was purely a local DB-internal affair
-
moneromooo
We would.
-
moneromooo
If a tx uses an output and it's not on the chain, you don't know whrether it's valid whether it's an offset or a hash.
-
hyc
sure, for newly arriving txs
-
hyc
but during IBD the node ignores new txs
-
moneromooo
I sure hope it doesn't.
-
moneromooo
Oh you mean it skips checks for stuff matching known good hashes ?
-
hyc
yeah
-
moneromooo
I don't see how using hashed would change anything there.
-
moneromooo
hashes*
-
hyc
it can't validate anything until it has a complete set of blocks
-
luigi1112
hyc afaik his new CCS is part time kinda general stuff. The compile time reduction CCS is still ongoing/hasn't been worked on much recently
-
moneromooo
Yes, 20 blocks atm.
-
moneromooo
But that's like a couple MB typically.
-
moneromooo
Anyway, I guess if you get the network as bottleneck, it might be a worthwhile change. It seems like a very large work/gain ratio though.
-
moneromooo
But do check sync_info first to make sure.
-
hyc
ok
-
hyc
I'll be setting up a new home node next week on a rockpro64
-
hyc
so will test add-exclusive-node across LAN again
-
moneromooo
That said, the code that decides who to ask for what data is very ad hoc, and is probably far from optimal. I dunno how to improve it though.
-
moneromooo
ie, when to decide to request from a second peer some data we need soon and we don't have (because that first peer might be very slow, malicious, or the like).
-
hyc
that sounds like something bittorrent nodes have already optimized
-
moneromooo
If libbittorrent had some code to intelligently do that, it might be a good thing to get (f licence compatible).
-
hyc
dunno if it's worth a look at their code