03:31:38 <br-m> <spirobel:kernal.eu> that number seems like a lot. is the blocks fetching and the blocks processing fully decoupled?  > <vtnerd> and this was the total scan time, not the unpack time diff
03:32:22 <br-m> <spirobel:kernal.eu> does it show up reliably? how often was the test run? > <vtnerd> and this was the total scan time, not the unpack time diff
03:34:12 <br-m> <spirobel:kernal.eu> ah okay ... yes we cant have it nested too deep. this is an issue with most formats though. if the structure diverges too much from flat arrays there will be a mess  > <vtnerd> the problem with the ZMQ style is that the JSON output requires unpacking the tx from its internal format, and then writing out each component piece-by-piece
16:28:30 <br-m> <vtnerd> LWS uses ZMQ-REP. The request for next chunk of blocks happens “in the background” as the current chunk of blocks is being decoded/scanned. So encoding speedups (in monerod) will only be noticeable when LWS has to wait on the next read. > <@spirobel:kernal.eu> that number seems like a lot. is the blocks fetching and the blocks processing fully decoupled? 
16:31:46 <br-m> <vtnerd> > <@spirobel:kernal.eu> does it show up reliably? how often was the test run?
16:31:46 <br-m> <vtnerd> Yes I would say it is reliable. The test was over the same portion of blocks. I threw out the first run to somewhat rule out file caching monerod side. I can inspect the numbers further later on with cbor, typically the slow downs come from decoding binary data in utf8 safe strings (for json), and to lesser extent integer decoding (there isn’t a ton of these though, more binary data)
16:33:08 <br-m> <vtnerd> and iirc, leaving the txes in cryptonote format and unpacking lws side was fastest overall, and put lowest lost on monerod. however, this forces someone to implement the highly specialized cryptonote native storage format. cbor/msgpack is a good tradeoff of perf and usability (with json being purely for people needing human readable for whatever reason)
16:34:05 <br-m> <vtnerd> *lowest load on monero
17:06:04 <br-m> <spirobel:kernal.eu> @vtnerd: yes this is a big blocker. but then again, we have the cuprate types, so it is possible to do this in rust. i use the cuprate types to consume the getblocksbin endpoint. so its doable. its a lot of tech debt though. 
17:06:46 <br-m> <syntheticbird> I confirm its a good idea. I'm the bank and I would be happy if you were in debt
17:07:11 <br-m> <syntheticbird> (i haven't followed the discussion so ignore my confirmation)
17:07:39 <br-m> <spirobel:kernal.eu> human readable is useful for debugging and so it is easier for people to build an intuition by just playing with the api. it also means clankers can interact with it easily and directly check the results, so there is a quick feedback loop
17:09:17 <br-m> <spirobel:kernal.eu> is there a buffer of blocks? does the chunk of blocks to be processed ever become empty on the lws side?  > <@vtnerd> LWS uses ZMQ-REP. The request for next chunk of blocks happens “in the background” as the current chunk of blocks is being decoded/scanned. So encoding speedups (in monerod) will only be noticeable when LWS has to wait on the next read.
17:09:24 <br-m> <spirobel:kernal.eu> (when not at tip) 
17:55:53 <br-m> <vtnerd> ZMQ-REP only supports one outstanding request. for simplicity LWS uses one ZMQ socket per thread (each thread can be at a different block). one request is being transmitted while the last is being processed. the biggest bottleneck should be the ECDH afaik, but I can profile this more closely than I have
17:59:25 <br-m> <vtnerd> I just went back through the numbers again, and 11% is probably the upper limit (ignore previously ramblings about further optimizations), where msgpack+epee compact format was used
18:00:27 <br-m> <vtnerd> I’ll get some numbers just for json vs cbor where the entire tx is transmitted in pieces (as opposed to annoying cryptonote format)
18:03:45 <br-m> <vtnerd> encoding to cbor is something like ~40-47% faster, but encoding a single json tx is still fairly fast. I’ll run the numbers again in a second as the first attempt was mediocre
18:04:02 <br-m> <vtnerd> the more relevant issue was drop in output size
18:33:46 <br-m> <spirobel:kernal.eu> > <@vtnerd> ZMQ-REP only supports one outstanding request. for simplicity LWS uses one ZMQ socket per thread (each thread can be at a different block). one request is being transmitted while the last is being processed. the biggest bottleneck should be the ECDH afaik, but I can profile this more closely than I have
18:33:46 <br-m> <spirobel:kernal.eu> yes i also think so. and in the future this will most likely disappear with monero-PSK then we only have to think about bottlenecks for consensus scaling 
18:34:38 <br-m> <spirobel:kernal.eu> @vtnerd: this also doesn't matter much. if we transfer over http it should be compressed in any case 
18:36:38 <br-m> <vtnerd> so more cpu time just to get a shittier version that cbor naturally supports? like a cbor encoder isn’t even 200 LOC
18:38:47 <br-m> <vtnerd> @spirobel:kernal.eu: assuming PSK gets any traction given its negatives
18:42:39 <br-m> <spirobel:kernal.eu> @vtnerd: it is mostly fud by people that are in love with their jamtis :D 
18:43:29 <br-m> <spirobel:kernal.eu> it will get traction because I will just implement it. i want users to have many many wallets and participate in escrow sessions 
18:43:53 <br-m> <spirobel:kernal.eu> and currently the sync ux in monero is just not good 
18:44:15 <br-m> <jbabb:cypherstack.com> I like both Monero-PSK and Jamtis's new ISP in Appendix C, but frankly just prefer Jamtis-ISP on a UX level because there's at least one less footgun
18:44:41 <br-m> <spirobel:kernal.eu> i dont care it is just fud
18:45:41 <br-m> <spirobel:kernal.eu> the footguns dont exist the way that has been claimed
18:45:55 <br-m> <jbabb:cypherstack.com> and I'm here for this.  Monero's ultimately permissionless and users can use anything they want > <@spirobel:kernal.eu> it will get traction because I will just implement it. i want users to have many many wallets and participate in escrow sessions 
18:45:55 <br-m> <jbabb:cypherstack.com> but the notes from others re: it being unsafe to even leave in the draft unless totally supported is that if there's only one dev maintaining an entire stack, it's dangerous to recommend that users trust it with their funds
18:45:55 <br-m> <jbabb:cypherstack.com> but at the end of the day, it's permissionless, and whatever people want to use most will get used
18:46:07 <br-m> <spirobel:kernal.eu> people just want to kill it for some reason  before its even done :D 
18:46:10 <br-m> <jpk68:matrix.org> Is it easier to get people to run a script that deploys LWS on Docker, or make every wallet dev implement another addressing protocol that users have to adopt to again? > <@spirobel:kernal.eu> and currently the sync ux in monero is just not good 
18:46:37 <br-m> <jpk68:matrix.org> And you only need LWS if you don't like the sync times
18:46:43 <br-m> <jbabb:cypherstack.com> communication channels have to stay open on an ongoing basis as well to ensure that new monero versions don't break Monero-PSK's impl
18:46:47 <br-m> <spirobel:kernal.eu> @jbabb:cypherstack.com: yes sure it needs scrutiny and i am all for it
18:47:10 <br-m> <jbabb:cypherstack.com> or worse, lost user funds somehow (now THAT is FUD tho :D)
18:47:30 <br-m> <spirobel:kernal.eu> but currently its just not productive 
18:47:36 <br-m> <spirobel:kernal.eu> https://nitter.net/spirobel/status/2054078154407637401#m please just watch the video 
18:47:45 <br-m> <spirobel:kernal.eu> i spent a year working on this its just 2 minutes 
18:48:55 <br-m> <jbabb:cypherstack.com> I have to say also that I appreciate the reorg protection in your api crate.  I notice the reorg_handling folder in the video that reminded me of that
18:49:00 <br-m> <spirobel:kernal.eu> also this https://nitter.net/spirobel/status/2056020125145600366#m if people can see my perspective: people will have lots of wallets to participate in these escrow sessions 
18:52:23 <br-m> <spirobel:kernal.eu> @jbabb:cypherstack.com: yes for sure they will stay open. just right now its people want to push their stuff and see this as a threat to their proposal. even though it genuinely is not. I tried to avoid this multiple times
18:53:15 <br-m> <spirobel:kernal.eu> said politely there is no collision between different addressing protocols. let a 1000 flowers bloom. no need to lock heads over nothing :) 
18:55:10 <br-m> <jbabb:cypherstack.com> it's just nice when things come together and we coordinate to get something into the c++
23:45:56 <br-m> <jberman> > <@spirobel:kernal.eu> people just want to kill it for some reason  before its even done :D 
23:45:56 <br-m> <jberman> tevador has clearly put in a lot of time to try to steelman the proposal as best as possible. I think the idea that has actually been explained thus far generally has atrocious UX. Apparently you have the perfect protocol in your brain that has no downsides and no one else understands it, and are launching ridiculous attacks to buttress your weak position
23:46:13 <br-m> <jberman> rather than just drafting an actual proposal
23:46:27 <br-m> <jberman> as has been asked 900 times so we can avoid wasting all this time trying to guess what's in your brain
23:48:13 <br-m> <jberman> "your idea thus far explained is bad" - multiple people trying to understand what you're proposing
23:48:13 <br-m> <jberman> "fud fud fud! PQ is bullshit and Jamtis is being shoved down our throats" - you