15:03:22 MRL meeting in this room in two hours. 16:20:17 I'll have limited availability in today's meeting unfortunately. I've been working pretty much exclusively on FCMP++ & Carrot stressnet issues. Repeating from last week, the issues we're seeing mainly seem to stem from existing issues in monerod. The FCMP++ & Carrot integration has had some hiccups, but for the most part the integration seems to be holding up well with hiccups patched 17:00:37 Meeting time! https://github.com/monero-project/meta/issues/1281 17:00:50 1. Greetings 17:00:56 Hi 17:01:57 Howdy 17:02:49 @jberman: I can second this, he's been working like a dog ;) I've also been working on working on existing issues in monerod which the stressnet exposed 17:02:56 hello 17:03:22 Hi 17:03:42 2. Updates. What is everyone working on? 17:03:44 It's been very interesting to say the least, and definitely highlights the importance of fixing existing issues of efficient transaction/block propagation b/c it'll be needed more than ever with 4x bigger txs 17:05:12 me: Keeping stressnet stressed. Keeping stressnet infrastructure (e.g. https://stressnetnode1.moneronet.info/ , https://stressnetnode2.moneronet.info/ , and seed nodes) alive and well. 17:06:13 me: PoWER and looking into ZMQ for Cuprate 17:06:42 me: legacy and balance keys now fully work in lws. Now working on an official docker image, as btcpayserver really needs it 17:07:02 Yay! 17:07:23 Me working on a write-up on the penalty and scaling. 17:08:20 3. Proof-of-Work-Enabled Relay ("PoWER") (https://github.com/monero-project/research-lab/issues/133). 17:08:40 I've updated my proposal from last meeting: https://github.com/monero-project/research-lab/issues/133#issuecomment-3377869740 17:09:01 The P2P interface is slightly different than RPC/ZMQ, it's a 1-time challenge/solution exchange 17:09:26 The RPC/ZMQ endpoint provides unique challenges per connection and holds them until an expiry time, solutions are verified at other TX relaying endpoints; these could be cleared occasionally or maybe cleaned if set.size() >= POWER_MAX_ACTIVE_NONCES 17:10:25 Unique connections are still left undefined but I think this can be decided elsewhere, for now I assuming some network details 17:11:25 Maybe I'm misunderstanding the proposal, but keeping a finite set of valid certificates is a DoS vector 17:11:32 I still believe requiring RPC connections to be held for PoW is fine 17:11:56 And simplifies the protocol a lot 17:12:36 @jeffro256: If there is no limit the node's memory can be exhausted 17:13:02 The protocol shouldn't require holding potential certificates in the first place 17:14:09 It should be holding already completed PoW only 17:14:09 Wallet2 and monero-oxide both hold rpc connections 17:15:16 @jeffro256: Doesn't there have to be a limit on this as well? 17:16:07 Yes, but the attacker would have to do X seconds of PoW to store 16 bytes on your machine 17:16:21 if this certificate/check can be generated from the connection details, it'd not be any extra data other than connection itself, bounded by connection limit 17:16:26 @boog900: FYI this would bypass the state problems, RPC could exchange like P2P/ZMQ 17:16:32 even if it has to store x bytes, it'd be bounded by connection limit 17:17:00 And worst case scenario, they get to re-use PoW for a separate connection after doing many, many thousand of iterations of PoW. Whereas worst case scenario w/ your current proposal is that connections doing 0 PoW can lock out honest peers 17:18:29 Ok I see, I misunderstood the idea, I think that's acceptable 17:18:58 @hinto: It only means people who don't hold RPC connections can't relay >8 input txs 17:19:43 The cert also needs to be signed (preferably symmetrically for performance) by the validating node, otherwise the attacker can make up challenges in advance 17:20:27 I guess lws gets to pass power along to client wallets? Otherwise the server has to do it which isnt ideal 17:20:44 What kind of expiry on the cert? 17:21:39 In other words, creation of challenges should be stateless (besides the node's permanent signing keys), otherwise it poses a DoS vector to honest peers. The onus of storing ones's own validation information should be on the proving peer 17:23:47 19:19:43 The cert also needs to be signed (preferably symmetrically for performance) by the validating node, otherwise the attacker can make up challenges in advance 17:23:49 (I again bring up TOTP as an example of how challenges could be computed with a time factor on the go, based on an agreed "secret" information) 17:23:52 Is there an issue I am not seeing by requiring people to hold RPC connections? 17:24:15 @jeffro256: Is this necessary on a 128-bit nonce? 17:25:43 DataHoarder: But why share secret information when secret information does not need to be shared? 17:26:19 the TOTP result is shared 17:26:19 @boog900: Wallet2 does it, monero-oxide does it. If we think this is acceptable, then we don't need to handle certificates, used nonces etc 17:26:19 @articmine: I think 1-5 minutes from the request would be okay 17:26:19 not the secret part 17:26:48 the agreed secret part doesn't specifically need to be sent over, it can be incoming connection details + local random secret 17:27:08 to re-verify the TOTP() of this you just need the connection details + time it was generated at 17:27:15 @hinto: That makes sense 17:27:34 @hinto: In my stateless proposal, yes. Not necessary in your proposal since it's infeasible to guess a 128-bit nonce generated on a node you don't operate 17:28:03 that'd generate a nonce (of however many bits) you can verify when the proving peer brings back a PoW result (along with the above^) 17:28:19 DataHoarder: How would the prover know your local random secret without asking you? 17:28:33 DataHoarder: This is susceptible to front-loading attacks 17:28:36 they don't, the prover doesn't run TOTP 17:29:28 local node, when RPC connection that must be verified comes in, returns result of TOTP(details | secret | time) + time 17:29:35 So the prover still has to request a challenge, yes? 17:29:48 yes, but this challenge doesn't need storage by node 17:30:08 it's also limited in time directly (As you get time when prover returns PoW, and can instantly remove these) 17:30:11 Okay then I think we're talking about the same thing 17:32:07 @boog900: i love how no one answered you probably because they have doubt in mind but don't manage to put words on it. 17:32:08 (i see no issue) 17:32:52 @boog900: Sorry, what specific actors you do mean by "people" and RPC to which software? 17:33:45 Wallets holding RPC connections to monerod 17:34:12 If we think this requirement is OK then we can just generate a random nonce on connection like we will do for P2P 17:35:26 I think it should be optional, especially for local nodes. But I'm okay with supporting PoW for wallet->daemon connections, specifically for transaction submission endpoints 17:35:48 as I suggested last time it's maybe better to have out of meeting conversations, but isn't it generally that 1) node must be able to generate challenges and not keep per-challenge state (single global state is ok as it's fixed byte size) and 2) the prover must not be able to generate new nonces ahead of time (or collect too many without a defined 17:35:50 expiration date) and 3) valid nonces for PoW have to expire after a defined amount 17:36:00 I wouldn't support it for other endpoints 17:36:29 unrestricted RPC could have it disabled, restricted RPC enabled? 17:36:50 It would still work the same as your proposals, with an endpoint to get the challenge and an endpoint to submit 17:37:24 Its just the challenge is for a specific connection so you cannot disconnect then reconnect and submit 17:37:42 DataHoarder: PoWER is skipped on local/trusted interfaces 17:37:53 DataHoarder: Agreed. State per success-post-verification is okay though, and is probably necessary. But yeah further implementation details of the exact should probably be discussed on the Github issue, not in here 17:38:48 post verification is bounded to RPC connection limits, so reasonable data usage there would be fine as it comes with a bound 17:39:58 Not just RPC connection limits, but PoW recently performed 17:41:28 Right. RPC connects/disconnect (doesn't keep TCP connection open) so ports are not relevant. 17:42:19 If given expiration explicitly via generation these can be kept exactly as long as needed then removed away 17:42:44 Tbf me and hinto have discussed it and he thought it would be unpopular to require wallets to hold connections. 17:43:12 I disagree though and can't see an issue, it would simplify the protocol a lot. 17:43:24 Oh hold connections? 17:43:42 Yeah I wouldn't want to implement it like that 17:44:01 It would probably break a lot of app developers' flows 17:44:10 Just do PoW when submmiting a tx 17:44:25 Wallet2 and monero-oxide both hold connections 17:45:34 @jeffro256: As long as you can do more than 1 request per connection you can still do this 17:45:56 one issue I’m seeing is with ZMQ router/dealer stuff. you can’t use those setups with PoWER probably 17:46:11 I dont know of anyone using those setups though 17:46:51 @jeffro256: Fwiw they will still work as long as you don't send a > 8-input tx 17:47:40 @boog900: Is wallet2 really doing that? Hmmm... Probably not ideal for remote node privacy 17:47:40 alternatively have send_raw_transaction allow a list of transactions :) 17:47:45 FWIW I agree it would simplify everything, although agree with jeffro 17:48:15 @jeffro256: Why? Better than reconnecting for every request 17:48:22 @boog900: Which is not a serious issue 17:49:42 I think we can further discuss on the issue, should we move on? 17:50:09 @boog900: Why would re-connecting ever be worse for privacy? Re-connecting means that RPC requests from different wallets with the same host have some mixing. It's not much, but it's something 17:52:44 @jeffro256: True but that's minimal and breaks my RPC PoWER idea D: 17:53:08 4. Transaction volume scaling parameters after FCMP hard fork (https://github.com/ArticMine/Monero-Documents/blob/master/MoneroScaling2025-07.pdf). Revisit FCMP++ transaction weight function (https://github.com/seraphis-migration/monero/issues/44). 17:53:50 Any questions on this? 17:54:46 In particular the square root weights 17:55:18 ...and quadratic fees 17:57:54 Seeing none I will continue with my writeup and suggest we move on 17:58:29 For the record, I still NACK for the same reasons as last week, I just don't have anything new to add at the moment 17:58:40 5. FCMP alpha stressnet (https://monero.town/post/6763165). 17:59:27 There was a new stressnet release, version 1.2: https://github.com/seraphis-migration/monero/releases 18:00:14 With large blocks, nodes are having problems staying connected. The release fixes a minor cause of the connection problems, but many more remain. 18:00:17 Working through issues at the moment. Is there any specific issue someone wants to bring up or re-iterate? 18:00:44 I am glad that you and jberman are invested in this stressnet ;) 18:01:15 As you said, many of the issues are longstanding monerod issues, not specifically FCMP issues 18:01:20 And we saw most of them on last year's stressnet 18:01:52 Last year's stressnet just fixed the truly fatal issues, like irreparable netsplits. 18:02:04 I'm not sure if its an existing issue with monerod or with the stressnet version but these bans that are happening on stressnet are causing connectivity issues between peers: 18:02:05 https://github.com/seraphis-migration/monero/issues/180 18:02:40 AFAIK, there is no easy way to clear all of your nodes' bans without restarting the node. 18:03:08 Can some console command and/or RPC method be written to clear all bans? 18:04:20 Even better, write something that can clear bans by CIDR subnet notation. Then you can clear all bans with a large CIDR, but clear specific subnets, too. 18:04:20 There is a script on of the guys made to clear the bans via the RPC which works but covers up the cause of the bans. If the bans are legit and supposed to happen then that is fine, the script that be used to clear the bans periodically. 18:04:20 get_bans already exists, and set_bans as well. set_bans with ban: false should unban? 18:04:54 could make something that is CIDR aware 18:04:56 There's this script which I haven't tested yet: https://github.com/seraphis-migration/monero/issues/180#issuecomment-3405320886 18:04:57 No idea if it works or not 18:05:11 @rucknium: This would be nice 18:06:25 Yes that is the script and its using get_bans and set_bans and it seems to resolve the issue. Just figured I would bring it up b/c it is annoying to have unban to get connectivity and not everyone running a node is doing so and as such I find my node gets isolated from the network quite often. 18:06:26 Anyway if this is expected behavior then that is fine. Just figured I would bring it up in case its not. 18:06:37 An external script is useful, but having a dedicated way to do it would help ordinary stressnet users 18:07:33 No, the bans aren't supposed to happen, but they do because nodes think you are misbehaving when the network has heavy load. 18:08:41 Might be more serialization issues. We're looking into it. Better logging is a must going forward IMO 18:08:57 IIRC, on last stressnet, the ban timeout was set low manually in a stressnet PR 18:09:01 Or other levin limits 18:09:17 @rucknium: That could be done here 18:09:19 Logging can stress a node too, AFAIK. That's double stress :D 18:09:39 aren't the bans cleared on restart? > <@rucknium> Can some console command and/or RPC method be written to clear all bans? 18:10:00 @spirobel:kernal.eu: Yes, but often you don't want to restart. 18:10:28 Last stressnet, restarting was very annoying since the node re-validated the entire txpool on restart. That's fixed for this one. 18:11:24 I had a few out-of-memory crashes on an 8-GB RAM VPS with log level 2. 18:11:42 yes I notice bans occuring even when there is much load, seems to happen on my node when its syncing. And then its exacerbated b/c my node will try to connect to a node multiple times where I am banned and then my node bans then and its just a death spiral of bans. I will try to add more info to the issue as I find it but I a [... too long, see https://mrelay.p2pool.observer/e/4tbFlL8KblM1WXNu ] 18:12:52 Here it is: https://github.com/spackle-xmr/monero/pull/7 "Peer ban duration to 2 minutes" 18:13:28 Will add to v1.3 TODO list 18:13:41 thanks 18:15:06 Is there a specific goal with stressnet debugging, such as "network can smoothly handle txpool size X and block size Y"? 18:15:46 @rucknium: would it make sense to split the concept of peer bans from an exponential back off? usually in a distributed system you would do an exponential back off when a queue gets overloaded. 18:16:04 This isn't the final FCMP stressnet, of course, but as the issues are being chipped away, it could be good to know when the, uh, sculpture is considered complete. 18:16:27 monerod always feels ban-happy to me. 18:17:04 @spirobel:kernal.eu: this peer ban death spiral sounds more like this type of situation when a back off would make sense compared to a ban 18:17:16 Not yet, we didnt't have a specific mind in goal going into alpha, but you and everyone else have found a laundry list of good issues to work on in the short term. Perhaps future time period can have more focused goals 18:17:29 Occasional bans on mainnet wouldn't be as much of a problem. On a small stressnet, you are banning one of only a few possible peers. 18:18:02 All these improvements on network efficiency also harden Monero against spam and DDOS attacks > <@rucknium> Is there a specific goal with stressnet debugging, such as "network can smoothly handle txpool size X and block size Y"? 18:18:27 It's hard to control measurement of "smoothness" on an asynchronous network of peers which you don't control 18:18:50 But yes, I agree that more specific goals should be finalized before a final stressnet 18:20:08 would the final stressnet have any block verification improvements (or multithreading?) 18:20:51 it's still quite noticeable when mining as doing submit_block (or on miners) they will keep mining the old template for a while, as monero still has not verified the submitted block via RPC and this takes 5-20s 18:21:04 Yes, it should. There are a few big CPU-time improvements in review at the moment 18:21:08 this effectively makes that time in between "lost" for miners 18:21:37 For example: https://github.com/monero-project/monero/pull/10157 should help that issue w/ large mempools 18:21:47 how is your impression of the current peer ban list code? I just know from the wallet work, that rpc requests from the wallet also ended up on this ban list and it felt like the bans were sprinkled into the code base. > <@jeffro256> It's hard to control measurement of "smoothness" on an asynchronous network of peers which you don't control 18:21:58 IMO monero should do what bitcoin does and relay blocks after checking PoW but before verifying txs 18:22:07 Assuming the change concept is sound, #10157 will be extended for FCMP++ txs 18:22:28 There's also: https://github.com/seraphis-migration/monero/pull/101 18:22:36 that is done by P2Pool now boog900 18:22:51 @boog900:monero.social: monerod doesn't do that now? 18:22:51 so everyone benefits, not just P2Pool blocks 18:22:52 yeah I know but we shouldn't rely on that IMO 18:23:02 no, it has to verify transactions first 18:23:12 or sending bad blocks to other nodes causes them get banned 18:23:22 In most circumstances, I guess it won't matter much if everything works smoothly with fluffy blocks. 18:23:25 @boog900: I've also thought about this. It's not a bad idea. I wonder how it would affect network health with a miner of similar hashpower to Qubic ~33%, willing to cause disruption for no return 18:23:54 @rucknium: No, it verifies all consensus rules and input proofs before relaying 18:23:59 10157 looks good. specifically, if the received block is for the expected next height and all txs in mempool were verified to be valid, shouldn't this block have a quicker verification time? Or is the verification re-triggered? 18:25:20 DataHoarder: It should yes. Transaction input verification is only re-triggered if the dereferenced chain state for a transaction is changed, which will only happen in a >10-block reorg 18:25:45 then 10157 would cover these delays seen while mining, good. 18:26:57 If it works, yes, it should. 18:28:40 More on stressnet? 18:29:45 Thank you all community members who are running stressnet nodes 🧡 18:30:02 6. Mining pool centralization: Temporary rolling DNS checkpoints (https://github.com/monero-project/monero/issues/10064), Share or Perish (https://github.com/monero-project/research-lab/issues/146), and Lucky transactions (https://github.com/monero-project/research-lab/issues/145). 18:32:31 checkpoints, still waiting for fixes. If we can get a proper repro case (via modified submit_block on known monero state) it'd be great, that way can attach a debugger and look at it. 18:32:34 Longer term splitting of the code paths as written in other channels makes more sense later on 18:32:54 Qubic broke something last week on their side and while fixing it they re-enabled selfish mining for a few more days, before it being disabled. They managed several reorgs of varying depths (some 5-7?) with ~1.6-1.8 GH/s. 18:33:08 They currently stay at around ~1.2 GH/s and have decided to withhold bonus XMR payments from miners after a vote to attempt to prevent their price going lower. Mining/marathon schedule unchanged. 18:33:59 Thanks for keeping up with the Qubic movements, DataHoarder. What is "bonus XMR payments"? 18:35:55 Before, if price at end of epoch was > than an amount (at start of epoch) miners would receive some XMR (converted to their token) in their payments. this was part of their profit calculations 18:36:23 if it was below, it was instead converted to qubic and burned. now any xmr they find will always be burned. 18:37:23 (their people kept complaining that miners dumped too much qubic on each payout) 18:37:51 I don't think it's relevant to discuss for MRL, but an interesting note that has made them even less profitable compared to XMR or XMR+merge mining 18:37:51 They are reducing the incentive to mine with Qubic now? 18:37:58 DataHoarder: The classic bear raid approach 18:38:29 It was an incentive originally thought to reduce selling pressure (if you don't sell price will go up which means more Qubic for you next week and so on) 18:38:55 it ofc, when they attracted short term miners, caused more selling pressure 18:40:21 This was the reasoning on their side to why stop now, Rucknium: https://irc.gammaspectra.live/36558af247094870/image.png 18:40:37 but yes. there is no extra incentive except you get Qubic instead of something else 18:41:25 I think further questions around that could go onto -offtopic or -lounge if on topic enough ^\ 18:42:36 Thanks for the Qubic update. 18:42:48 For next week, should this go on the agenda? https://github.com/hbs/MoneroMisc/blob/master/CARROT-discussion.md 18:42:48 It suggests allowing user to not have an outgoing view key that can see outbound txs. 18:43:44 There was discussion in #monero-research-lounge:monero.social about it. 18:44:47 I say let the discussion in Lounge continue first. 18:45:58 If there is some loose consensus there then bring it here 18:46:21 There were some issues found with current schemes that would break current carrot construction if some of the keys were made the same, and the alternative was for users to keep using legacy wallets (with the lack of forward secrecy) 18:47:08 @articmine: Alright. 18:47:25 We can end the meeting here. Thank you everyone. 18:48:18 Thanks 20:04:44 articmine: Legacy wallets also get forward secrecy under Carrot (from an attacker who doesn't know a public address). They also get Janus attack protection. 20:05:18 My recommendation would be to allow users to generate a legacy wallet (with k_s and k_v) instead of a new wallet (with the new keys, including the outgoing view key). This has good plausible deniability because all wallets made before the fork are legacy wallets anyways. 20:07:01 New wallets should definitely not be constructed with k_ps = s_ga because: (1) Such use was not part of the audit. (2) Doing so loses internal forward secrecy. (3) It's impractical because it needs the private spend key to scan for owned enotes. 20:07:13 *k_ps = s_vb 22:49:12 On stressnet: taking stock and looking at the issues, there is a lot we can solve in the short term/medium term to improve daemon performance before we hit a wall. I think we should continue to knock out these issues and then re-assess upon hitting a wall 22:49:32 Clearly the daemon is having issues with a 300mb pool and 5mb blocks, so I think a rational immediate goal is for the daemon to not have issues at these levels 22:49:55 Examples of issues with clear fixes that we can solve short/medium term: 22:49:55 Example1: daemons dropping a connection that finds a fluffy block >4MB. That is a clear issue with a simple fix (that we've now rolled out in alpha stressnet v1.2) 22:49:55 Example2: daemons re-relaying every complete tx in their pool every x interval causing dropped connections. We know how to fix that (tx relay v2 massively helps, and IMO this PR is helpful too: https://github.com/seraphis-migration/monero/pull/174) 23:14:04 tevador: That sounds like the reasonable and practical way to make a new wallet without an OVK