14:02:01 I added more aggressive scaling examples: https://github.com/monero-project/research-lab/issues/152#issuecomment-3461700149 15:01:32 MRL meeting in this room in two hours. 17:00:20 Meeting time! https://github.com/monero-project/meta/issues/1289 17:00:25 1. Greetings 17:00:39 Hi 17:00:57 hi 17:01:09 hello 17:01:48 hello 17:01:54 waves 17:02:49 Howdy 17:02:51 2. Updates. What is everyone working on? 17:03:21 me: Looking at some changes in spy node behavior. 17:03:42 working on a lws db/msgpack bug. unable to reproduce locally, so I may have to move on unfortunately 17:04:46 Reviewed p2p SSL https://github.com/monero-project/monero/pull/8996 (thank you @vtnerd:monero.social), continuing stressnet investigating / bug fixing (found and patched an issue in monerod where it over-counts pool weight in memory) 17:05:04 oh yeah, updated that pr too! looking good hopefully 17:05:34 I had to pause my scaling work to review this 10% scaling proposal 17:05:58 I have not had a chance to look at the latest version 17:06:50 3. Spy nodes https://github.com/monero-project/meta/issues/1124 17:06:58 Me: working on extending integration of Carrot devices and Carrot-derived account keys in wallet2 17:07:29 The goal is for Carrot-derived wallets to be available before the beta stressnet 17:07:59 As well as hybrid wallets 17:08:01 @boog900:monero.social noticed that the spy node test is showing some spy nodes as false negatives, starting about a week ago: https://moneronet.info/ 17:08:23 They finally updated ;) 17:08:46 Nice of them to wait until after the p2p selection improvements were reviewed and released 17:08:47 These nodes are still on the same IP addresses. About 100 in DigitalOcean and 100 in Hetzner. 17:09:06 The LionLink spy nodes have the same behavior as before. 17:10:17 The MRL ban list is still valid. The network should be monitored for "metastasizing" behavior of the spy nodes, i.e. new spy nodes without the peerID fingerprint 17:10:36 But, the spy nodes may still be detectable with other tests. 17:11:34 This recent paper monitors packet data: Kopyciok, Y., Schmid, S., & Victor, F. 2025. Friend or Foe? Identifying Anomalous Peers in Moneros P2P Network. https://moneroresearch.info/280 17:11:57 and finds a lot of anomalous behavior. 17:13:05 Their code is open source, so I could try to run it. They use regular passive Monero nodes to collect the data, so it won't be as fast or complete as the network crawler scan that moneronet.info uses. 17:13:27 AFAIK, @syntheticbird:monero.social discovered another way to deduce spy nodes. 17:14:23 I could push Core to change the DNS ban list to include more spy nodes: https://github.com/monero-project/meta/issues/1242 17:14:40 IMHO, this issue ^ is mature and has had time for comments. 17:16:21 I set up moneronet.info because it shows pretty graphs and because it helps reveal changes in spy node behavior. It's doing its job 😁 17:17:00 Any more comments on spy nodes? 17:18:04 a picture says 1000 words, nice 17:19:18 4. Transaction volume scaling parameters after FCMP hard fork (https://github.com/ArticMine/Monero-Documents/blob/master/MoneroScaling2025-07.pdf). Revisit FCMP++ transaction weight function (https://github.com/seraphis-migration/monero/issues/44). Simple Market-Based Monero Fee Proposal (https://github.com/monero-project/research-lab/issues/152). 17:19:47 Yes, but it isn't deterministic. Under some circumstances there could be false positives, but depending on some criteria we can limit them to probably zero. I'll need to go into some archives as librejo is now down and everything was collected on a private repository. > <@rucknium> AFAIK, @syntheticbird:monero.social discovered another way to deduce spy nodes. 17:20:38 s/deterministic/100% accurate 17:20:38 This discussion topic continues to be challenging. If there are suggestions about how to structure the discussion (especially in the future), please bring them :) 17:21:08 As I mentioned before I do not have to report anything new since I believe that addressing this new 10% proposal is required 17:22:13 I added two more aggressive scaling examples to my proposal. My proposal aims to simplify scaling and fees. Other parameters can be chosen, so feel free to ask me for different numbers. 17:22:13 10% is my suggestion and one example parameter, but any other value (e.g. 50%) can be chosen if desired. I included numbers for 25% and 50% this morning 17:23:11 I agree that this topic is challenging primarily because of the sheer amount of fud and misinformation that has been imported from Bitcoin and related coins 17:24:58 As for this 10 % proposal my preliminary assessment is that it is a complete disaster and also a serious breach of the Monero's can Cryptonote social covenants 17:25:35 I think there are 2 core problems to solve on this front: 1) tx weight, and 2) scaling algo. Personally I would prefer to tackle the former first, and the latter second. They can be solved together in one pass, but I think we'd end up stuck in bikeshedding hell trying to 17:26:57 @jberman: Actually the scaling algorithm need to be addressed first 17:27:17 At present, Monero's scaling design gives no consideration to the capabilities of the daemon. Regardless of the path chosen, I believe this must change in the future. 17:27:32 Since what we are ultimately dealing is fee incentives 17:28:05 My proposal works for any transaction weight calculation, so it would be fine to handle those separately from that perspective 17:28:12 sorry fkr dragging off topic, but what is "weight" ? Is this just some made up concept that is a combination of various factors, but then given a value? 17:28:29 (since its not the byte size on the tx, im not sure what its supposed to be) 17:28:40 @sgp_: No it doesn't 17:29:35 weight can be calculated by number of elements, byte size, or estimated cost of verification etc. 17:29:44 Monero seems to have weight and size used interchangeably throughout the codebase, but apparently they aren't the same thing 17:29:47 @ofrnxmr: Yeah basically. The idea of "weight" is a value assigned to how much space a transaction should take up in a block, and thus indirectly, what ratio of fees it should pay compared to other txs 17:30:13 @ofrnxmr: Yeah we love using inconsistent terminology in the codebase 17:30:17 weight = size + other relevant factors, whatever those are 17:30:21 on current mainnet BP+ txs, weight is byte size,with exception to BP+ txs with >2 outputs, which add marginally larger weight for the BP+ clawback 17:30:39 monero used to use size until verification became more costly for some things than others, that's why it changed 17:30:40 @jeffro256: ban & block 🥲 17:30:47 But IIRC, weight was the same as byte size until the first BP fork 17:31:42 Ok thanks 17:32:11 I don't have anything else to discuss from my end for this topic 17:32:23 (this of note because sum of fcmp txs != advertised block size) 17:32:45 @sgp_: Same here 17:33:12 So the "1.4mb" block we see, only has like 800kb of txs. And a 130kb tx has a weight of like 700kb 17:33:25 the core pushback on weight = roughly byte size for FCMP++ comes from @kayabanerve:monero.social, who argues this incentivizes larger txs over n broken up txs, to spend the same number of inputs 17:33:35 @jberman: I agree with j-berman on his point to try to determine weight now 17:34:07 And I also think it should be approx eq to byte size b/c of the arguments enumerated last week 17:34:10 kayaba also expressed weight roughly = byte size would not be a blocker 17:34:43 on the subject of scaling, im firmly against uniformity taking precedent over scaling. Scaling is more important than uniformity 17:34:55 @jeffro256: I can do this based upon my proposed scaling 17:35:06 I think the argument for weight roughly = byte size (simplicity, minimize byte size spending n inputs, minimize n outputs on the chain to spend n inputs), is sound enough to warrant the decision 17:36:22 Maybe someone should calculate how much developer labor costs it would take for the service that needs to do lots of consolidation txs to develop and deploy the ideal tx algorithm, compared to just accepting a little higher fee costs for suboptimal consolidation. And then compare the actual costs to node operators for storing those txs, in the two methods. 17:36:23 I think participants today have rough consensus on weight roughly = byte size 17:36:30 I think the difference would not be large. 17:36:46 @jberman: Yes 17:37:25 would pruned size (long term size for "small" nodes) be considered for weight? 17:38:04 No 17:42:00 @rucknium: Is this analysis required to come to rough consensus here in your view? 17:42:05 Has the discussion about tx weight concluded? (Or the minority voices been worn down enough to give up 😉 ?) 17:42:35 @jberman:monero.social: No. My point is that the points of contention are small details that don't matter much. 17:43:02 Ya I agree with that as well 17:43:53 > Or the minority voices been worn down enough to give up 17:43:53 I think simplicity in design over tinkering with minutiae details is a huge plus here to move forward with 17:44:20 To continue ad absurdium, you could compute the labor cost to computing those figures, and then compute the cost to compute the ith, etc. 17:44:56 I doubt there would be a large empirical effect if one option were chosen compared to the other. 17:45:52 Linear increasing fees was probably the main alternative, and then we end up stuck deciding how 1 input/output/extra should count toward the weight, and rehash similar arguments 17:46:56 All this to say, I think there is strong merit in the simplicity of (and arguments for) weight roughly = byte size, such that the detracting views are not strong enough to warrant deviation from it imo 17:47:19 Lets move on to the next item. I will see if I can come up with more structure for the scaling discussion for next time. In the meantime, please feel free to share ideas for a structure. 17:47:34 5. FCMP alpha stressnet. https://monero.town/post/6763165 17:48:37 I'm not a real programmer, but it seems to me that monerod is full of technical debt. 17:49:34 seen worse. its still manageable, but others may have different thoughts 17:49:37 @rucknium: Yes, I agree and this debt needs to be repaid before the hard fork 17:49:55 I'm a programmer, I agree, specially in the older parts of the codebase. They are usually also uncommented and have generic naming 17:50:10 there are some oddities with epee and how that hooks into things, I wish that was done a little differently. the refactor of the tcp server probably helped 17:50:21 I recognize that this may require additional time and resources 17:50:41 ofrn and I discussed blocker issues for mainnet launch: 1) OOM after synced (we're going to figure this out), 2) pool consistently cutting down to 1 tx (hopefully solved with the latest bug fix), 3) wallet complaining about double spends 17:50:46 Including CCS to fund the repayment of this devt 17:51:11 Debt 17:51:31 I think @jeffro256:monero.social and @jberman:monero.social 's fixes to the stressnet code created other problems, like OOM (RAM exhaustion). Fixing problems causing more problems seems to be a symptom of bad initial architecture. 17:51:52 Ends up with fun side discovery trips like ge_fromfe_frombytes https://web.getmonero.org/resources/research-lab/pubs/ge_fromfe.pdf 17:52:02 Its not just making the changes, its getting them reviewed and merged 17:52:07 The OOM was there from the start, we think it's related to something from the FCMP++ integration, potentially the FFI 17:52:21 Monerod doesn't really move that fast, for good reason 17:52:22 which now got fully resolved by kayaba&others 17:53:38 @boog900: It is a lot of work that takes time. This needs to be recognized 17:53:39 > Fixing problems causing more problems 17:53:39 Just judging the stressnet channel, it seems to me like we have noticed actual improvement on connection issues and such. So it doesn't look like problems are increasing to me, but maybe you guys have a different experience @ofrnxmr:monero.social @rucknium:monero.social ? 17:53:54 there's also the (probably coming at a later point) DNS checkpoints being intertwined with block check code which ends up hard blocking nodes when used on specific thresholds 17:54:31 I mean the OOM on sync catch-up. My nodes with 8GB of RAM OOM'ed a lot on sync catch-up recently. 17:55:16 I think major improvements have been done but the underlying technical debt is seen on mainnet as well 17:55:17 @rucknium: The ooms have been there since the beginning 17:55:24 Some of the connection problems are related to hard-coded limits. You can raise the limits, but that doesn't solve the architecture problems. 17:55:34 @rucknium: This has existed on master/mainnet forever 17:55:54 it's just surprises when things that have been with us all along end up not working fully, either to breakage (or never worked properly) 17:55:58 @jberman: I also thought that this was the case. My stressnet node has been taking up excessive RAM since v1.1 17:56:01 Its just not noticable due to checkpoints 17:56:27 I think splitting what is FCMP++ issues vs mainnet would be a starter towards fixing these for beta stressnet? 17:56:27 I don't mean that the fixes caused the problems directly, but the fixes can reveal another lay of problems. Sorry to have worded that poorly. 17:56:57 another layer* 17:57:07 Some of the fixes that we think are upstream issue have been adding to the FCMP++ upstreaming plan: https://github.com/seraphis-migration/monero/issues/103 17:57:10 @rucknium: Node mines >4mb fluffy block, relays it to another node that doesn't accept >4mb blocks because of hardcoded limit. We now relay empty fluffly block as expected, and the limit shouldn't be 4mb anyway 17:57:12 @rucknium: Its neither. Its that the blockchain has grown and extended periods of large blocks are exposing the long standing broken spans 17:57:24 yeah. symptom vs actual problem 17:57:30 DataHoarder: They are, for the most part 17:57:38 @jberman: The architecture is established to allow 0 tx fluffy blocks to relay, but we weren't using it 17:57:56 The last stressnet was when i showed the runaway spans to boog and 0xfffc 17:58:45 Some issues are definitely architectural and require significant structural changes. For example, I personally don't think daemons should keep any counter state in memory in sync with db, because that is bug-prone. But there are little bugs that are fixable like counting correctly that allow us to move forward 17:59:20 @ofrnxmr: I couldn't figure out the problem at the time, but I did find another issue. I think now I have found the problem: https://github.com/seraphis-migration/monero/issues/147#issuecomment-3446862454 17:59:22 @ofrnxmr:monero.social: Yes. I think about how rough last year's stressnet was, even after the small fixes that prevented netsplits. It's more than a year later. I don't know where the labor and expertise will appear to fix all these things in a short period of time so the node can scale smoothly. 18:00:29 I can bring this to stressnet channel later, but do we want to do one last stressnet run with leak detectors or other tools engaged to gather data? for old or new issues 18:01:15 We have been, ASAN isn't proving helpful unfortunately 18:01:17 @rucknium: Boog found a vuln associated with runaway spans. 0x attempted to fix them in the dynamic span pr. But they still arent reliably solved 18:01:35 DataHoarder: I think that it would be great if someone could compile monerod in debug mode and set their OS to save core dumps on OOM then send me or Berman the core dump 18:01:47 Sorry for being late. I've been working on some Ed25519 code and optimizations. I'll also note, somewhat as a circle-jerk but with relevance due to jeffro256's commentary, I've increasingly been working in code which doesn't allocate at all for usage within monero-oxide, and @boog900:monero.social: has done remarkable work on a more efficient DB. 18:01:55 The dynamic span pr seems to work to keep the spans below 1gb, but it still runs away a bit, just not to 8gb 18:02:27 It's probably not a "true" memory leak in the sense that the memory is still accessible, but something somewhere isn't being pruned. ASAN won't catch the former, but hand-inspecting the latter might 18:02:46 @jeffro256: I havent had an running oom in a while and it seems to happpen when the txpool is huge 18:02:49 19:02:09 DataHoarder: I think that it would be great if someone could compile monerod in debug mode and set their OS to save core dumps on OOM then send me or Berman the core dump 18:02:49 ^ I can do that, I ran monerod in debug mode during first stressnet days... but I'd have to move this elsewhere, as the OOM is killing more important things 18:02:57 And only when the txpool is huge 18:03:04 unbounded memory growth :) 18:03:22 would relwithdebug info help? 18:03:42 only happening when tx pool is huge is actually interesting new info 18:03:49 @jeffro256: it could very well be the txpool.. 18:04:30 oh woah 18:04:33 Is this not just the OOM that txrealy v2 is trying to fix? 18:04:38 DataHoarder: Debug build would probably be more helpful, but much, much slower to run / sync 18:04:42 @boog900: Could be 18:05:08 I have two core dumps of fcmp monero 18:05:08 on debug mode 18:05:08 from the 1st of October 18:05:34 Was that OOM'ed too? 18:05:44 That was before the fork tho, yeah? 18:05:49 have you had one since v1.3? > <@ofrnxmr> I havent had an running oom in a while and it seems to happpen when the txpool is huge 18:05:56 yea, large txpool during spam attack caused nodes to repeatedlt oom 18:05:57 probably it was during db conversion 18:06:00 @jberman: No 18:06:04 I would need to check system logs, it's SIGABRT 18:06:12 @ofrnxmr: Or. maybe.. 18:06:31 since v1.3 I had two OOMs and one preventive restart from me 18:06:31 where monerod was using 40+ GiB on stressnet with no RPC usage 18:06:42 I told you the last time i had one. Not sure the date on that, but the txpool went from 20mb to 200mb when it did 18:07:20 I think we can discuss all of this on stressnet channel, if it's FCMP++ monero stressnet specific issues 18:07:43 the running oom is probably tx-relay 18:07:53 I think boog's right 18:08:35 @0xfffc:monero.social pushed an update addressing @vtnerd:monero.social 's review on master 18:08:46 Ok, generally I think we're going to need tx relay v2 and caching input validation (which is ready), and to observe alpha stressnet perf after both of those are in 18:08:55 I dont know if its "ready", but ive run it and it seems ok 18:09:22 Let's count resources. Who thinks they are able to fix deep performance issues in monerod? Who would be willing to dedicate a lot of time to that in the next 6 months, 1 year, 2 years, etc.? Who thinks they could recruit new contributors who could and would do it? 18:09:25 If we see significantly improved behavior after that point, then I'd vote to move toward beta stressnet 18:09:48 And there is a need to review proposed changes. What are the resources for that? 18:10:20 @ofrnxmr: Main issue is that it still has amplification, as it requests missing txs from every peer at once. I think sequentially would be better, but other might disagree (?) 18:10:26 you'd probably at least 2x of the resources for making the changes and then review 18:10:31 This ^ type of work would be "good" to have before the FCMP hard fork, but it's most relevant for scaling discussions. 18:10:38 specially if doing modifications/refactors as needed 18:10:58 Paging @perfect-daemon 18:11:12 @jberman: shhhhh 18:11:41 @jberman:monero.social: Well, need something to be reviewable. 18:12:42 I'm fine with reviewing whatever. I would personally like to prioritize FCMP++, Serai, and async scanner 18:12:44 @ofrnxmr: @ofrnxmr:monero.social: run which one? Caching input validation or tx relay v2? 18:13:17 @jeffro256: V2 18:13:31 Havent tested the caching input pr 18:15:00 With amplification, its still about = to boogs calculated 75% savings. W/o amplification, i think were upwards of 95% > <@ofrnxmr> Main issue is that it still has amplification, as it requests missing txs from every peer at once. I think sequentially would be better, but other might disagree (?) 18:15:52 More discussion about stressnet? 18:16:49 I'll throw on that I rewrote my RPC code for Monero and it should be a fascinating presentation of why the RPC is frustrating to use as-is, circling into vtnerd's prior commentary on debt and the HTTP server. 18:17:09 I think its going well, and the IBD oom is a master issue that needs to be solved at some point (and doesnt req a hf to fix) 18:18:49 @0xfffc:monero.social: 9933 is ready for re-review? 18:18:57 I'll ask 0x to update & rebase the dynamic span pr and i'll retest it against stressnet.. it wasnt well received in review, but the general idea seemed to work well enough 18:19:27 @kayabanerve:matrix.org agree, reminds me of the other discussions a few weeks ago about undocumented areas or inconsistent usage or bad data on specific calls 18:19:28 <0xfffc> @jberman: Yes, it has been reviewed once by @vtnerd:monero.social . Of course there are still problems. ( since it is a huge PR ). But it is ready for review / testing. 18:19:31 @jberman: I believe so. 18:19:54 dynamic span PR is in stressnet already and seemed ok to me. It was the restricted RPC PR that I had some thoughts on that we didn't include in stressnet 18:20:19 Dynamic bss is in stressnet,not dynamic span 18:20:36 ah, misread 18:20:58 Dynamic span was intended to curb the runaway spans by stopping download if sync time exceeded a set value 18:21:59 By default, only downloading what can be synced within 2mins. (it still runs away occasionally, but not to 8+gb) 18:22:53 https://github.com/monero-project/monero/pull/9495 18:23:47 Yeah that PR didn't change the stripe_proceed_main condition IIRC, where I am almost certain the issue is 18:24:02 ok, I think I will pause continuing on OOM while already synced and focus on reviewing 9933 instead, and will review the issue boog pointed out for runaway spans 18:25:46 > <@jberman> ofrn and I discussed blocker issues for mainnet launch: 1) OOM after synced (we're going to figure this out), 2) pool consistently cutting down to 1 tx (hopefully solved with the latest bug fix), 3) wallet complaining about double spends 18:25:46 but generally I think we want to solve these issues before mainnet launch. And once tx relay v2 + cache validated txs is in and perf observed, we decide on beta stressnet 18:27:45 6. Mining pool centralization: Temporary rolling DNS checkpoints (https://github.com/monero-project/monero/issues/10064), Share or Perish (https://github.com/monero-project/research-lab/issues/146), and Lucky transactions (https://github.com/monero-project/research-lab/issues/145). 18:28:10 i think we should release 2) (195) on stressnet and see if anyone runs into the issue again 18:28:54 typo? 18:29:00 Im running on a few nodes, but its not a consistent condition to reproduce, but atleast 3 people on 1.3 have hit it 18:29:23 195 = the fix for the double counting of txs 18:29:37 agree wasn't sure what the 2) was before 18:30:09 still blocked by the issue on monero around hitting a checkpoint in the wrong place 18:30:19 we can get a release out today. Possibly will include cache re-validation too 18:30:23