15:01:54 <m-relay> <r​ucknium:monero.social> MRL meeting in this room in two hours.
17:00:30 <m-relay> <0​xfffc:monero.social> Hi everyone
17:00:43 <m-relay> <a​rticmine:monero.social> Hi
17:00:45 <m-relay> <r​ucknium:monero.social> Meeting time! https://github.com/monero-project/meta/issues/1020
17:00:53 <m-relay> <r​ucknium:monero.social> 1) Greetings
17:00:58 <rbrunner> Hello
17:01:18 <jberman> hello
17:01:23 <spackle> hi
17:02:44 <m-relay> <r​ucknium:monero.social> 2) Updates. What is everyone working on?
17:02:58 <m-relay> <c​haser:monero.social> hello
17:03:53 <m-relay> <v​tnerd:monero.social> Hi
17:03:56 <jberman> me: trim_tree for the fcmp tree
17:04:06 <m-relay> <r​ucknium:monero.social> me: Working a little more on the black marble optimal ring size and fee analysis. Helping spackle with setting up the new stressnet. Working on my MoneroKon presentation "Hard Data on Banking the Unbanked with Cryptocurrency".
17:04:34 <m-relay> <a​rticmine:monero.social> I am finalizing the scaling changes. Will be presenting them at MoneroKon, on Friday.
17:04:57 <m-relay> <b​oog900:monero.social> Hi
17:05:24 <m-relay> <v​tnerd:monero.social> Me: testing the LWS remote scanning feature, which led to a few bug fixes
17:06:13 <m-relay> <v​tnerd:monero.social> Although the bugs were unrelated to the feature, and required some backporting to release branches
17:06:15 <m-relay> <0​xfffc:monero.social> me: worked on few reviews here and there. Worked on a fix for duplicate transactions in fluff queue. Had no familiarity with fluff and stem, so took a long time to understand the code, and finally it is done. Initial planning with spackle and rucknium on monero stressnet and monerod torture test.
17:06:57 <m-relay> <r​ucknium:monero.social> 3) Stress testing `monerod` https://github.com/monero-project/monero/issues/9348
17:07:02 <spackle> me: running a testnet fork for stress testing
17:07:16 <m-relay> <r​ucknium:monero.social> spackle, do you want to explain what you've been working on?
17:08:12 <spackle> I've made a few attempts at stress testing monerod
17:08:39 <spackle> Running a 10 node private testnet on a single machine did not stress things as desired.
17:09:12 <spackle> So I have forked the testnet to create an abusable network for testing monerod.
17:09:19 <spackle> The 'stressnet' is running now (https://github.com/spackle-xmr/monero)
17:09:26 <spackle> The next important step is creating a release + binaries for others to run. Gitian is new to me, and I am having some difficulties with the process.
17:10:11 <m-relay> <r​ucknium:monero.social> Recently we got the txpool to almost 200MB. More than 16 hours of backlogged txs.
17:10:24 <spackle> That said, things are running fine and I have been able to see some of the limits of the daemon with some early spamming. Even with low connection counts, performance limits have been observed.
17:11:47 <m-relay> <r​ucknium:monero.social> The monerod with the v17 hardfork for testnet self-compiles and runs fine. The harder part is trying to have a real build process for the binaries on all the operating systems so people who can't/won't compile can join the network if they want.
17:12:44 <m-relay> <r​ucknium:monero.social> I think we saw problems with tx and block propagation already with just 6 nodes.
17:13:25 <m-relay> <r​ucknium:monero.social> plowsof and selsta have helped with trying to troubleshoot the gitian build process :)
17:14:36 <spackle> Which is much appreciated, of course.
17:15:50 <m-relay> <r​ucknium:monero.social> I think 0xfffc  is going to help with how to measure performance precisely and try to figure out where in the code the bottlenecks are. I think what we have is a workable base.
17:16:00 <sech1> is there any write up of the problems? What was the bottleneck - CPU, memory, network, SSD?
17:16:21 <m-relay> <0​xfffc:monero.social> Yes. Once we got the stressnet, the other side (debugging/profiling) will be my responsibility
17:17:12 <m-relay> <0​xfffc:monero.social> That yes was reply to this message.
17:17:39 <m-relay> <r​ucknium:monero.social> sech1: I think at this stage we are just experimenting and making sure that the network is running correctly and the spam script works. And figuring out the rough limits.
17:19:09 <m-relay> <r​ucknium:monero.social> spackle's node that received the spam got to 17GB of RAM at one point. Before we had a hypothesis that maybe monerod's LMDB was loading the DB into RAM and that's why some people were seeing too much RAM usage. But the testnet DB is less than 10 GB, so that hypothesis seems unlikely now.,
17:19:30 <m-relay> <r​ucknium:monero.social> "A lot of 150/2 transactions in the txpool causes memory spike / OOM daemon" https://github.com/monero-project/monero/issues/9317
17:19:37 <m-relay> <b​oog900:monero.social> I made a small program using Cuprates P2P stack to make and maintain loads of connections to a single node, my plan is to pop blocks back to around when nodes crashed and start pushing txs from the blocks after to the txpool
17:20:06 <m-relay> <r​ucknium:monero.social> My hypothesis is that something about preparing txs to be sent p2p and to wallets uses RAM, and then the RAM isn't released maybe.
17:20:26 <m-relay> <b​oog900:monero.social> the connections don't do anything, just enough to stay connected, but monerod will still fluff txs to them
17:23:11 <m-relay> <a​rticmine:monero.social> What kind of TPS are we talking about here?
17:24:51 <m-relay> <r​ucknium:monero.social> I think spackle was pushing 5 tx/sec. It was pushing up txpool size since blocks can't include that many txs at that rate of course
17:25:00 <spackle> For the stressnet? In my trials I have used a single daemon instance with an rpc wallet. That setup is limited to creating ~15tps, and I believe that entirely occupies the daemon.
17:25:22 <m-relay> <0​xfffc:monero.social> Once we had stressnet (which is very useful idea), it is going to be much easier to debug the monerod code and find bottlenecks. I would focus on getting stressnet going as first step. ( Thanks to spackle and ruck )
17:25:23 <m-relay> <r​ucknium:monero.social> spackle seems to have a configurable spammer. It can change the tx/sec and the fee of the txs
17:26:48 <spackle> During tx creation the daemon connected to the rpc wallet would not be able to send txs to other nodes unless there was a break in the spam. At times, there was 40MB of txs that were not relayed.
17:26:56 <m-relay> <r​ucknium:monero.social> Anyone can join the stressnet by compiling this monerod and running as `--testnet`: https://github.com/spackle-xmr/monero . Ignore the releases because we are still not getting the releases right.
17:27:42 <m-relay> <r​ucknium:monero.social> `--max-connections-per-ip=1000` is also recommended since there are a few nodes running from the same IP
17:28:10 <m-relay> <v​tnerd:monero.social> Rucknium:  there's a couple of vectors and deque where shrink to fit could be used. Seems unlikely but worth a look
17:32:16 <m-relay> <v​tnerd:monero.social> Or no deque, just a vector I think
17:32:52 <jberman> sounding like a productive endeavor, thank you guys
17:34:42 <m-relay> <0​xfffc:monero.social> One of the areas that needs that stressnet is locking. My rwlock, initially introduces something like %15 speed up compared to baseline, but later it causes slowdown of %10 compared to baseline. (As sgp tested it under their heavy usage node). Once we had stressnet, I can run it long time and find out the reason for slowdown for rwlock. Right now I don’t have access to any kind <clipped message>
17:34:43 <m-relay> <0​xfffc:monero.social> of heavy usage node.
17:34:59 <m-relay> <a​rticmine:monero.social> I agree. This is a very  productive endeavor. Thank you.
17:37:00 <m-relay> <r​ucknium:monero.social> Special thanks to spackle!
17:37:06 <m-relay> <r​ucknium:monero.social> 4) Potential measures against a black marble attack https://github.com/monero-project/research-lab/issues/119
17:40:21 <m-relay> <r​ucknium:monero.social> I write two more solution concepts for my model. The solution concept in the current paper draft is the ring size/fee optimizers of cost effectiveness when you have certain inequality constraints. Two other ones: 1) Best cost effectiveness when you have the _equality_ constraint of a certain effective ring size (i.e. must be on the effective ring size line). 2) Give Alice a budget<clipped message
17:40:23 <m-relay> <r​ucknium:monero.social>  constraint. Alice has to spend less than a certain amount on aggregate tx fees and node storage costs. That's another inequality constraint to the main model. I don't have results to share about those two solution concepts yet.
17:43:25 <m-relay> <c​haser:monero.social> great! I don't know how helpful that is, but I think a constraint on the tx generation/verification time would (also) be useful
17:44:02 <m-relay> <r​ucknium:monero.social> That could be a good idea, too. Thanks.
17:44:11 <m-relay> <r​ucknium:monero.social> 5) Research Pre-Seraphis Full-Chain Membership Proofs. https://www.getmonero.org/2024/04/27/fcmps.html
17:46:29 <m-relay> <r​ucknium:monero.social> Any updates on FCMP to discuss?
17:46:38 <jberman> no news from me
17:47:19 <m-relay> <r​ucknium:monero.social> Any other agenda items?
17:50:44 <m-relay> <r​ucknium:monero.social> I have a question maybe someone could answer. When exactly does monerod verify the expensive cryptography in a tx? When nodes get a new tx from peers, there is some verification, but from my view of the code it looks like it may be quick hash verification. But probably nodes do not wait until they get a fluffy block to do the full verification.
17:52:00 <m-relay> <r​ucknium:monero.social> To me it looks like nodes do the hash verification on every tx that they get from a peer even if they have seen the tx from another peer before. If they are doing the full verification at that step, that's more reason to consider boog900 's suggestion to reduce duplicate tx gossip messages.
17:53:26 <m-relay> <r​ucknium:monero.social> This log message fires whenever nodes get new txs from peers: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L990
17:54:02 <m-relay> <r​ucknium:monero.social> Which seems to run this hash verification function: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_basic/cryptonote_format_utils.cpp#L253
17:56:17 <jberman> in here: https://github.com/monero-project/monero/blob/c8214782fb2a769c57382a999eaf099691c836e7/src/cryptonote_protocol/cryptonote_protocol_handler.inl#L1038
17:56:30 <m-relay> <r​ucknium:monero.social> I have done most of the parsing work of the p2p logs that were running during some of the spam. This question formed when I was looking at the data since I get all these messages about each tx being added (to something).
17:56:54 <m-relay> <r​ucknium:monero.social> Thanks!
17:58:39 <m-relay> <r​ucknium:monero.social> We can end the meeting here. Thanks everyone.
17:58:58 <m-relay> <0​xfffc:monero.social> Thanks everyone
17:59:02 <m-relay> <c​haser:monero.social> thank you all for your work!
18:00:23 <m-relay> <v​tnerd:monero.social> https://github.com/monero-project/monero/blob/cc73fe71162d564ffda8e549b79a350bca53c454/src/cryptonote_core/tx_pool.cpp#L263
18:00:23 <m-relay> <v​tnerd:monero.social> I believe is where the expensive crypto checks are performed. This will only occur if the tx isn't in the txpool already
18:05:15 <m-relay> <v​tnerd:monero.social> https://github.com/monero-project/monero/pull/9135 is a review that should speed up new tx processing a bit. A reminder (mainly for myself) to review as this could/should be shipped by now