00:02:52 i added it to the merge queue already 00:03:09 but not sure when luigi will get to merges 01:10:44 Is anyone else experiencing constant monerod crashing? 01:49:30 <1​23bob123:matrix.org> Yep 01:49:40 <1​23bob123:matrix.org> Out of sync too 01:56:22 Bad update or bad nodes? 02:21:38 Hard to say. Probably bad nodes. 02:21:45 Very high CPU usage. 02:22:59 My umbrel is fine 02:23:14 i am still synced 02:34:16 lots of 100Kb txs with ~150/2 in/out recently. Could that be causing higher cpu usage etc? 03:15:40 actually seeing 3 come through every block but not seeing them in the txpool beforehand (just keeping track via xmrchain.net) 03:49:12 Seeing absolutely huge CPU usage. Never seen it this high before, not even close. 04:06:20 @selsta are you experiencing the same? 04:09:40 We need urgent help debugging monerod's extreme CPU usage and unresponsiveness if anyone's available. 04:10:01 My master node crashed in the afternoon 04:10:02 is it a public node? 04:10:02 Causing some issue. 04:10:03 P2P (master) node is like **idle-2%** 04:10:04 RPC (slaves) nodes are like 15-20% of one core, each 04:10:05 When master "crashed" a few hours ago, nothing was in the log, and it was still running and even syncing, but the other nodes including my slaves node could not connect to it 04:10:06 Was not ressource related, I have no idea about what append, I just restarted it 04:10:45 did the node crash or did it get killed for running out of memory? 04:10:50 No. 04:10:59 I have seen a couple OOM, but no crash 04:11:19 No it's running at the moment, but using insane CPU since the whole 100kb+ txs started. 04:11:33 Super slow and unresponsive despite having enough CPU cores 04:12:02 Alex | LocalMonero | AgoraDesk Do you have a lot of RPC request on yours? 04:12:12 not seeing high CPU usage here 04:13:47 Yeah there's wallets connected, but it's not a public node 04:14:07 which version are the wallets running and how heavy is the RPC traffic? 04:15:20 Latest versions. 04:15:42 Is there any way I can PM you from matrix? 04:15:56 not possible 04:16:16 From what I see with mine, RPC greatly seam to amplify the CPU usage, like if RPC is only done by one core or something. 04:16:17 It's why I separated my node to have one for P2P only and a bunch for RPC. 04:16:47 yes, multiple RPC connections + a large txpool is currently not well optimized 04:21:43 Even with one RPC the CPU is insane. 04:21:54 One RPC wallet connection, that is. 04:22:38 what does that wallet do? just being connected? or tx creation and more? 04:23:01 Connects, syncs, reads txs, sends txs 04:26:11 does it connect and disconnect? 04:26:43 and are the wallets on the same network? 04:29:28 I can't discuss these details in the public chat. Is there any way I can PM you? 04:29:56 i won't be available until later, it's super late here 04:30:55 Very unfortunate. Maybe someone else can assist? This is really urgent and severe. 04:32:52 I could try to help, you can PM me. 04:37:28 I could help, if you have "many wallets" that RPC to the node 04:37:29 Assuming you have enough hardware, ie, make something like that 04:37:30 It does help but you need, fast ram, lot of it. Ideally 8 channels kind of Epyc CPU. 04:37:31 Even then the expanding mempool induce a slowdown for the RPC request... 04:37:32 And after what I did see today, I do recommand two "master" as mine did stop to accept connections earlier today when I was sleeping... 04:37:33 https://matrix.monero.social/_matrix/media/v1/download/xmr.mx/FMKigAiFNsgCPdAYZSPQQFaC 04:37:59 Alex | LocalMonero | AgoraDesk 04:39:05 All are full node, storage seam to be totally ok with one Nvme drive 04:40:18 It's also not possible to use that kind of abusive setup if you have only "one wallet" that do ton and ton of RPC call 04:40:33 Because it will all endup routed to one of the slave 04:44:33 There is way to route "each packets" to a different node instead of mapping connections by src ip:port 04:44:34 It might work for that, in that case if could work with "one wallet". But I did not test that setup as I normally map connection by src ip:port because that setup work for TLS unlike load balancing each packets. But who know, it might work for monero, would need to ask someone else 05:06:36 I don't know if a DEV (or someone who know) can answer me. 05:06:37 Question : If say 4 from the same IP endup connecting on the same node, does that will trigger that node to ban my IP? 05:06:38 From what I see today, I think I can ditch the master and let the other node connect directly to the others internet node instead of just being allowed to connect to the master node, then I can just also load balance incoming 18080. 05:07:08 I don't know if a DEV (or someone who know) can answer me. 05:07:09 Question : If say 4 nodes from the same IP endup connecting on the same random internet node, does that will trigger that node to ban my IP? 05:07:10 From what I see today, I think I can ditch the master and let the other node connect directly to the others internet node instead of just being allowed to connect to the master node, then I can just also load balance incoming 18080. 05:11:53 Seems to have resolved itself for now... Not sure what changed. 05:26:15 <3​21bob321:monero.social> Ctrl alt defeat 05:30:58 It's me or the asshole broadcast 3 x ~148/2 every time a block get mined 06:21:38 ravfx No, the newest 100 kb tx (146/2 - 150/2) in mempool is currently 3 hours old 06:30:14 ok, good to know, so xmrchain.net don't show all the mempool 06:31:20 That explain the discrepancy 06:31:21 https://matrix.monero.social/_matrix/media/v1/download/xmr.mx/LVQHEbmNuIxRATCJBBvNxQyR 06:31:22 https://matrix.monero.social/_matrix/media/v1/download/xmr.mx/hMywoaDeQwHgvmqGUIwxCjqP 06:31:36 yes, it probably running with 3 hour txpool timeout 06:32:15 https://p2pool.io/explorer/ shows everything as far as I'm aware 06:32:35 Will use that one instead, thanks 06:42:23 v0.18.4 then? the PR is mature and such a massive UX improvement its a shame to leave it sitting unmerged 07:24:48 +1 to the high cpu usage - saw highest CPU peaks on monerod with 4 "p2pool" instances connected to it 07:25:27 starting about 20:00 UTC to about a few hours ago (still high but within normal levels high) 07:26:58 notably it caused an even taller peak of read I/O 07:27:36 400x over baseline 07:28:11 (and some increased network traffic, which is probably just the transactions going around) 11:06:31 looks like automatic fee is still kinda broken in latest wave of txs 11:07:21 are there further improvements that can be made? 11:17:22 Automatic fee worked as expected, but it never increases the fee above normal level 11:17:34 Latest wave used normal level too 11:20:13 the wave where auto was broken? 11:24:55 When selsta fixed the auto fee adjustment bug to get it back to the intended behavior, I thought to myself "Is moving from 1st to 2nd tier enough? What about to 3rd or even 4th tier?" But the fix was included so close to a new release date. I didn't want to have us re-think the concepts and have to do more testing, etc. 11:26:00 Fee prediction is on my CCS research agenda. That doesn't necessarily mean that I will be able to fully research it during this CCS round. Depends on prioritization with other things. 11:29:00 yes by "kinda broken" i meant its not doing enough to function as user expects during latest mempool surge 11:29:37 please do. having fees that "just work" is such an understated part of using monero 11:29:52 good to hear. having fees that "just work" is such an understated part of using monero 11:31:19 Monero's rules increases the stakes for fee prediction. Wallets have to get it right the first time. Monero has no replace-by-fee nor child-pays-for parent. Once a tx is in the txpool, users cannot increase the fee if they made a "mistake" with their fee bid. 11:33:26 I make these points in Section 7 "Transaction confirmation delay" of my little paper: https://github.com/Rucknium/misc-research/blob/main/Monero-Black-Marble-Flood/pdf/monero-black-marble-flood.pdf 11:35:31 Fees "just work" when there is no congestion :). The Monero ecosystem's fee selection is primitive. 11:37:58 But these considerations make the question _interesting_: 1) Want tx uniformity for privacy, 2) No way to change tx fee bid, 3) Dynamic relay fee/block size/miner penalty rules, 4) To expand the block size, there _must_ be a period of txpool/blockchain congestion. 13:52:43 Auto fee just work when monero is not under consolidation attack right... 13:52:44 We we bump fee higher in auto selection algo, the attacker will just use higher tier fee for his consolidation attack and we are back to the same problem. 13:52:45 He seam to always use low fee for his 1/2 attack. 13:52:46 Main issue is people not updating there wallet, helped another one yesterday, he send a TX with low fee with monero gui 0.18.3.1. Monero gui show a nag in the middle of the window that ask you to update.... It's been over a month. 13:56:17 ravfx: Normally the auto-updater pops up upon startup 13:57:05 could be that, could be a good idea to check for update every day or so without having to restart the wallet (for the people that leave it on) 14:03:59 I sent a min fee tx during the large mempool and the CLI stated that it would take 7 blocks at that fee level and was that ok, meanwhile it took 211 blocks 14:04:51 no problem as I was sleeping and I figured it would take that long, just reporting incase anyone thinks it needs refinement 15:09:50 Yup. The CPU usage was way out of proportion. It's as if there's some kind of bottleneck. 15:15:51 I would leave the auto as is. Block grow as needed (we saw that yesterday). 15:15:52 Allow people to set tier 3 as needed if said user want to send to a payment processor or exchange now 15:15:53 having auto to auto scale to 3 or 4 will endup making bigger block faster and will give more room for attacker to flood black marbles and bloat the chain. 15:52:25 same here nioCat, the wallet's estimated backlog was way off. told me no backlog at priority 2, while hundreds of those maxed-out, 100kb consolidations cut in front of me, paying an extra 2-4 nanoneros per byte 15:55:34 for me it wasn't an issue of "they" using a custom fee as I was using min fee, or was it? 16:20:42 related comment: in simplewallet.cpp, only transfer_main and print_fee_info call estimate_backlog. sweep_main and sweep_single don't seem to check for backlog at all. i think, even if users are just churning to themselves and don't mind waiting a couple hours, they should be made aware of the potential delay 23:29:28 Hello, is this the wrong place to ask for advice? 23:31:26 sorry, just read this Non-dev questions -> #monero