i added it to the merge queue already

but not sure when luigi will get to merges

<a​lex:agoradesk.com> Is anyone else experiencing constant monerod crashing?

<1​23bob123:matrix.org> Yep

<1​23bob123:matrix.org> Out of sync too

<t​bcode:matrix.org> Bad update or bad nodes?

<a​lex:agoradesk.com> Hard to say. Probably bad nodes.

<a​lex:agoradesk.com> Very high CPU usage.

<w​hiskey00:matrix.org> My umbrel is fine

<w​hiskey00:matrix.org> i am still synced

lots of 100Kb txs with ~150/2 in/out recently. Could that be causing higher cpu usage etc?

actually seeing 3 come through every block but not seeing them in the txpool beforehand (just keeping track via xmrchain.net)

<a​lex:agoradesk.com> Seeing absolutely huge CPU usage. Never seen it this high before, not even close.

<a​lex:agoradesk.com> @selsta are you experiencing the same?

<a​lex:agoradesk.com> We need urgent help debugging monerod's extreme CPU usage and unresponsiveness if anyone's available.

<r​avfx:xmr.mx> My master node crashed in the afternoon

is it a public node?

<r​avfx:xmr.mx> Causing some issue.

<r​avfx:xmr.mx> P2P (master) node is like **idle-2%**

<r​avfx:xmr.mx> RPC (slaves) nodes are like 15-20% of one core, each

<r​avfx:xmr.mx> When master "crashed" a few hours ago, nothing was in the log, and it was still running and even syncing, but the other nodes including my slaves node could not connect to it

<r​avfx:xmr.mx> Was not ressource related, I have no idea about what append, I just restarted it

did the node crash or did it get killed for running out of memory?

<a​lex:agoradesk.com> No.

I have seen a couple OOM, but no crash

<a​lex:agoradesk.com> No it's running at the moment, but using insane CPU since the whole 100kb+ txs started.

<a​lex:agoradesk.com> Super slow and unresponsive despite having enough CPU cores

<r​avfx:xmr.mx> Alex | LocalMonero | AgoraDesk Do you have a lot of RPC request on yours?

not seeing high CPU usage here

<a​lex:agoradesk.com> Yeah there's wallets connected, but it's not a public node

which version are the wallets running and how heavy is the RPC traffic?

<a​lex:agoradesk.com> Latest versions.

<a​lex:agoradesk.com> Is there any way I can PM you from matrix?

not possible

<r​avfx:xmr.mx> From what I see with mine, RPC greatly seam to amplify the CPU usage, like if RPC is only done by one core or something.

<r​avfx:xmr.mx> It's why I separated my node to have one for P2P only and a bunch for RPC.

yes, multiple RPC connections + a large txpool is currently not well optimized

<a​lex:agoradesk.com> Even with one RPC the CPU is insane.

<a​lex:agoradesk.com> One RPC wallet connection, that is.

what does that wallet do? just being connected? or tx creation and more?

<a​lex:agoradesk.com> Connects, syncs, reads txs, sends txs

does it connect and disconnect?

and are the wallets on the same network?

<a​lex:agoradesk.com> I can't discuss these details in the public chat. Is there any way I can PM you?

i won't be available until later, it's super late here

<a​lex:agoradesk.com> Very unfortunate. Maybe someone else can assist? This is really urgent and severe.

<r​avfx:xmr.mx> I could try to help, you can PM me.

<r​avfx:xmr.mx> I could help, if you have "many wallets" that RPC to the node

<r​avfx:xmr.mx> Assuming you have enough hardware, ie, make something like that

<r​avfx:xmr.mx> It does help but you need, fast ram, lot of it. Ideally 8 channels kind of Epyc CPU.

<r​avfx:xmr.mx> Even then the expanding mempool induce a slowdown for the RPC request...

<r​avfx:xmr.mx> And after what I did see today, I do recommand two "master" as mine did stop to accept connections earlier today when I was sleeping...

<r​avfx:xmr.mx> matrix.monero.social/_matrix/media/…oad/xmr.mx/FMKigAiFNsgCPdAYZSPQQFaC

<r​avfx:xmr.mx> Alex | LocalMonero | AgoraDesk

<r​avfx:xmr.mx> All are full node, storage seam to be totally ok with one Nvme drive

<r​avfx:xmr.mx> It's also not possible to use that kind of abusive setup if you have only "one wallet" that do ton and ton of RPC call

<r​avfx:xmr.mx> Because it will all endup routed to one of the slave

<r​avfx:xmr.mx> There is way to route "each packets" to a different node instead of mapping connections by src ip:port

<r​avfx:xmr.mx> It might work for that, in that case if could work with "one wallet". But I did not test that setup as I normally map connection by src ip:port because that setup work for TLS unlike load balancing each packets. But who know, it might work for monero, would need to ask someone else

<r​avfx:xmr.mx> I don't know if a DEV (or someone who know) can answer me.

<r​avfx:xmr.mx> Question : If say 4 from the same IP endup connecting on the same node, does that will trigger that node to ban my IP?

<r​avfx:xmr.mx> From what I see today, I think I can ditch the master and let the other node connect directly to the others internet node instead of just being allowed to connect to the master node, then I can just also load balance incoming 18080.

<r​avfx:xmr.mx> I don't know if a DEV (or someone who know) can answer me.

<r​avfx:xmr.mx> Question : If say 4 nodes from the same IP endup connecting on the same random internet node, does that will trigger that node to ban my IP?

<r​avfx:xmr.mx> From what I see today, I think I can ditch the master and let the other node connect directly to the others internet node instead of just being allowed to connect to the master node, then I can just also load balance incoming 18080.

<a​lex:agoradesk.com> Seems to have resolved itself for now... Not sure what changed.

<3​21bob321:monero.social> Ctrl alt defeat

<r​avfx:xmr.mx> It's me or the asshole broadcast 3 x ~148/2 every time a block get mined

ravfx No, the newest 100 kb tx (146/2 - 150/2) in mempool is currently 3 hours old

<r​avfx:xmr.mx> ok, good to know, so xmrchain.net don't show all the mempool

<r​avfx:xmr.mx> That explain the discrepancy

<r​avfx:xmr.mx> matrix.monero.social/_matrix/media/…oad/xmr.mx/LVQHEbmNuIxRATCJBBvNxQyR

<r​avfx:xmr.mx> matrix.monero.social/_matrix/media/…oad/xmr.mx/hMywoaDeQwHgvmqGUIwxCjqP

yes, it probably running with 3 hour txpool timeout

p2pool.io/explorer shows everything as far as I'm aware

<r​avfx:xmr.mx> Will use that one instead, thanks

<r​4v3r23:monero.social> v0.18.4 then? the PR is mature and such a massive UX improvement its a shame to leave it sitting unmerged

+1 to the high cpu usage - saw highest CPU peaks on monerod with 4 "p2pool" instances connected to it

starting about 20:00 UTC to about a few hours ago (still high but within normal levels high)

notably it caused an even taller peak of read I/O

400x over baseline

(and some increased network traffic, which is probably just the transactions going around)

<r​4v3r23:monero.social> looks like automatic fee is still kinda broken in latest wave of txs

<r​4v3r23:monero.social> are there further improvements that can be made?

Automatic fee worked as expected, but it never increases the fee above normal level

Latest wave used normal level too

<r​4v3r23:monero.social> the wave where auto was broken?

<r​ucknium:monero.social> When selsta fixed the auto fee adjustment bug to get it back to the intended behavior, I thought to myself "Is moving from 1st to 2nd tier enough? What about to 3rd or even 4th tier?" But the fix was included so close to a new release date. I didn't want to have us re-think the concepts and have to do more testing, etc.

<r​ucknium:monero.social> Fee prediction is on my CCS research agenda. That doesn't necessarily mean that I will be able to fully research it during this CCS round. Depends on prioritization with other things.

<r​4v3r23:monero.social> yes by "kinda broken" i meant its not doing enough to function as user expects during latest mempool surge

<r​4v3r23:monero.social> please do. having fees that "just work" is such an understated part of using monero

<r​4v3r23:monero.social> good to hear. having fees that "just work" is such an understated part of using monero

<r​ucknium:monero.social> Monero's rules increases the stakes for fee prediction. Wallets have to get it right the first time. Monero has no replace-by-fee nor child-pays-for parent. Once a tx is in the txpool, users cannot increase the fee if they made a "mistake" with their fee bid.

<r​ucknium:monero.social> I make these points in Section 7 "Transaction confirmation delay" of my little paper: github.com/Rucknium/misc-research/b…d/pdf/monero-black-marble-flood.pdf

<r​ucknium:monero.social> Fees "just work" when there is no congestion :). The Monero ecosystem's fee selection is primitive.

<r​ucknium:monero.social> But these considerations make the question _interesting_: 1) Want tx uniformity for privacy, 2) No way to change tx fee bid, 3) Dynamic relay fee/block size/miner penalty rules, 4) To expand the block size, there _must_ be a period of txpool/blockchain congestion.

<r​avfx:xmr.mx> Auto fee just work when monero is not under consolidation attack right...

<r​avfx:xmr.mx> We we bump fee higher in auto selection algo, the attacker will just use higher tier fee for his consolidation attack and we are back to the same problem.

<r​avfx:xmr.mx> He seam to always use low fee for his 1/2 attack.

<r​avfx:xmr.mx> Main issue is people not updating there wallet, helped another one yesterday, he send a TX with low fee with monero gui 0.18.3.1. Monero gui show a nag in the middle of the window that ask you to update.... It's been over a month.

ravfx: Normally the auto-updater pops up upon startup

<r​avfx:xmr.mx> could be that, could be a good idea to check for update every day or so without having to restart the wallet (for the people that leave it on)

I sent a min fee tx during the large mempool and the CLI stated that it would take 7 blocks at that fee level and was that ok, meanwhile it took 211 blocks

no problem as I was sleeping and I figured it would take that long, just reporting incase anyone thinks it needs refinement

<a​lex:agoradesk.com> Yup. The CPU usage was way out of proportion. It's as if there's some kind of bottleneck.

<r​avfx:xmr.mx> I would leave the auto as is. Block grow as needed (we saw that yesterday).

<r​avfx:xmr.mx> Allow people to set tier 3 as needed if said user want to send to a payment processor or exchange now

<r​avfx:xmr.mx> having auto to auto scale to 3 or 4 will endup making bigger block faster and will give more room for attacker to flood black marbles and bloat the chain.

same here nioCat, the wallet's estimated backlog was way off. told me no backlog at priority 2, while hundreds of those maxed-out, 100kb consolidations cut in front of me, paying an extra 2-4 nanoneros per byte

for me it wasn't an issue of "they" using a custom fee as I was using min fee, or was it?

related comment: in simplewallet.cpp, only transfer_main and print_fee_info call estimate_backlog. sweep_main and sweep_single don't seem to check for backlog at all. i think, even if users are just churning to themselves and don't mind waiting a couple hours, they should be made aware of the potential delay

Hello, is this the wrong place to ask for advice?

sorry, just read this Non-dev questions -> #monero