<r​4v3r23:monero.social> guys transacting over the past few days using the "auto" fee setting shows that its clearly broken and not adjusting to the heavy mempool load

there is a PR open for it

<r​4v3r23:monero.social> just saw

<r​4v3r23:monero.social> testing the fix now

.merges

selsta I commented in #9219 and #9221

I don't want to open another PR to fix #9221, since you already have one and can just add the second fix there

<c​onordev:matrix.org> hey folks, kraken dev here, having very slow node this morning and understand it may be related to mempool or peers issue

<c​onordev:matrix.org> node seems to keep falling out of sync

<c​onordev:matrix.org> we have a ton of peers, would limiting help

Are you running v0.18.3.1?

sudo perf top - check what the functions are the top are. If they're crypto stuff, it's probably just verifying txes.

My node seems to be fine, although sending and receiving a lot of data:

Received 64215859394 bytes (59.81 GB) in 36309827 packets in 10.2 hours, average 1.68 MB/s = 1.68% of the limit of 100.00 MB/s

Sent 285882647255 bytes (266.25 GB) in 9980436 packets in 10.2 hours, average 7.46 MB/s = 7.46% of the limit of 100.00 MB/s

If it's std::map etc functions, it's probably txpool needing speeding up.

what does "status" command show in monerod? How many in/out peers do you have?

more than 100 is not needed, it will only slow down the node

add --in-peers 64 --out-peers 32 to the command line to limit it

limiting peers definitely helps - the more peers you have connected, the more bandwidth you need. You can see I get up to almost 10 MB/s (100 Mbit) with only a 100 peers

<c​onordev:matrix.org> we are on v0.18.1.2

<c​onordev:matrix.org> will start limiting peers

if limiting peers doesn't help, it's better to upgrade to v0.18.3.1. There were a number of important fixes since v0.18.1.2

for example v0.18.2.0 added monero-project/monero #8676 which can really speed up the node

<c​onordev:matrix.org> ok, will start on that

actually, #8676 gives bigger speed up when mempool is bigger which is exactly what's happening now. Mempool is huge due to a spam attack.

so nodes before v0.18.2.0 will be affected the most

<c​onordev:matrix.org> ok

<c​onordev:matrix.org> thanks for info, getting started on upgrade now

even with upgraded node it's still better to also limit the peer count

<y​pavtv97lx:matrix.org> network congestion seems getting higher (at the moment ~7k tx in pool), shouldn't block size increase faster ?

It increases slowly (+1.5 kB every 50 blocks) because they use minimal fees

<y​pavtv97lx:matrix.org> minimal fees as in less than default ? last time I have sent XMR with default fee from my node i had to wait 11 blocks

<y​pavtv97lx:matrix.org> *last time = two days ago when the attack was ongoing

<c​onordev:matrix.org> is limiting peers sticky between restarts or something you have to set every time, if you are using the daemon rpc /out_peers

<e​ndor00:matrix.org> If you use the config file, it will get picked up every time

either add it to monerod command line "--in-peers 64 --out-peers 32" - then it will stick if it's in whatever script that you use to start monerod

or add it to bitmonero.conf if you use it

out-peers=32

in-peers=64

default values, IIRC, are unlimited in-peers and 12 out-peers

which is not very good if you get 1000 incoming connections and then run out of file descriptors (ulimit)

bitmonero.conf should be in /home/USER_NAME/.bitmonero/bitmonero.conf

you can always check "status" in monerod to see if your setting works

<c​onordev:matrix.org> yeah got it, we use our own conf, will put setting there

daemon rpc /out_peers doesn't stick between restarts

ypavtv97lx default fee is the minimal fee due to a bug, so you need to set higher fee manually reddit.com/r/Monero/comments/1b946w…ncrease_your_transaction_fee_if_you

<c​onordev:matrix.org> I assume monero rpc transfer getmonero.org/resources/developer-guides/wallet-rpc.html#transfer `priority` setting is still working properly?

yes

priority 2 or higher is recommended now

<c​onordev:matrix.org> we've been doing priority 2 since the first mempool bump on tuesday

<c​onordev:matrix.org> so we at least dont have a tx backlog

That's nice, I always knew Kraken is on top of things

<c​onordev:matrix.org> Just for reference, were there any wallet db migrations between v0.18.1.2 and v0.18.3.1 - just need a game plane in the unlikely case we hit wallet issues in the upgrade

<c​onordev:matrix.org> Just for reference, were there any wallet db migrations between v0.18.1.2 and v0.18.3.1? - just need a game plane in the unlikely case we hit wallet issues in the upgrade

monerod doesn't work with wallets, you only need to update monerod to v0.18.3.1

but no, I don't remember any issues with my wallets when I was upgrading

You can always stay on the safe side and update monero-wallet-rpc later

<c​onordev:matrix.org> we usually upgrade everything together ive never actually run monerod and monero-wallet-rpc on different versions, didnt know if that could ever cause serialization incompatibility, but good to know

minor and patch releases are interchangeable

so theorethically even v0.18.0.0 and v0.18.3.1 can work together in any combinations

I don't know your procedures, but I guess you do a full wallet backup before starting a new version

so you can just roll back everything and re-scan using the old version if issues happen

if anything in RPC changes, monerod will report a newer RPC protocol version to monero-wallet-rpc. If the wallet doesn't like it, it will show some error

hmm, IIRC I use v0.18.1.0 CLI on one of my machines, and it connects to my v0.18.3.1 node just fine

but again, it's better to upgrade the rpc wallet version too to get the latest fixes

Don't forget to check SHA256 and pgp signatures (obviously). Better safe than sorry :D

<c​onordev:matrix.org> definitely! - always do

<a​lex:agoradesk.com> conor [Kraken]: PM'd you with some more tips.

c​onordev: you have to upgrade both daemon and wallet to v0.18.3.1 to take advantage of the txpool performance improvement

<a​lex:agoradesk.com> There's time required for testing an updated version of the wallet-rpc. If their situation is urgent they can try just upgrading monerod first, as that will also bring improvements.

They had issues with monerod being slow, which is why I suggested upgrading only monerod

monero-project/monero #8800 specifically requires both monerod and monero-wallet-rpc to be upgraded

it doesn't break backwards compatibility

<c​onordev:matrix.org> ive upgraded just monerod and daemon performance is very much improved - we will schedule a wallet-rpc for a bit later after we get to do a bit more testing

yeah

it's Friday after all

never deploy on Fridays :D

did you also limit the incoming/outgoing peers?

<c​onordev:matrix.org> thats actually a goal of ours 😂

<c​onordev:matrix.org> yes

It should be enough for now

conordev: how many wallets do you have connected to the node?

<c​onordev:matrix.org> just one

<c​onordev:matrix.org> we are all prepped to also deploy wallet rpc upgrade if we see insufficient performance, just dont want to avoid it today if i can and wait until next week

i think the node falling behind comes from really rpc traffic, maybe you can add a delay between operations if everything else doesn't work

In my experience, node can fall behind with too many incoming connections - either all network bandwidth will be used, or it will hit ulimit (1024 by default)

and of course a lot of transactions flying around. The cache in 0.18.2.0 really helps

at least on my nodes it only happened on ones with public rpc enabled

I never enable public rpc, it's a performance killer

everyone will just connect and abuse it

when M5M400 had his node on Hetzner, Hetzner was pissed

it was often hitting 1 Gbit

Public RPC nodes should run on dedicated servers and should be used just for that - public RPC

<a​lex:agoradesk.com> Awesome! Congrats.

hooray for this foresight: monero-project/monero #1982

I don't think managing the pool in daemon memory is the problem. I think transferring all these txs between daemons, and then from daemon to wallets for "public" nodes, cause problems

mempool size is 15 MB, according to block explorers. Almost nothing to deal with purely in RAM, a pain to transfer around if a wallet e.g. asks for the pool every minute

(and does not yet run my incremental update algorithm, that is)

also, every wallets don't cache anything at all

like, if I have two wallets that are a month behind, and I sync one, then sync the other, I end up downloading all that shit twice for no good reason

I get it's a space thing but I feel like there could be a reasonable option for caching and a lot of public node users would take advantage. decent in-between running your own and pub node to cache at least the most recent blocks

Lyza1, that's why you run a Node then sync all wallets to it

I run my own node aog but even then it re-downloads the stuff voer my LAN every time, or what if I ran my node on a VPS

Well, there is a very good reason. The code that does this, the `wallet2` class, is so spaghetti, it would be quite a pain to make it more clever dealing with multiple wallets

Lyza1, sweep the wallets higher up the tx chain, and sync only from higher block onward

yeeaaah I feel like it might make more sense to implement wallet side, I was talking to tobtoht a little about it the other day

aog again I'm talking about ways to reduce load on public nodes

takes like 5 minutes to sync a month using own node

yeah I get that

Especially if your own node is on your LAN

Lyza1, the way to reduce load on pub nodes is for ppl to run own private node :)

<p​cre:matrix.org> Do you guys remember the day when the Bitcoin daemon was part of the Bitcoin client?

<p​cre:matrix.org> Always keep the Unix philosophy in mind. Do one thing do it good.

<p​cre:matrix.org> char Strings is the universal interface

<p​cre:matrix.org> If we stick to that we will be fine.

bitcoin daemon is still part of the client :|

i.e. Bitcoin core is still node+wallet in one software prolly /offtopic tho

<p​cre:matrix.org> it is? Tough luck.

<p​cre:matrix.org> I thought they removed that 8 years ago

<p​cre:matrix.org> Then we are always one step ahead

<g​fdshygti53:monero.social> Most wallets based on bitcoin core QT wallet have the node integrated yeah

<p​cre:matrix.org> I don't know whether to react with surprise or disgust - to hear that.

.merges

Merge queue empty

.merge+ 9219 9220

...

should we include monero-project/monero #9218 ? UkoeHB had some concerns

I found a small issue with p2pool mining in these mempool conditions, I'm preparing a PR now

#9222 and #9223

As for #9218, I addressed UkoeHB's concerns

jeffro256: can you open 9218 against release?

<j​effro256:monero.social> yes

.merge+ 9218 9224

Added

<v​bonini:matrix.org> Hi Team. We run XMR nodes in EKS/K8s, and we've noticed an issue recently (with blocks >50% full) where our XMR nodes wont use more than 1 vCPU to both process blocks and field RPC requests. We've allocated 4+ but it doesn't seem to respect --max-concurrency. Do you have any tips on running XMR nodes in K8s pods successfully?

Are you using the latest release (v0.18.3.1)?

I've just checked htop and my node uses 3-4 cores all the time (not to 100% though)

max concurrency is "how many transient (pooled if someone pedantic is around) threads to use in parallel for heavy stuff like BP checks". It does not mean "how many cores to use".

you need to check htop and sort monerod threads by CPU usage, then you'll see if it runs things in parallel or not

<v​bonini:matrix.org> Understood. We are on v0.18.2.2, but this has always been the behavior. It only just recently became an issue over the last few days.

<v​bonini:matrix.org> Our starting args: --log-level=0 --out-peers=64 --disable-rpc-ban --max-concurrency 4 --ban-list=/home/monero/block.txt --limit-rate-up=1048576 --p2p-bind-port=18080 --block-sync-size=10 --rpc-restricted-bind-ip=0.0.0.0 --no-igd --p2p-external-port=18088 --zmq-rpc-bind-ip=0.0.0.0 --db-sync-mode=safe --rpc-restricted-bind-port=18081 --fast-block-sync=1 --prep-blocks-threads 2 --p2p<clipped message>

<v​bonini:matrix.org> -bind-ip=0.0.0.0 --rpc-bind-port=18089 --confirm-external-bind --zmq-rpc-bind-port=5555 --limit-rate-down=1048576 --rpc-bind-ip=0.0.0.0 --enable-dns-blocklist --in-peers=1024 --hide-my-port --non-interactive

why so many in-peers, you don't need that many

<v​bonini:matrix.org> Monero is just one thread

<v​bonini:matrix.org> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

<v​bonini:matrix.org> 166 monero 20 0 203.0g 3.6g 2.1g S 102.0 5.8 404:31.90 monerod

Those are processes. You can switch top to display all threads.

(-H)

<v​bonini:matrix.org> Ok. This was set up before my time, so some of the reasoning behind these aren't known. I'll review the peers arg.

<v​bonini:matrix.org> Ok thank you top -H does show multiple threads.

<v​bonini:matrix.org> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

<v​bonini:matrix.org> 181 monero 20 0 203.0g 3.6g 2.1g R 70.0 5.8 143:18.67 monerod

<v​bonini:matrix.org> 180 monero 20 0 203.0g 3.6g 2.1g S 30.7 5.8 142:58.65 monerod

<v​bonini:matrix.org> 193 monero 20 0 203.0g 3.6g 2.1g S 0.3 5.8 11:02.49 monerod

<v​bonini:matrix.org> 1 monero 20 0 3572 2564 2216 S 0.0 0.0 0:00.02 entrypoint.sh

<v​bonini:matrix.org> 166 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 0:51.90 monerod

<v​bonini:matrix.org> 170 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 0:00.00 monerod

<v​bonini:matrix.org> 172 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.19 monerod

<v​bonini:matrix.org> 173 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.56 monerod

<v​bonini:matrix.org> 174 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.51 monerod

Paste multiline stuff to paste.debian.net please.

even in this paste you have 2 active threads in monerod

<v​bonini:matrix.org> Right. So I'm not sure why it's not using all of this: paste.debian.net/1309912

<a​js_:matrix.org> Once wallets update and correct for a fee market, would the low txs stuck in mempool eventually drop off after 3 days? github.com/monero-project/monero/bl…master/src/cryptonote_config.h#L102

<a​js_:matrix.org> If yes, would it make sense to reduce the drop off time to a day or maybe a few hours to clear out the mempool faster?

No

Afaict, Under proper fee pressure the tx should get confirmed

<l​ilith_adam:matrix.org> hi guys, I'm newbie here, is someone facing network problem now?

the network is running, you might have problems if you use public nodes that are overloaded

<l​ilith_adam:matrix.org> thank you, i'm using my own one

what issue are you having?

.merge+ 9222 9223

Added

<l​ilith_adam:matrix.org> thank you, we'll chack

<l​ilith_adam:matrix.org> thank you, we'll check

luigi1111: are you available to merge the remaining PR and then tag?

.merges

"the remaining PR"

:D

there's been a bit of activity today :D

haha. Usually don't like merging things sub 24hrs, but these are probably simple enough

hyc can you comment on the questions on 9222?

I'm running 9222 on my node now and it works as expected

I'm not concerned just want "double confirm" since it's so short

will take a look

yeah it's all good

nice

luigi1111w I guess you can merge 9222/9223 now

thanks

i found one more issue that might have to be addressed

which is?

Hi is there a changelog for v0.18.3.2

there will be once it's released

<s​omeoneelse495495:matrix.org> if it isn't too late is there a way for merging 9211 before release ?

<s​omeoneelse495495:matrix.org> low priority but have been ready for a week now

Not too late

rbrunner moneromoooo hyc or who is here now, please review monero-project/monero #9225

s​omeoneelse495495: can you squash your commits?

<s​omeoneelse495495:matrix.org> I honestly don't know how to do that, but I guess I can learn asap

If you use Github desktop, you can select commits (holdint Ctrl and clicking on them), then right-click and select "squash"

<s​omeoneelse495495:matrix.org> github desktop is for racists ima use cli like a good dev

git-tower.com/learn/git/faq/git-squash then, but github desktop is many times faster for these kinds of things

<s​omeoneelse495495:matrix.org> done

<j​effro256:monero.social> sech: just now saw your PR lol. We found the txcompare issue concurrently: monero-project/monero #9227

<j​effro256:monero.social> good find

lol, nice

approved

<j​effro256:monero.social> Someone should make a reddit post telling miner operators to restart their daemon every few hours until they update to the new release

can you also approve #9226

<j​effro256:monero.social> That will reset `m_txs_by_fee_and_receive_time` and get those txs unstuck in the short term

not necessary

Done. Gratuitous changes in the compare func but meh.

transactions arrive to different nodes at different times

so someone will mine them eventually

<j​effro256:monero.social> Yeah eventually, but it's practically affecting tx wait times rn, probably since individual pools have large shares in current mining landscape

selsta when will it be released?

yes, restarting nodes should help

Guest2: if we don't find another bug then we will tag today, and hopefully release tomorrow or sunday

wow, this one has been stuck for 20 hours xmrchain.net/tx/bcf3f15f1a877a0be48…92d5d3a3200db452b7ec615b59c08e07dd4

<j​effro256:monero.social> Yeah there's 22 txs that have been stuck for 3+ hours

p2pool.io/explorer is running the curent release-v0.18 branch + #9226 on top of it

xmrchain.net/txpool shows bigger times (scroll down)

probably because it wasn't restarted recently

this one has been stuck for 16 hours despite having higher than minimal fee xmrchain.net/tx/32cbd8640fddcf7e9d4…4521f065d90f27a7c72984ef2c6d690d5ff

and a non-standard fee at that

<j​effro256:monero.social> thats what you get for putting a non-standard fee bozo

all flood transactions use lower fee

<j​effro256:monero.social> was just joking

<a​lex:agoradesk.com> @sech1 @selsta jeffro256 the cli wallet's multisig tx building is creating txs with fees lower than low on single sig despite the priority being set as normal. A bug in the tx builder not taking into consideration the higher multisig tx size perhaps?

maybe

so they don't get propagated because of too low fee/byte?

<a​lex:agoradesk.com> Yup.

weird, how did this never get noticed before?

or did they get included when the txpool was empty?

<a​lex:agoradesk.com> I'm guessing this.

no, nodes discard everything less than 20000 piconero per byte

I think they allow 5% leeway but not more

<a​lex:agoradesk.com> My mistake, not lower than low. Almost as low as low. 20000 for single sig low and the multisig normal is being build with 23000

<a​lex:agoradesk.com> being built*

hmm, that transaction is exactly 23000

<a​lex:agoradesk.com> Not exactly 23000, I rounded.

what fee does it use when you create it with low?

no, it's 22000 there

<a​lex:agoradesk.com> Will test

<j​effro256:monero.social> Wdym "the higher multisig tx size"? Multisig txs should look exactly the same on-chain as normal AFAIK

he means the fee calculation doesn't account for multisig

<a​lex:agoradesk.com> Right.

Hello everyone. I am a highly motivated noob coder. I would love to contribute to this project to the best of my abilities. How can I find out what needs to be done to help this project succeed?

<a​lex:agoradesk.com> @selsta @sech1 jeffro256: apologies, seems to only be a thing in testnet multisig txs.

<a​lex:agoradesk.com> Mainnet all good

<h​ardenedsteel:monero.social> getmonero.org/get-started/contributing

Guest92: DMed you.

.merges

.merge+ 9225 9226

Added

s​omeoneelse495495: will include your patch next time, it's not 2 weeks old yet and not urgent enough to include early

.merges

luigi1111w: we have everything for the tag

after these 3 are merged