01:20:10 guys transacting over the past few days using the "auto" fee setting shows that its clearly broken and not adjusting to the heavy mempool load 01:20:53 there is a PR open for it 01:21:46 just saw 01:31:32 testing the fix now 02:19:58 .merges 02:19:58 -xmr-pr- 9166 9167 9168 9169 9170 9179 9184 9187 9195 9214 9217 07:32:11 selsta I commented in #9219 and #9221 07:33:07 I don't want to open another PR to fix #9221, since you already have one and can just add the second fix there 08:03:49 hey folks, kraken dev here, having very slow node this morning and understand it may be related to mempool or peers issue 08:04:18 node seems to keep falling out of sync 08:04:36 we have a ton of peers, would limiting help 08:08:05 Are you running v0.18.3.1? 08:08:14 sudo perf top - check what the functions are the top are. If they're crypto stuff, it's probably just verifying txes. 08:08:36 My node seems to be fine, although sending and receiving a lot of data: 08:08:37 Received 64215859394 bytes (59.81 GB) in 36309827 packets in 10.2 hours, average 1.68 MB/s = 1.68% of the limit of 100.00 MB/s 08:08:37 Sent 285882647255 bytes (266.25 GB) in 9980436 packets in 10.2 hours, average 7.46 MB/s = 7.46% of the limit of 100.00 MB/s 08:08:46 If it's std::map etc functions, it's probably txpool needing speeding up. 08:09:35 what does "status" command show in monerod? How many in/out peers do you have? 08:09:47 more than 100 is not needed, it will only slow down the node 08:10:16 add --in-peers 64 --out-peers 32 to the command line to limit it 08:11:24 limiting peers definitely helps - the more peers you have connected, the more bandwidth you need. You can see I get up to almost 10 MB/s (100 Mbit) with only a 100 peers 08:13:46 we are on v0.18.1.2 08:13:50 will start limiting peers 08:14:52 if limiting peers doesn't help, it's better to upgrade to v0.18.3.1. There were a number of important fixes since v0.18.1.2 08:16:53 for example v0.18.2.0 added https://github.com/monero-project/monero/pull/8676 which can really speed up the node 08:17:22 ok, will start on that 08:18:58 actually, #8676 gives bigger speed up when mempool is bigger which is exactly what's happening now. Mempool is huge due to a spam attack. 08:19:32 so nodes before v0.18.2.0 will be affected the most 08:22:16 ok 08:22:18 thanks for info, getting started on upgrade now 08:24:52 even with upgraded node it's still better to also limit the peer count 08:25:01 network congestion seems getting higher (at the moment ~7k tx in pool), shouldn't block size increase faster ? 08:25:26 It increases slowly (+1.5 kB every 50 blocks) because they use minimal fees 08:29:55 minimal fees as in less than default ? last time I have sent XMR with default fee from my node i had to wait 11 blocks 08:31:14 *last time = two days ago when the attack was ongoing 08:33:58 is limiting peers sticky between restarts or something you have to set every time, if you are using the daemon rpc /out_peers 08:34:56 If you use the config file, it will get picked up every time 08:36:23 either add it to monerod command line "--in-peers 64 --out-peers 32" - then it will stick if it's in whatever script that you use to start monerod 08:36:37 or add it to bitmonero.conf if you use it 08:36:48 out-peers=32 08:36:48 in-peers=64 08:37:21 default values, IIRC, are unlimited in-peers and 12 out-peers 08:37:54 which is not very good if you get 1000 incoming connections and then run out of file descriptors (ulimit) 08:39:04 bitmonero.conf should be in /home/USER_NAME/.bitmonero/bitmonero.conf 08:39:27 you can always check "status" in monerod to see if your setting works 08:39:33 yeah got it, we use our own conf, will put setting there 08:40:37 daemon rpc /out_peers doesn't stick between restarts 08:45:28 ypavtv97lx default fee is the minimal fee due to a bug, so you need to set higher fee manually https://www.reddit.com/r/Monero/comments/1b946w3/psa_manually_increase_your_transaction_fee_if_you/ 08:57:47 I assume monero rpc transfer https://www.getmonero.org/resources/developer-guides/wallet-rpc.html#transfer `priority` setting is still working properly? 08:57:58 yes 08:58:20 priority 2 or higher is recommended now 09:15:40 we've been doing priority 2 since the first mempool bump on tuesday 09:16:11 so we at least dont have a tx backlog 09:17:24 That's nice, I always knew Kraken is on top of things 10:15:06 Just for reference, were there any wallet db migrations between v0.18.1.2 and v0.18.3.1 - just need a game plane in the unlikely case we hit wallet issues in the upgrade 10:15:28 Just for reference, were there any wallet db migrations between v0.18.1.2 and v0.18.3.1? - just need a game plane in the unlikely case we hit wallet issues in the upgrade 10:19:19 monerod doesn't work with wallets, you only need to update monerod to v0.18.3.1 10:19:35 but no, I don't remember any issues with my wallets when I was upgrading 10:20:12 You can always stay on the safe side and update monero-wallet-rpc later 10:23:22 we usually upgrade everything together ive never actually run monerod and monero-wallet-rpc on different versions, didnt know if that could ever cause serialization incompatibility, but good to know 10:24:00 minor and patch releases are interchangeable 10:24:20 so theorethically even v0.18.0.0 and v0.18.3.1 can work together in any combinations 10:24:52 I don't know your procedures, but I guess you do a full wallet backup before starting a new version 10:25:09 so you can just roll back everything and re-scan using the old version if issues happen 10:26:14 if anything in RPC changes, monerod will report a newer RPC protocol version to monero-wallet-rpc. If the wallet doesn't like it, it will show some error 10:27:03 hmm, IIRC I use v0.18.1.0 CLI on one of my machines, and it connects to my v0.18.3.1 node just fine 10:29:05 but again, it's better to upgrade the rpc wallet version too to get the latest fixes 10:33:55 Don't forget to check SHA256 and pgp signatures (obviously). Better safe than sorry :D 10:43:30 definitely! - always do 10:50:16 conor [Kraken]: PM'd you with some more tips. 11:23:21 c​onordev: you have to upgrade both daemon and wallet to v0.18.3.1 to take advantage of the txpool performance improvement 11:27:37 There's time required for testing an updated version of the wallet-rpc. If their situation is urgent they can try just upgrading monerod first, as that will also bring improvements. 11:29:22 They had issues with monerod being slow, which is why I suggested upgrading only monerod 11:30:01 https://github.com/monero-project/monero/pull/8800 specifically requires both monerod and monero-wallet-rpc to be upgraded 11:31:17 it doesn't break backwards compatibility 11:40:04 ive upgraded just monerod and daemon performance is very much improved - we will schedule a wallet-rpc for a bit later after we get to do a bit more testing 11:40:19 yeah 11:40:22 it's Friday after all 11:40:27 never deploy on Fridays :D 11:40:56 did you also limit the incoming/outgoing peers? 11:41:01 thats actually a goal of ours 😂 11:41:05 yes 11:41:23 It should be enough for now 11:41:23 conordev: how many wallets do you have connected to the node? 11:41:40 just one 11:42:26 we are all prepped to also deploy wallet rpc upgrade if we see insufficient performance, just dont want to avoid it today if i can and wait until next week 11:43:30 i think the node falling behind comes from really rpc traffic, maybe you can add a delay between operations if everything else doesn't work 11:44:36 In my experience, node can fall behind with too many incoming connections - either all network bandwidth will be used, or it will hit ulimit (1024 by default) 11:44:58 and of course a lot of transactions flying around. The cache in 0.18.2.0 really helps 11:45:07 at least on my nodes it only happened on ones with public rpc enabled 11:45:28 I never enable public rpc, it's a performance killer 11:45:34 everyone will just connect and abuse it 11:46:01 when M5M400 had his node on Hetzner, Hetzner was pissed 11:46:13 it was often hitting 1 Gbit 11:48:40 Public RPC nodes should run on dedicated servers and should be used just for that - public RPC 11:50:32 Awesome! Congrats. 12:23:23 hooray for this foresight: https://github.com/monero-project/monero/pull/1982 12:59:04 I don't think managing the pool in daemon memory is the problem. I think transferring all these txs between daemons, and then from daemon to wallets for "public" nodes, cause problems 13:00:44 mempool size is 15 MB, according to block explorers. Almost nothing to deal with purely in RAM, a pain to transfer around if a wallet e.g. asks for the pool every minute 13:01:25 (and does not yet run my incremental update algorithm, that is) 13:05:19 also, every wallets don't cache anything at all 13:05:39 like, if I have two wallets that are a month behind, and I sync one, then sync the other, I end up downloading all that shit twice for no good reason 13:06:36 I get it's a space thing but I feel like there could be a reasonable option for caching and a lot of public node users would take advantage. decent in-between running your own and pub node to cache at least the most recent blocks 13:06:43 Lyza1, that's why you run a Node then sync all wallets to it 13:07:08 I run my own node aog but even then it re-downloads the stuff voer my LAN every time, or what if I ran my node on a VPS 13:07:13 Well, there is a very good reason. The code that does this, the `wallet2` class, is so spaghetti, it would be quite a pain to make it more clever dealing with multiple wallets 13:08:17 Lyza1, sweep the wallets higher up the tx chain, and sync only from higher block onward 13:08:26 yeeaaah I feel like it might make more sense to implement wallet side, I was talking to tobtoht a little about it the other day 13:08:40 aog again I'm talking about ways to reduce load on public nodes 13:08:55 takes like 5 minutes to sync a month using own node 13:09:06 yeah I get that 13:09:21 Especially if your own node is on your LAN 13:09:43 Lyza1, the way to reduce load on pub nodes is for ppl to run own private node :) 13:30:53 Do you guys remember the day when the Bitcoin daemon was part of the Bitcoin client? 13:31:42 Always keep the Unix philosophy in mind. Do one thing do it good. 13:32:07 char Strings is the universal interface 13:32:52 If we stick to that we will be fine. 13:34:06 bitcoin daemon is still part of the client :| 13:34:37 i.e. Bitcoin core is still node+wallet in one software prolly /offtopic tho 13:35:11 it is? Tough luck. 13:35:24 I thought they removed that 8 years ago 13:35:46 Then we are always one step ahead 13:36:43 Most wallets based on bitcoin core QT wallet have the node integrated yeah 13:40:45 I don't know whether to react with surprise or disgust - to hear that. 13:44:24 .merges 13:44:24 Merge queue empty 13:44:45 .merge+ 9219 9220 13:44:45 ... 13:45:10 should we include https://github.com/monero-project/monero/pull/9218 ? UkoeHB had some concerns 13:46:23 I found a small issue with p2pool mining in these mempool conditions, I'm preparing a PR now 13:56:01 #9222 and #9223 14:00:19 As for #9218, I addressed UkoeHB's concerns 14:11:17 jeffro256: can you open 9218 against release? 14:11:29 yes 14:17:04 .merge+ 9218 9224 14:17:04 Added 14:51:41 Hi Team. We run XMR nodes in EKS/K8s, and we've noticed an issue recently (with blocks >50% full) where our XMR nodes wont use more than 1 vCPU to both process blocks and field RPC requests. We've allocated 4+ but it doesn't seem to respect --max-concurrency. Do you have any tips on running XMR nodes in K8s pods successfully? 14:53:34 Are you using the latest release (v0.18.3.1)? 14:54:49 I've just checked htop and my node uses 3-4 cores all the time (not to 100% though) 14:58:05 max concurrency is "how many transient (pooled if someone pedantic is around) threads to use in parallel for heavy stuff like BP checks". It does not mean "how many cores to use". 15:00:08 you need to check htop and sort monerod threads by CPU usage, then you'll see if it runs things in parallel or not 15:01:07 Understood. We are on v0.18.2.2, but this has always been the behavior. It only just recently became an issue over the last few days. 15:01:07 Our starting args: --log-level=0 --out-peers=64 --disable-rpc-ban --max-concurrency 4 --ban-list=/home/monero/block.txt --limit-rate-up=1048576 --p2p-bind-port=18080 --block-sync-size=10 --rpc-restricted-bind-ip=0.0.0.0 --no-igd --p2p-external-port=18088 --zmq-rpc-bind-ip=0.0.0.0 --db-sync-mode=safe --rpc-restricted-bind-port=18081 --fast-block-sync=1 --prep-blocks-threads 2 --p2p 15:01:08 -bind-ip=0.0.0.0 --rpc-bind-port=18089 --confirm-external-bind --zmq-rpc-bind-port=5555 --limit-rate-down=1048576 --rpc-bind-ip=0.0.0.0 --enable-dns-blocklist --in-peers=1024 --hide-my-port --non-interactive 15:01:56 why so many in-peers, you don't need that many 15:02:32 Monero is just one thread 15:02:32 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15:02:33 166 monero 20 0 203.0g 3.6g 2.1g S 102.0 5.8 404:31.90 monerod 15:02:58 Those are processes. You can switch top to display all threads. 15:03:08 (-H) 15:03:27 Ok. This was set up before my time, so some of the reasoning behind these aren't known. I'll review the peers arg. 15:04:29 Ok thank you top -H does show multiple threads. 15:04:29 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15:04:29 181 monero 20 0 203.0g 3.6g 2.1g R 70.0 5.8 143:18.67 monerod 15:04:30 180 monero 20 0 203.0g 3.6g 2.1g S 30.7 5.8 142:58.65 monerod 15:04:30 193 monero 20 0 203.0g 3.6g 2.1g S 0.3 5.8 11:02.49 monerod 15:04:31 1 monero 20 0 3572 2564 2216 S 0.0 0.0 0:00.02 entrypoint.sh 15:04:31 166 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 0:51.90 monerod 15:04:32 170 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 0:00.00 monerod 15:04:32 172 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.19 monerod 15:04:33 173 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.56 monerod 15:04:33 174 monero 20 0 203.0g 3.6g 2.1g S 0.0 5.8 1:05.51 monerod 15:04:48 Paste multiline stuff to paste.debian.net please. 15:05:48 even in this paste you have 2 active threads in monerod 15:06:16 Right. So I'm not sure why it's not using all of this: https://paste.debian.net/1309912/ 15:15:45 Once wallets update and correct for a fee market, would the low txs stuck in mempool eventually drop off after 3 days? https://github.com/monero-project/monero/blob/master/src/cryptonote_config.h#L102 15:16:56 If yes, would it make sense to reduce the drop off time to a day or maybe a few hours to clear out the mempool faster? 15:17:50 No 15:18:20 Afaict, Under proper fee pressure the tx should get confirmed 15:28:11 hi guys, I'm newbie here, is someone facing network problem now? 15:28:50 the network is running, you might have problems if you use public nodes that are overloaded 15:30:14 thank you, i'm using my own one 15:30:59 what issue are you having? 15:31:32 .merge+ 9222 9223 15:31:32 Added 15:40:11 thank you, we'll chack 15:40:18 thank you, we'll check 15:41:49 luigi1111: are you available to merge the remaining PR and then tag? 15:41:59 .merges 15:41:59 -xmr-pr- 9218 9219 9220 9222 9223 9224 15:42:07 "the remaining PR" 15:42:14 :D 15:42:24 there's been a bit of activity today :D 15:46:09 haha. Usually don't like merging things sub 24hrs, but these are probably simple enough 15:50:37 hyc can you comment on the questions on 9222? 15:50:59 I'm running 9222 on my node now and it works as expected 15:52:06 I'm not concerned just want "double confirm" since it's so short 18:39:24 will take a look 18:42:45 yeah it's all good 18:44:00 nice 18:44:13 luigi1111w I guess you can merge 9222/9223 now 18:44:54 thanks 18:47:02 i found one more issue that might have to be addressed 18:47:26 which is? 19:33:34 Hi is there a changelog for v0.18.3.2 20:06:03 there will be once it's released 20:08:55 if it isn't too late is there a way for merging 9211 before release ? 20:09:24 low priority but have been ready for a week now 20:10:11 Not too late 20:10:57 rbrunner moneromoooo hyc or who is here now, please review https://github.com/monero-project/monero/pull/9225 20:14:25 s​omeoneelse495495: can you squash your commits? 20:16:47 I honestly don't know how to do that, but I guess I can learn asap 20:22:43 If you use Github desktop, you can select commits (holdint Ctrl and clicking on them), then right-click and select "squash" 20:24:05 github desktop is for racists ima use cli like a good dev 20:26:04 https://www.git-tower.com/learn/git/faq/git-squash then, but github desktop is many times faster for these kinds of things 20:33:16 done 20:40:01 sech: just now saw your PR lol. We found the txcompare issue concurrently: https://github.com/monero-project/monero/pull/9227 20:40:07 good find 20:43:25 lol, nice 20:44:30 approved 20:44:56 Someone should make a reddit post telling miner operators to restart their daemon every few hours until they update to the new release 20:45:28 can you also approve #9226 20:45:44 That will reset `m_txs_by_fee_and_receive_time` and get those txs unstuck in the short term 20:46:20 not necessary 20:46:22 Done. Gratuitous changes in the compare func but meh. 20:46:28 transactions arrive to different nodes at different times 20:46:34 so someone will mine them eventually 20:49:12 Yeah eventually, but it's practically affecting tx wait times rn, probably since individual pools have large shares in current mining landscape 20:50:02 selsta when will it be released? 20:51:11 yes, restarting nodes should help 20:51:56 Guest2: if we don't find another bug then we will tag today, and hopefully release tomorrow or sunday 20:52:22 wow, this one has been stuck for 20 hours https://xmrchain.net/tx/bcf3f15f1a877a0be48ec5a498c4b92d5d3a3200db452b7ec615b59c08e07dd4 20:54:00 Yeah there's 22 txs that have been stuck for 3+ hours 20:54:13 https://p2pool.io/explorer/ is running the curent release-v0.18 branch + #9226 on top of it 20:55:45 https://xmrchain.net/txpool shows bigger times (scroll down) 20:55:53 probably because it wasn't restarted recently 21:09:09 this one has been stuck for 16 hours despite having higher than minimal fee https://xmrchain.net/tx/32cbd8640fddcf7e9d449382a3d6a4521f065d90f27a7c72984ef2c6d690d5ff 21:09:19 and a non-standard fee at that 21:11:05 thats what you get for putting a non-standard fee bozo 21:12:14 all flood transactions use lower fee 21:13:32 was just joking 21:13:53 @sech1 @selsta jeffro256 the cli wallet's multisig tx building is creating txs with fees lower than low on single sig despite the priority being set as normal. A bug in the tx builder not taking into consideration the higher multisig tx size perhaps? 21:14:59 maybe 21:15:09 so they don't get propagated because of too low fee/byte? 21:15:15 Yup. 21:16:03 weird, how did this never get noticed before? 21:16:13 or did they get included when the txpool was empty? 21:16:27 I'm guessing this. 21:16:39 no, nodes discard everything less than 20000 piconero per byte 21:17:20 I think they allow 5% leeway but not more 21:18:44 My mistake, not lower than low. Almost as low as low. 20000 for single sig low and the multisig normal is being build with 23000 21:18:55 being built* 21:19:29 hmm, that transaction is exactly 23000 21:19:46 Not exactly 23000, I rounded. 21:20:17 what fee does it use when you create it with low? 21:20:27 no, it's 22000 there 21:20:38 Will test 21:23:02 Wdym "the higher multisig tx size"? Multisig txs should look exactly the same on-chain as normal AFAIK 21:23:31 he means the fee calculation doesn't account for multisig 21:24:04 Right. 22:04:55 Hello everyone. I am a highly motivated noob coder. I would love to contribute to this project to the best of my abilities. How can I find out what needs to be done to help this project succeed? 22:27:15 @selsta @sech1 jeffro256: apologies, seems to only be a thing in testnet multisig txs. 22:27:40 Mainnet all good 22:33:15 https://www.getmonero.org/get-started/contributing/ 22:37:05 Guest92: DMed you. 23:34:02 .merges 23:34:02 -xmr-pr- 9219 23:34:28 .merge+ 9225 9226 23:34:28 Added 23:47:52 s​omeoneelse495495: will include your patch next time, it's not 2 weeks old yet and not urgent enough to include early 23:48:08 .merges 23:48:08 -xmr-pr- 9219 9225 9226 23:48:18 luigi1111w: we have everything for the tag 23:48:21 after these 3 are merged