whoever is running p2pool at 142.202.205.118, check your monerod, it's probably stuck

memory leak check: yesterday it was 3469620/37780/10096 (VIRT/RES/SHR), today 3470972/52032/10096

hyc I had the same problem with zmq_reader - it doesn't wake up until it gets a message, so I send a fake message on shutdown

Node running at 142.202.205.118 is constantly 1 Monero block behind for whatever reason, so other nodes reject its p2pool blocks. We have a lot of orphans from that node because of that. But it's all orphans from that node

not constantly, but it's syncing with Monero chain so slowly, literally 1-2 minutes later than other nodes

would e.g. it beign an rPi with software AES possibly explain such lag?

might be

my node has 0 orphans, so it's not my problem :D

rPi with slow HDD probably :D

I need to think about banning such nodes still. My node sends full unpruned blocks to it every time because it's so far behind. This is wasteful for the bandwidth.

that lagging node is not an rPi, it's crowncloud VPS

probably something like 2 vCPUs and 1 GB RAM

and they do have HDD servers there, not only SSD

31 orphans, and it's only those that came from non-lagging Monero blocks: paste.debian.net/hidden/11ddf392

I bet that node has thousands of orphans

Even my shares have 3% uncles, it was 1% yesterday. Lagging nodes are bad

But it's like the worst possible condition because p2pool runs at 1 second block time and that node is literally on the other side of the globe, relative to mine

Uncle blocks are worth 20% less with default settings

<sech1> "I need to think about banning..." <- Banning like just for a hour or something only for you or globally? Also does p2pool require full blockchain to be downloaded or it can be pruned blockchain/public node? Also isn't 1 second too small? If somebody have ping larger than 800ms to another note it will create an insane lag (and if I understand that correctly, it can cause a softwork)

> <@sech1:libera.chat> I need to think about banning such nodes still. My node sends full unpruned blocks to it every time because it's so far behind. This is wasteful for the bandwidth.

* Banning like just for a hour or something only for you or globally? Also does p2pool require full blockchain to be downloaded or it can be pruned blockchain/public node? Also isn't 1 second too small? If somebody have ping larger than 800ms to another note it will create an insane lag (and if I understand that correctly, it can cause a softfork)

Please don't edit messages, it spam IRC side

Banning locally. We run with 1 second block time to run a stress-test

Pruned Monero node is ok

High ping by itself shouldn't be a problem, uncle blocks fix this in 99% cases. But that node is probably a low-tier VPS running on an HDD. Monero is already heavy on HDD, and if some neighbour on that VPS does something heavy too, it will lag - and it does

Other p2pool nodes don't tolerate when someone sends p2pool block build on top of an old Monero block because it useless for Monero side. So they just ignore such blocks and the node that sent them will get orphans

But they don't ban such node (yet). I need to think of when to ban these nodes without making too many false positives

Okay, I think that I get it now, thanks sech1!

sech1: uv_async_send() is the fake message to wake up the event loop

hyc I also submitted my block cache stuff and restarted p2pool node at 148.251.81.38 using the filled cache

let's see if anything breaks

cool

cache is just a memory mapped file that's flushed in the background. When p2pool loads it, it uses the same code path as external downloaded blocks, so it still does all checks

but it should work fine with corrupt cache

initial sync with cache will be limited by CPU - mostly how fast it can calculate PoW

10-20 seconds on modern CPU without this much logging

hmm, if cache gets corrupt it will ban whatever client it's syncing from when it gets to a corrupt p2pool block :D

prob should see that it's using the cache and skip that banning step ;)

yeah

Why do blocks with bad PoW get into the cache in the first place ?

they don't

but cache is on the disc and it's used to skip network requests for blocks on startup

and it can get corrupt in uncountable ways :)

Aleph zero corruptions on the disk, aleph zero corruptions... Take one and fix it...

actually block cache was supposed to have a keccak hash check before using the loaded block in the first place

But I forgot it :D

with this check random corruptions will be detected, but maliciously inserted blocks... On the other hand, it's a local file and if someone does it, system is already compromised

lol SChernykh/p2pool 4837b4c

now it checks that's everything is good, from binary format to keccak hash

oh, that west-coast VPS is no longer running, 0 orphans again

btw we passed 100k blocks mark \o/

so the miner connects to his own local p2pool instance?

yes

it can run on a different machine, it's just a stratum server

and currently testnet for a while longer to iron out wrinkles?

yes

probably quite a while

we still need to test RandomX epoch change - how nodes handle it, how syncing for a new node works when epoch change is in PPLNS window etc

it's every 2048 Monero blocks

next epoch change on testnet is in ~2 days

I run p2pool debug build (with address sanitizer) on my dev pc and it hasn't crashed, so it's quite stable. I'm more worried about sidechain syncing bugs

soft forks, net splits and so on

moneromoooo did you stop your node? I see only 2 wallets (both mine) in PPLNS window

no, I see 159.48.53.164 is still connected, maybe only xmrig stopped there

p2pool running, xmrig temporarily stopped.

ok

oh, mu p2pool died yesterday, didn't notice

and i didn't have cores enabled, dang

unlimited now\

died on which box?

I haven't got any crashes so far.

it's probably OOM killer again

dmesg usually records OOM kills.

My node just banned 142.202.205.118 (super-lagging west-coast node). It broadcasted a block that lagged so far it couldn't be verified by my node

it was also mining at Monero height 1795092 while my node was already at 1795123, 1 hour behind

This is why you don't run heavy stuff on a cheap VPS with HDD

actually both my nodes banned it at the same time and for the same reason

yeah, dmesg shows it was an oom kill

too bad it doesn't work on my mac

trying it as a debug build now

sech1: this is pretty consistent on mac paste.debian.net/1208872

I'll try to figure out why

hyc your node keeps asking for block id 850a4148f33ba689d4fdf74c8d638a885cb66562dd72c9fd80be988381815270

can you grep your logs for it?

because this block id doesn't exist on my node

hm, that's not in the log on my mac

it comes from 84.203.25.127

ah yes it's all over my rockpro64 log

my node mined it while it was syncing, so it has sidechain height 10 or something

then I restarted it so now my node doesn't know this block

NOTICE 2021-08-24 16:20:02.6860 P2PServer sending BLOCK_REQUEST for id = 850a4148f33ba689d4fdf74c8d638a885cb66562dd72c9fd80be988381815270

NOTICE 2021-08-24 16:20:02.7395 P2PServer block 850a4148f33ba689d4fdf74c8d638a885cb66562dd72c9fd80be988381815270 was received before, skipping it

the thing is, this block was broadcasted to your node so it should have it

mostly it's BLOCK_REQUESTS

damn I think now every node wants this block :D

well, this IP address is my router, the actual node could have been either the mac or the rockpro64 depending on which was runinng at the time

and it whatever block references it doesn't get pruned for some reason

just restart both your nodes and let's see if it forgets about this block

ok

I need to tweak pruning logic to deal with these border cases

hm the rockpro64 was already restarted due to oom

I have 138 messages in 20 minutes of this: P2PServer got a request for block with id 850a4148f33ba689d4fdf74c8d638a885cb66562dd72c9fd80be988381815270 but couldn't find it

Otherwise seems to work fine.

it's harmless

but it creates some spam on the network, so it's better to fix it

well I'm keeping the rockpro64 off now since it doesn't have enough RAM

so this is where the mac aborts

106 // b.m_lock should already be locked here, but try to lock it for reading anyway

-> 107 const int lock_result = uv_rwlock_tryrdlock(&b.m_lock);

perhaps tryrdlock doesn't like it if you already have a readlock. sounds bizarre tho

Default mutex on some platforms is recursive, on some it's not. Might be related.

it's not a default mutex, it's read-write lock

but actually this is what might be happening there

hyc can you try to just remove that lock? I think it's not needed and the comment above is correct

it's already locked at this point

yes, I just double checked. It's called only from one place and it's locked there

ok I'll try that

definitely got past that now

should I PR this too?

ah I see you already did, cool

hyc are you trying to change p2pool config? If your consensus id changes, other nodes will disconnect you

I see it in my node log that you disconnect at handshake step

I haven't changed anything

but I suppose the mac would have gotten a different node ID than the rockpro64

how would I set that?

status says I have 4 p2p connections

Hmm, I restarted my node and it connected

NOTICE 2021-08-24 18:37:22.7127 P2PServer peer 84.203.25.127:37890 sent HANDSHAKE_CHALLENGE

followed by an immediate disconnect

can you search the log for my IP around that time? This is CEST time zone

logs should always use UTC ...

just search for any warnings with IP 31.208.56.53

I think I found it

NOTICE 2021-08-24 17:37:00.6790 P2PServer new connection from 31.208.56.53:62106

NOTICE 2021-08-24 17:37:00.6790 P2PServer sending HANDSHAKE_CHALLENGE

NOTICE 2021-08-24 17:37:00.6790 P2PServer peer 31.208.56.53:62106 sent HANDSHAKE_CHALLENGE

WARNING 2021-08-24 17:37:00.6790 P2PServer tried to connect to the same peer twice: current connection 31.208.56.53:62099, new connection 31.208.56.53:62106

NOTICE 2021-08-24 17:37:00.6790 P2PServer peer 31.208.56.53:62106 disconnected

aha

not a problem then, my node just thought it was a different peer on the same IP

it would've disconnected if it got HANDSHAKE_CHALLENGE from your node, but your node found it first and disconnected

so no issue here then

"gmtime may not be thread-safe". If you have better idea how to get UTC time in a thread-safe manner, you can do a PR (log.cpp, Stream::writeCurrentTime())

There is a gmtime_r IIRC.

Might be a GNU extension though.

not a part of C++ standard

not a big problem though, I already have #ifdef there

Then I guess gettimeofday ?

I'm 99% sure we can just replace localtime_* with gmtime_* in the code and that's it

yeap

yeah they have identical API/semantics

my node is trying to connect to yours twice again for some reason

did it get saved twice in p2p_peers?

I check the list of connected IPs first before making new connection

so it shouldn't happen

hyc I think this is what's happening: my node gets ECONNRESET on connection to your node and then tries to reconnect, but your side of old connection is still connected

WARNING 2021-08-24 17:08:56.1699 P2PServer client: failed to read response, err = ECONNRESET

NOTICE 2021-08-24 17:08:56.1699 P2PServer peer 84.203.25.127:37890 disconnected

and then this loop of handshake-disconnect starts

this is UTC time now

hmm, I probably need to drop connections which weren't alive for some time (no meaningful packets sent)

hyc there's definitely something with connection to your node. My node sometimes connects to yours, but then it doesn't receive updates from it for some time, and then receives a bunch of broadcasts and ECONNRESET shortly after

don't know what changed, but it's fine now

nope, ECONNRESET again

is this normal "2021-08-24 17:44:41.716 W There were 73 blocks in the last 90 minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Monero network or under attack, or your computer's time is off. Or it could be just sheer bad luck." ?

on testnet?

yes, I get the same

it's "large hash rate changes"

yeah testnet

because testnet was at 2 kh/s and now more people are mining it

got it

do you need some hashrate for testing ? or just running p2pool and node

p2pool, node and xmrig mining with 1 CPU thread is enough

hashrate will be needed for mainnet testing

not now

well, this is the M1 mac, which also had connection leak issues in monerod

so it's probably just a flaky build

it doesn't change the fact that other nodes need to handle flaky connections properly

I wonder if I can find newer or older gcc for it

i can donate 1khs of mining power to some p2pool if you provide me ip, but can't run p2pool at the moment

1kh is prob excessive

for testnet

p2pool on testnet is already like 75% of the whole network

good that it can't do 51% because it's decentralized :)

lol

I see 6 unique wallets, ATH \o/

nice

6 payouts: testnet.xmrchain.net/tx/d45d5bbaa73…9604093a3fd67e142d4a215964617edf3df

i see this from time to time, don't know if it's a problem "P2PServer peer 148.251.81.38:37890 is ahead on mainchain (height 1795263, your height 1795262). Is your monerod stuck or lagging?"

my internet should be good

if it's ahead by 1 block it's not a problem

different nodes receive new Monero block at different times

it can also happen if some peer mines a Monero block and sends p2pool block to you. Then you will see +1 height, because p2pool block almost always gets to you faster than Monero block

I should probably only print this in log if it's height+2 or more

how long does it take for peers to forget my node's consensus ID? should I just turn it off for a while?

this ID is derived from p2pool config, it's the same for everyone

your node IP is forgotten after 10 failed connection attempts IIRC

ok

seems pointless to have so many connection attempts paste.debian.net/1208914

the reasoning is maybe that node is just restarting, so 10 attempts

so many redundant connections

from my side it looked like broken connections, I have a lot of ECONNRESET warnings

from my side: paste.debian.net/hidden/90507fc5

so it saw only 1 disconnect

interesting. perhaps it should log the local addr:port too

my local addr:port wouldn't be what you see because I'm behind not 1 but 2 NATs :D

ah...

even 1 NAT would change it

and yet you were able to forward ports in?

no

nothing works here for incoming connections

ah. sounds like a common home network

interesting that you have a pending connection attempt. should have gotten ECONNREFUSED right away

gonna assume 10 attempts have passed. starting up again

yeap, your node is not in p2pool_peers.txt

cool. I see no redundant conns in lsof now

I see no new warnings in log so far

4.5 kh/s on sidechain, more than network hashrate

somebody ramped up

hmm

maybe I shouldn't make sidechain diff higher than network diff :D

hmm... but it's not higher because 1 second block time.

so not a problem

my monerod has 56 inbound conns. never knew so many nodes were on testnet

10(out)+61(in) connections here

not here, but on my server

btw, my p2pool currently shows 579MB used

monerod only 327M

is this the new build with block cache?

552452 KB on my server

so the same ballpark

looks like yes, has block cache

327d1455fe4f861df82624d4c5a4d6eb4b15a7a7

it's not the latest, but was pulled today

yes, it's with the cache

cache is 480 MB, but I think not all of it is in physical RAM because of how memory mapped files work

likely

well the mac has 32GB RAM and not much happening on it so it could all be resident

5120 blocks * 4 KB (blocks are small now) = 20 MB most likely

Mac M1 has 4 KB pages, right? Or bigger?

16KB

so 80 MB

cache uses 96 KB per block and most of it is never touched now, so it's probably not in RAM

no big deal either way

moneromoooo is it safe to submit block template with enough PoW on a different Monero node? I see that p2pool blocks arrive before Monero blocks (mined by p2pool) quite often

so p2pool could broadcast mined Monero blocks more efficiently

for example block testnet.xmrchain.net/block/1795518 - my node received p2pool broadcast of this block 0.1 seconds before it got ZMQ notification from monerod

or maybe it's monerod that took 0.1 seconds to add the block and then send notification

could be

check timestamps in monerod log for block receipt

I'm not sure it's safe to submit the same block concurrently

if it's not safe that'd be a pretty silly vulnerability

which log level in monerod is needed for block receipts?

eh, level 1 ought to be enough. don't recall

I'll wait for next p2pool block then with level 1

confirmed 0.1 second delay: paste.debian.net/hidden/09d05249

p2pool is faster

this is testnet.xmrchain.net/block/1795523

p2pool block id 708f76dc5ffa092798664b4cc5821f1b410607de729d7226d659da064b3ba993 which can be found in tx extra for that block, so it's 100% that block

I'll add submitting of broadcasted blocks tomorrow and we'll see if anything breaks

So p2pool will be able to speed up block propagation in Monero network, wow

cool

but only for blocks it mines

I do not understand that question.

if monerod receives fluffy block and at the same time the same block via RPC, is it safe?

i.e. no crashes

Should be safe.

I assume it should be protected by a lock somewhere

ok

It is.

so a p2pool miner found a block, and broadcasted it to all other p2pool peers

it would also have submitted it to its local monerod

so its local monerod broadcasted the block more slowly than p2pool ?

yes

first, because there were more hops over the Internet

second, because monerod checks PoW in light mode (+10 ms on each hop)

and also does a bunch of other transaction checks before broadcasting it further

you can see in my paste that the block came from a different IP address

sounds ok. so all p2pool miner's monerods can be updated a little bit ahead of rest of network

IIRC supportxmr has this fast broadcast system between its nodes

to reduce orphan rate

clusters of wafer-scale compute ... wired.com/story/cerebras-chip-cluster-neural-networks-ai

GPU-friendly PoW will crumple

p2pool 581M now. so, some slow growth happening. will check again tomorrow