552452 KB yesterday, now 567816 KB. p2pool has a nunmber of preallocated buffers that can grow with time, so it's not 100% conclusive that there are leaks

hey

i'm pretty excited about the p2pool launch

I was just talking to a friend the other day about ways to make it so solo mining is actually worth it and then he found the reddit thread a couple days later

What was the reason this has not happened on BTC?

i'm not really sure

I think I saw some explanation for it, but can't remember what it was.

as it seems to be an excellent way to decentralize mining

it does, and there is a ton of potential for real-time web monetization if we can cut out the middle man (pools) now

like you could make users generate X number of hash per bytes of data transferred

and mine it directly to your p2pool

node

i don't think a lot of people realize how game changing this can be, especially if we find something to replace javascript miners and make it so hashpower can be used to replace payments

I think the short of it is that p2pool is best suited for coins that are not already dominated by ASICs… since any big ASIC miner will run their own pool anyway

that's a good point, I figured it had something to do with ASIC resistance

speaking of ASICs, remember Linzhi 1400 MH/s ETH miner? They promoted it a lot and went silent. Now it turns out, they decided to not sell it and are mining themselves...

This is the essence of ASIC mining.

yes, p2pool is best suited for small miners, so only CPU/GPU mined coins benefit from it

lol market started heating up so they decided to ride the wave

I updated the code to reduce log spam. Moved some most useless messages to log level 6 (log level 5 is default)

"BlockCache couldn't open p2pool.cache, error 5", the file is there

Windows?

yes

error 5 = access denied

do you have write access in that folder?

or maybe you're already running p2pool instance that opened this file

it was running fine, i just updated and ran it again

kill task

it's not a critical error, it can run without cache

My Windows node doesn't have this problem and I restart it a lot

seems good now, after a closed the terminal, maybe it was still running

Hey! If I understand this correctly - p2pool is a separate thing from monero, right? Like it doesn't affect the network as it is now? And I'm curious about one thing - if there will be a malicious p2pool binary, what's the worst thing it can do? Can it skip payment to all the miners and just take everything for themselves, while still mining in p2pool?

And what will happen if I won't reach `0.0005` XMR in the payout window? Because I'm quite sure that it will happen in case when everybody abandon pool mining and switch to p2pool

p2pool is separate, it doesn't require hardfork to work, but it will require point release of Monero

no, it can't skip payment to other miners because other nodes will not accept such block

if you don't have shares in the payout window, you will not be paid when p2pool finds a block. But when you eventually find 1 share and p2pool finds a block after that, you'll get paid 0.0005 XMR

so low hashrate miners will have payouts every few days, but it will be the "whole" 0.0005 XMR at once

assuming that p2pool grows to 30 MH/s or more

if p2pool is 100 MH/s with 10 seconds block time and you're a 500 h/s laptop miner, you'll be finding pool share every ~23 days and then get a 0.0005 XMR payout (maybe more than 1 if the pool is lucky)

this is like the worst case for a small miner, still much better than solo mining

the worst thing a malicious miner could do? Probably withholding mined Monero blocks (attacker still can't take the reward, only withhold it from everyone). This will hurt pool profitability, but it won't gain the attacker any profit

Monero blocks mined by attacker I mean

but the same attack can be done with regular pools

Ah cool, so that sounds like solo mining v2 to me

solo mining with closer to "pool like" payouts

Would it be accurate or inaccurate to say that RandomX uses less energy than ASICs? (Apart from the silicon waste)

Context: monero-project/monero-site #1772#discussion_r693960265

in some sense yes

many of PCs mining Monero would otherwise be just idling and consuming power doing that

But similarly ASICs are more efficient in regards to hashes / watt.

hashes/watt is completely irrelevant

Why?

because it isn't interesting how many hashes you producs. What is interesting is how much energy is used to secure the chain

producs = produce

Yes, that's what I wrote in the linked comment.

But inside the same algorithm hashes / watt is relevant for energy efficiency.

except that it doesn't really matter, as the spent energy will increase until a new quilibrium of power cost vs block reward is reached

equilibrium*

I might have found what's leaking: P2PServer failed to initiate tcp connection, error EMFILE

Also: P2PServer failed to save peer list

And it's not EIO/ENOSPC.

Loads of shite in /proc/PID/fd, they're sockets. netstat does not show anything out of the ordinary.

good catch

I'm still on the commit hash where you added the build documentation.

so you hit max open descriptors limit?

That's what EMFILE says.

ls /proc/PID|fd | wc -l says 1025, ulimit -n says 1024.

ls /proc/47710/fd | wc -l says 181 for me and it's been running non-stop for more than 24 hours

still, 181 is a bit high

I'll keep an eye on this number

For the record, I have these changes too: paste.debian.net/hidden/bce87af9

I'll pull latest now and run with that, alright ?

it's better to get latest and run with --light-mode

Unless you want me to run stats/commands ?

I am also running with --light-mode.

I fixed a few thread races in latest

maybe leaks were because of them

paste.debian.net/hidden/12f5f14e is a good fix for the libs.

updated and restarted my node, now ls /proc/PID|fd | wc -l says 45

yes, it looks like this number grows every time I disconnect and reconnect to my node

so it's accumulating p2p sockets?

looks like it

I call uv_close() on them, not sure what's wrong

if I kill local xmrig, stratum disconnects it and I see the fd number go down by 1

stratum and p2p both use the same code

every time you see "peer ... disconnected" in the log, it's from uv_close() callback so it was successfull

It's going up and I've not killed xmrig yet. It was 59 shortly after startup (started less than an hour ago), now at 97.

I think I found a workaround, but it's ugly

libuv doesn't close the socket in uv_close() callback for some reason, so I just kill it with linux.die.net/man/3/close

Maybe waiting for shutdown first.

doesn't work, p2pool crashes with this force close

so maybe it close it much later, after TCP timeout or something

*closes

moneromoooo can you try paste.debian.net/hidden/b943586c ? It helped me

definitely works here, so I committed it

Restarting still gets me the "not enough data to verify" spam.

(I was running the version with the cache before the restart too)

I should probably rephrase that message

it means it can't verify the block right away and needs to download its parent/uncles

cache works through the same "download block from remote peer" code path, so you still get it

pushed it, now you'll have less spam at startup

running latest on my server now. For the reference: RES = 374456 KB, fd = 49

we'll see in a few hours

Still slowly increasing.

From 40 right after staring to 55 now.

51 here

these are probably legit sockets

64

still 51

not sure what else can be done there

I see the number go down when I disconnect from the server

I just restarted with latest too, res 556M

I have fd = 62, went up a bit

lsof -p PID | grep TCP shows a few dangling sockets, not sure how they appeared

ugh I've got over 100 dangling sockets

lemme do another pull

your connection has been flaky the whole time

I still get lots of ECONNRESET from you

after restart right now I have just a bunch of ESTABLISHED conns, so nothing looks bad at the moment

so I got 3 more dangling sockets and the latest in the list is an established connection from your IP

number of open descriptors growing is probably related to it

this is what \I have right now paste.debian.net/1209022

ok, I think I know the reason. It leaks after ECONNRESET

Now how to fix it...

btw, where is the block cache? I don't see a file/fd for it

p2pool 61235 monero mem REG 9,2 503316480 2493300 /home/monero/test/p2pool.cache

just do lsof -p PID

it's in the beginning

oh, you're on Mac

not implemented for Mac

ah ok

you can try mmap code for Linux to see if it works

will take a look, shouldn't be much different

pushed one more commit that will hopefully fix leaks after ECONNRESET

although I have no idea what I'm doing :D

ok, cache code built with no trouble

nice, so you can just adjust #ifdef there

yes

-#if defined(__linux__) || defined(__unix__) || defined(_POSIX_VERSION)

+#if defined(__linux__) || defined(__unix__) || defined(_POSIX_VERSION) || defined(__MACH__)

in block_cache.cpp and common.h

got a few dangling sockets, but now I see it's p2pool trying to connect to peers

want me to PR it or you want to toss it in yourself?

p2pool 63157 monero 38u IPv4 897611 0t0 TCP ... (SYN_SENT)

and then just p2pool 63157 monero 38u sock 0,9 0t0 897611 protocol: TCP

hyc I'll update myself, thanks!

cool

7 peers now. yay ;)

so dangling sockets are just SYN_SENT packets, it means they shouldn't grow in number

it depends on how long the system keeps such sockets. It might be very long

typical TCP timeot, 2*MSL

should be 2 minutes by default

so |I have one dangling now

p2pool 30072 hyc 53u IPv4 0x692d8611815b6841 0t0 TCP 192.168.1.214:51725->31.208.56.53:37888 (CLOSED)

I changed the code to send TCP RST packet instead of normal close when node shuts down

basically and TCPClient::close() call will send TCP RST

so you might get dangling socket after it

ahhh I think I know where it leaks. Socket that was in SYN_SENT state is not removed, which means failed connections (for whatever reason) are not cleared. Now I know where to fix

hmm, not always. Just saw SYN_SENT socket that later disappeared from the list, so it wasn't left dangling

and not all SYN_SENT are failures, some of them will eventually successfully open

I just need to try calling uv_close() when connect fails and check if sockets keep leaking or not

hyc moneromoooo found the leak: SChernykh/p2pool 230d3e4

running it on both the server and my dev pc

all good so far, no dangling sockets

RES = 369604 KB, fd = 45

it was leaking handles on Windows too, now the handle count in Process Exporer doesn't increase too

hyc it's better to get latest, because I see a lot of ECONNRESET form you in the log. I reverted "force TCP close" hack because it's not needed anymore

Testnet height 1796161 will be RandomX epoch change, nodes should regenerate cache/dataset and check shares using cache in the meantime

in ~30 minutes

PPLNS window = 2160 blocks (+111 uncles, 0 orphans)

Your shares = 0 blocks (+0 uncles, 0 orphans)

Next payout = 0.000919829079 XMR

Is that expected to have a payout with no blocks on the share chain ?

(I had xmrig down for a while)

I did find a few shares since. Might be why.

nvm.

next payout assumes that you will mine 1 share

maybe it's a bug if you're not mining at all, I need to fix that calculation

I had started mining again a couple minutes before checking status.

ok, next block should be RandomX epoch change, watch your nodes

looks like it went smooth, dataset updated and it was checking shares all the time

i have a ECONNRESET error in the log every 2 seconds now

WARNING 2021-08-25 16:09:00.8806 P2PServer client: failed to read response, err = ECONNRESET

bad peer ?

it's hyc's node

he needs to update

doesn't p2pool give up after a number of tries ?

no, it's your node connecting to his and his node just force resets connection every time

is there a way i can ban his node on my end ?

ok will update

The fd leak seems fixed here too.

sech1: this commit is broken. d246ee155231ac85337faa4d32fb71e03df10828

as I said above you also need to fix the ifdef in common.h

oh right

fixed

fd = 45, rock solid with latest code

leak is fixed

I thought my miner was paused but it was running the whole time. oh well, p2pool is caught up now\

RandomX epoch change, running nodes handled it fine

good

I restarted my node right after the change, and it failed to sync - "can't calculate PoW" or something

it worked after several restarts

that sounds annoying

It synced to the previous epoch block faster than it could calculate 2 RandomX caches

It's probably safer to let RandomX initialize fully before starting sync

yeah

actually the problem is only with "old" RandomX cache which doesn't have time to initialize. I run it asynchronously and p2pserver starts syncing before it finishes

I can just run in synchronously and that's it

yes, it was an easy fix

fd = 45 still, 0 leaks so far

The p2pool splits the blockreward into multiple outputs? Am I reading this correctly? I thought only one output was allowed per coinbase

no, outputs are unlimited in coinbase

So does this mean you can build massive blocks by having amazing reward distributed among 100,000 people?

*mining reward, not amazing reward

Also, that blockreward has a tx_extra. Does p2pool rely on tx_extra?

Or have we simply standardized tx_extra so that it is always on?

If you go over the median block size, you start decreasing the miner reward.

Right forgot thanks mooo

Monero also rejects a block that's more than twice the median block size.

Right, yes, didn’t mean to get this into a discussion about blocksize.

tx_extra is used to store p2pool block hash

Oh interesting. So this may also end the discussion about removing tx_extra then

Or at least affect it

the discussion was about regular transactions, not coinbase

I can help see if p2pool successfully builds on an intel MacBook, Debian, or Fedora later today if that would be of help

Sech1 oh right fair enough

What block hash in "tx_extra is used to store p2pool block hash" ?

sidechain block hash, it's stored in merge mining tag

for example, testnet.xmrchain.net/tx/90f30500309…947ce6ad866ccfe6a73edbeb9261012d99b

block hash stored is 40cdf18a1fe322530c393a41c338ad3ac767bababd01700d81df02d1b5aeb85d

you can grep it in the log

Oh, the hash of the *previous* block on the share chain ?

no, it's the hash of the side chain block that was merge mined with this Monero block

you can read the big comment in pool_block.h, I don't know how to explain better

sidechain block's keccak hash is used as this block id, this keccak hash is just stored in the "Monero" part of the block

this is how it ends up on Monero blockchain

changing any byte in Monero block template or in sidechain data will change the hash and also Monero PoW hash because it's part of the coinbase tx

I didn't know how to better solve this problem - it must be impossible to change sidechain data without changing PoW hash, and PoW hash by definition must be Monero's PoW

OK, the nonce being zeroed makes this make more sense :

So does this make it incompatible with merge mining ?

(I actually use merge mining)

incompatible how? It is merge mining

Hmm, OK. I guess it must be since it has an extra chain...

does this mean p2pool plus towforget merge mining will override each other?

townforge&

they shouldn't

as long as hashes both chains that hashes in the merge mining tag belong to them

*as long as both chains that hashes in the merge mining tag belong to them

damn, I need to take a break :D

heh

when you see merge mining tag, don't just assume it's your chain, check that the hash belongs to it first

btw, submitting external p2pool blocks with enough PoW to monerod works fine

It reduces latency between p2pool and Monero block ZMQ update from 0.1 to 0.03 seconds on my machine

yeah, didn't think it'd be a problem. would've been too obvious as a DDOS

even better, only 13 ms latency: paste.debian.net/hidden/e798ce2c

probably because of slow PoW check on monerod side

btw the last line in this paste is exactly what I was talking about. p2pool checks the hash and tries to search for it

is there any way to make monerod use dataset to check new block PoW?

yes, but it only allocates dataset if you activate its miner

hmm, and then it uses also to check new blocks?

yes

nice

1 solo mining thread won't hurt

doubles hugepage requirement

add xmrig to that :D

My server has been running without tcp socket leaks for several hours, I guess it's really fixed

mine looks fine too

If p2pool just puts the hash of its block template/data in the MM tag, it'll conflict. If it puts the merkle root of the MM hashes, it should be compatible.

I don't see tre_branch/tree_path used in the source, so it's likely it doesn't put the merkle root.

what merkle root? There are not transactions on sidechain

*no transactions

Merkle root of merge mined blocks.

I need a bit more detailed description

You have two chains A and B merge mining with Monero. A creates a block template with hash HA, B another with HB. The MM tag doesn't get HA nor HB, but the merkle root of the merkle tree with HA and HB in it.

But I have 1 chain

My MM proxy in TF does this automagically (I made this to be compatible with tari).

Yes, that's the point :)

I think I get what you mean

I'm not nearly as technical as you guys, but I'm kindof curious what kind of replacements are feasible for javascript mining? Would it be possible to mine to a p2pool node via a firefox/chrome plugin?

1 Monero block can be on a block on 2 different merge mined chains?

If you use the merkle tree system, yes.

If you don't, the two chains are incompatible for merge mining.

But then p2pool will need to be aware of those other chains

tbh the details are... a bit complicated.

It would have to be aware of multiple chain-icity, not any particlar chain.

Like, my MM proxy *should* work with Tari, but doesn't know tari at all.

Mysterion p2pool node is the same as a regular pool to the regular miners (xmrig)

But I've only tested it with two TF chains.

yeah I just know javascript mining has pretty much been kneecapped over the last couple years, I'm just wondering if there are any alternative solutions to make it viable again via in browser mining

Not with RandomX

i've been out of the mining scene for a couple years, I don't really know much about RandomX

There is repo.getmonero.org/selene/primo, which can be used to mine with a browser.

The actual mining's done by a non-js process though, just controlled by a browser plugin.

I actually did see primo, it looks promising

This fixes the "website stuffs js without hte user's knowledge" problem too.

Because screw running any random site's code.

I still don't get why would 2 different merge mined chains care about each other

p2pool expects its block template hash in the MM tag, right ?

yes

There is one MM tag. I guess it might work if there's one MM tag per chain... Though it would increase the Monero block size.

yeah I like a more transparent solution, I don't necessarily want site ops to have to be shady about mining, best to be up front about it as if you were requiring a payment to get behind a paywall

If there's several, I guess they don't need to. But if there's a single MM tag, one chain is not going to find its block hash in there.

why? It will find its hash because it was mined by it

What is "it" in this sentence ? p2pool ?

something like client generates X number of shares and then they get an access token that's available for X amount of seconds/minutes/hours/days or maybe it could be valid for X amount of bytes, just depends on how the site operator wants to configure it

p2pool

only one subchain will find any given block\

p2pool will find the hash in blocks mined by p2pool

Then yes, p2pool will find its block hash. The other chain will not.

so they don't have to care about the existence of other mergers

If you place a merle root, both chains will find theirs.

Or all three etc.

If I place merkle root, I need to somehow find other chains' hashes before looking for my own

that's more unnecessary complexity

Primo does that. You mine ahead of time to generate N credits, then you spend them to access various pages/features/etc.

For my particular case (and AFAIK tari's similar), you don't point xmrig to monerod, but to the MM proxy, whch does the magic.

Similar for you actually, you mine to the p2pool proxy.

there's gonna be some really cool stuff possible when you combine p2pool and primo

So you'd have a chain of proxies. There's a new RPC in monerod I added a while bakc to help with that.

Anyway, I'm not asking you to do that. I hope you might consider doing it but I certainly won't push :D

It'd also be possible to integrate it into the ZeroNet protocol so peers could automatically pay eachother in shares to seed sites to eachother

I need to first understand it :D

Let me try to remember how it actually wors...

sooo my wild speculation: you mine with xmrig and point it to MM proxy, you put merkle hash into the block template and proxy can submit found shares to multiple chains, right?

depending on the difficulty of share

The proxy receives a getblocktemplate notification. It forwards it monerod (or the next proxy). It gets it. It adds its block template hash to the response and forwards the response back.

Yes.

so you main with xmrig to all the chains at the same time

hmm, interesting

well, the easiest thing would be to not use merge mining tag (for now)

If the getblocktemplate comes back with a MM tag in its response already, the MM proxy adds its tag and does merkle magic and reutrns that instead.

p2pool would use some different tag in tx_extra for own block hashes and some time later would also put merge mining tag, right?

Well, ideally it'd be better if you played along with the others, so all hash rate could accumulate.

Without adding more data in the monero block. I guess 33 bytes isn't that large but still.

"more hash rate would accumulate" that would probably break p2pool reward calculation...

if other chains don't know about p2pool, they wouldn't notify p2pool nodes about their hashrate

that's right, looking at the Primo page I remembered what my concern was with it...it requires Apache. I guess everything has to start somewhere, but that'll be a problem eventually

Any server can use primo. Only apache has a plugin, but you could have one for tor, bittorrent, whatever else.

I see

The only problem being "is it a good idea".

I think so

Any service that requires payment for service can be patched to squawk primo.

it may not be the proper implementation but the concept is definitely something worth trying to perfect IMO

I can't find the namecoin doc I used, but this has the rough idea.

we're essentially creating a way for people to use electricity as currency

kinda crazy when you think about it

Otherwise, there's the MM proxy source: git.townforge.net/townforge/townfor…_mining_proxy/merge_mining_proxy.cc

It's a big chunk to swallow, but the interesting parts are:

add_add_aux_pow, on_get_block_template

Those will alter the block template to place the merkle root of all MM chains so far in the MM tag.

I feel like I forgot how this works -_-