.merges

Hey y'all I wanted to start a discussion about community nodes and start asking for community node operators: monero-project/monero #8624

For anyone that was interested, the demo yesterday youtu.be/aAvSpfll9z4

struct SpendableOutput... (full message at <libera.ems.host/_matrix/media/v3/do…a0b55d649b8b50811a29ec82657b6226123>)

rct is short for ringct, which is short for ring confidential transactions.

That's the current signing sytem monero uses.

As an optional string, I'm not sure what the semantics would be in your structure though.

ridauhebauye[m]: yes, I am updating monerod

your suggested change wouldn't help much becaue an attacker can have 1+ connections; an outbound connection is not controllable externally

sorry meant 2+ connections to an IP

intentionally skipping that link creates a trivial to identify situation

is there any way to implement a consensus algorithm that's <= O(log n), where n is number of transactions?

the question doesn't make sense. do you have an example of an algo that is > O(log n) for your definition of n?

cvmn: the PoW consensus algorithm is independent of tx count, maybe you could clarify

hyc: yeah. e.g. bitcoin for example. as transactions are added (including mining announcements with hash proofs; enough zeros, that is), they add a blog stuffed with transactions.

the cosensus is implemented by the blockchain, which grows O(n) where n is transactions number.

the blockchain grows O(n) yes but that's not because of the consensus algo

as UkoeHB said, PoW consensus is independent of tx count

consensus is the longest chain, where length here is computational power. right?

no

You want to define your terms. It's so vague nobody can say yes or no.

"consensus" is the very vague term here.

Your second line implies chain size.

ok. to me, consensus is the mechanism to have an agreement about whether a certain number of transactions happened.

s/number//

Yes, but what part of it do you want to be O(log(n)) :)

consensus in blockchains has nothing to do with number of txns

And in number of what ops.

hyc: yes, not their number, but whether transactions happened.

Typically, it's "number of operations to verify a chain". but your second line implies it's not what you want.

\blocks are mined on a set time interval, they could be empty or full or anywhere in between. the number of txns is irrelevant.

Well, maybe it is. In which case, it's not possible, because you need to verify every single tx.

true. then, the longest chain is followed as the global agreement of where the world is.

There are shortcuts if yo're willing to forego security, such as allowing chcekpoints where PoS "protects" a ptoof all txes up to that point were verified by "someone", and you trust that.

in bitcoin, individual transactions are added to blocks. blocks with more transactions are larger. so, it's effectively O(n).

irrelevant

Monero is pretty similar to Bitcoin in all such things.

consensus is not based on the longest chain

Above, I meant PoW, not PoS, lol

hyc: what is it based on, then?

and 'length of a chain" depends only on the number of blocks, not the number of txns contained in the blocks

consensus is based on the chain with the most work, in PoW chains

"longest" is common speak for "with most cum work".

well, a block ls longer if it has more transactions. so, a chain effectively becomes longer as some blocks will be taller than others.

cvmn: no

OK, so you're on about size now. There are chains that allow "merging" some txes (Mimblewimble).

there are different types of lengths here. one length is the computational power and another is the length in bytes on disk. i'm referring to the bytes on disk.

Not sure how much you gain, but there's still a linear component AFAIK.

cvmn: still wrong

any idea which terms am i abusing?

You can also have a balance based chain. In that case you can collapse a number of things *after* you verify txes. That's close to O(1).

(it's O(a) wiht a being number of accounts)

Balance based chains are not privacy friendly at all though.

cvmn: I already gave you the correct definitions above.

You may want to lookup Cryptonite, the mini blockchain.

<hyc> \blocks are mined on a set time interval, they could be empty or full or anywhere in between. the number of txns is irrelevant. <--- this?

It was balanced based. AFAIK Ethereum also is. But has a massive unusable chain so...

that's part of it, yes

hyc: are you saying that, even the disk-space wise is independent of number of transactions?

the disk space has nothing to do with consensus

doesn't bitcoin's consensus require connectivity all the way back to the genesis block?

connectivity of blocks, yes

Are you really on about disk space (after verufucation) or chain size (data needed to verify the chain in the first place) ?

at least, during initial synchronisation, a full node will require to download all transactions.

but again - it doesn't matter how many txns are inside a block

txn validation and blockchain consensus are competely independent matters

txns are validated by the first node that receives them. can be rejected immediately if invalid. "consensus" is irrelevant. it is O(1) operation.

is it possible to validate a block's hash, without knowing transactions inside it?

yes

even without knowing hashes of transactions inside it?

a miner sees the hashing header only, not all of the actual transactions

so, we require having all hashes of transactions in a block, in order to verify that block. right?

<moneromooo> "As an optional string, I'm not..." <- I found it from here github.com/mymonero/mymonero-core-c…r/src/monero_transfer_utils.hpp#L64

Ask in #mymonero maybe. I think most people there are here too though.

what will happen if I have a rct, and how can I get it?

i wrote that codr

<jadeydi[m]> "I found it from here https://..." <- the rct is 'coinbase' or a mask

Hey everyone 👋️... (full message at <libera.ems.host/_matrix/media/v3/do…90cc61f4e83185746e36555a10ccf2f989d>)

Ideally, if you were to spend more time on this? Where would you like to take it ?

make a Rust full node implementation which would... (full message at <libera.ems.host/_matrix/media/v3/do…decdefacd6278cc72127f5feb40f9b7e276>)

> <@vorot93:matrix.org> Hey everyone 👋️... (full message at <libera.ems.host/_matrix/media/v3/do…b7ad70db63633adbb332e6b326c1e1ca5c1>)

> <@vorot93:matrix.org> make a Rust full node implementation which would... (full message at <libera.ems.host/_matrix/media/v3/do…e33da756d11398b76190c1840d4ab5c6fe7>)

is there a way to make the bridge post the full message, instead of a link to libera.ems.host?

ghostwaynoted

there's no public repo yet, and I put it on ice for now as I'm still prototyping the juicy bits (snapshots, libp2p) using working full node (Akula) first

once those are done, I'll copy them over and present an alternative to C++ monerod with killer-feature set

Yep just saw your tweet saying it's not public yet. I see, so no early-contributions heh. Good luck! Quite excited

Artem Vorotnikov: Sounds very interesting. Do you have a use case in mind? Or just indefinitely experimental?

Rucknium

- Much faster to sync up, which is relevant for mobile use case

- MDBX, which should give even better performance having been stress-tested and improved on multi-terabyte databases of Akula and Erigon

- 10x more modular, cleaner and accessible to developers because of staged sync and Rust - consequently much easier to add features like custom RPC calls

What is staged sync ?

also forgot:

- plaintext chainspec format, which allows for adding support for any network, provided it doesn't require any unsupported protocol features, example for an Ethereum testnet: github.com/akula-bft/akula/blob/master/src/res/chainspec/sepolia.ron

Yeah, what is staged sync. Much faster how?

moneromooo... (full message at <libera.ems.host/_matrix/media/v3/do…a5a023795c05bea2b07595e22b3197e497f>)

rbrunnercombined with other innovations, Akula can sync an archive Ethereum node (2+ TB worth) in under 48h on good machine

Compare this to Go Ethereum which can (non-state-sync) sync in..... months?

You're claiming doing this is much faster because of... cache friendliness ? Other ?

Anyway, if you can speed sync up noticeably while checking the same things, it sounds like a great win.

Staged sync is more about easiness and simplification of node development, but it's also faster because CPU-heavy logic can much easier be parallelized

Not sure about Monero, but at least in Ethereum we have extraction of tx sender from secp256k1 sig - since it's a separate stage we actually parallelize all of it for a massive win worth several hours of sync at least

Monero sync is not massively parallel. I can see how *that* could speed things up.

Some parts of verification is parallel, but more could be.

Monero sync's heavy on disk I/O. Optimizing this might get you good wins too.

Akula's data model is heavily optimized towards random reads, sequential writes, keeping within recommended storage engine parameters, so I/O is _not_ a bottleneck for us

but since you use (almost) the same storage engine, I believe it's not a big problem for you either (cc @hyc)

we've done a similar experiment in OpenLDAP, years ago. when bulk loading a database, just load entries first, and then iterate over attribute indices one at a time

while each individual pass is relatively fast, the sum of all separate passes still takes more time than ading entries and indexing at once

loading all the entries by themselves first certainly is sequential and fast. but index creation is always random access

unless you use external extract-transform-load like we do

then it becomes sequential

then you require an external memory or data store large enough to hold an entire index

If you've got the RAM, you might be able to create it with seq access by writing in already sorted order from RAM.

Well, yes.

sure, anything can be made to go faster if you have enough RAM

we don't do ETL on tmpfs exactly because indexes are large

and they would compete for RAM with MDBX' mmapping anyway

naturally, syncing a 2TB database on exactly 2TB free space is a bad idea

but it's always a bad idea for anything, at least on SSDs

we don't peak at much-much more than the final DB size though

anyway, have fun re-learning with MDBX everything we already tested and discarded in OpenLDAP 20 years ago

lol

The fun started already ...

It's much easier to have fun if you are young. Old people like hyc and I are pretty hard to excite anymore :)

lol

Is it correct to say that this Rust node would have to implement the communication protocol of the C++ `monerod` (in addition to any separate improved protocols) too so that it can communicate with the rest of the network?

Rucknium

maybe - there are several ways to go about it

I am most certainly _not_ planning on starting with rewriting Levin, if anything it's the hardest _and_ least important to get the node to function.

Rucknium[m]: I don't see how else you would expect to interact with the existing network

I would say so, yes. Otherwise you wouldn't have a single network

Akula started as a companion node to Erigon, leeching off its MDBX on-disk database

Yeah, if you want a proof of concept that demonstrates faster sync and superior modularization you just implement enough to fetch blocks from somewhere

then since I'm planning libp2p anyway, I could make an overlay network, connected to Levin-based network through such companion nodes

Levin is a stretch goal in any case, not a hard requirement _even_ for prod

how does erigon compare to turbo-geth?

ehm, Erigon was renamed from turbo-geth

ah

no wonder it sounds familiar ;)

naturally, it's even more shiny since it's quite a few commits away :)

so this still describes your ETL? github.com/BTAutist/turbo-geth/blob/master/common/etl

another problem with approaches like these are typical Windows users. the process you describe requires uninterrupted operation from 0 to tip.

correct, but each index stage only requires a few hours, not days, so 🤷‍♂️️

MDBX's freepage reuse policy, reuse most-recently-used pages first, will also destroy SSDs faster

that's not quite enough workload to significantly shorten modern SSD's lifespan

it depends entirely on how good the wear leveling in the controller is

Well it all depends on scale..

not something worth betting on, really

well then ancient/obscure hardware users are free to use LMDB :)

but so far Erigon and Akula have had zero reports about dead SSDs - and Ethereum's workload is magnitudes heavier compared to Monero

I am almost looking forward to the Monero dev team outed as hobby programmers and even dimwits and get a much faster and more modular Monero daemon in Rust as a recompense

We will get over it, believe me :)

lol

> <@vorot93:matrix.org> moneromooo... (full message at <libera.ems.host/_matrix/media/v3/do…510d54191bd18c726a6b3cf723069fd15bb>)

If you download all data first and then check its validity, you will find out that you received invalid garbage only after you've already downloaded it all and done all the checks

it may be garbage data, but it loaded fast!

lol

Artem Vorotnikov: all I can say, when you release something about this (or need a little contributer help), I'd like to get notified :)

Ethereum is also a BFT network, mind you, so it's _not_ correct that Akula assumes data to be 100% valid

That said, blockchain is at least partially spam-protected by PoW seal and if you have invalid blocks floating around with good difficulty, then new nodes syncing up is the _least_ of your problems

Also, I don't think treating levin as a "stretch goal" is a useful idea - if you can't talk to the other nodes, you're just excluding yourself from the network

ghostwayabsolutely

A malicious node could craft a chain fork with valid (POW-wise) blocks, and you'd only find out after you've already one through all the data

endor00overlay network bridged using companion nodes leeching of LMDB is a good first way to get going

> <@endor00:matrix.org> A malicious node could craft a chain fork with valid (POW-wise) blocks, and you'd only find out after you've already one through all the data

in practice it would mean that adversary has huge computational power enough to rival the network

again, in this case getting _new_ nodes to sync is going to be the least of your pains

and even so it can be mitigated

Doesn't have to: they could just mine a few blocks to start a fork in the chain, and your nodes that can't talk to the "regular" nodes would be none the wiser

well then you have eclipse attack and it's the problem either way, staged sync or not

merope: Sounds like it would just be impractical, it's pow

Right, but since you're creating a nearly-isolated network (at least initially), it would be quite easy to pull off

Not to mention the fact that the "bridge" nodes would essentially become trusted nodes

Couldn’t you just sync chunks at a time instead of the entire thing?

* ~~Sounds like it would just be impractical, it's pow ~~

Unless a user ran both implementations - in which case yours would be redundant

UkoeHBincremental staged sync, yes - we do have this mode in Akula

ghostway[m]: Double-spend attacks can be quite elaborate

> <@endor00:matrix.org>

> Double-spend attacks can be quite elaborate

Double-spend on unsynced node is interesting

If you want to replace levin with something better, the sane way to do it is to support both at first, and then push levin out after a hardfork (once the new p2p lib is deemed stable enough)

I'd expect established services to have synced-up infra, with backups even - at that point staged sync becomes regular (all stages for 1 block) sync

> <@endor00:matrix.org> If you want to replace levin with something better, the sane way to do it is to support both at first, and then push levin out after a hardfork (once the new p2p lib is deemed stable enough)

I'm not advocating for changes in monerod, I'm making a plan to make a new node and push it into usable state as fast as possible

Right - but if you forgo security in exchange for convenience, you're kinda missing the whole point :D

ArtemVorotnikov[: Many users sync their nodes on-and-off

if we were to talk about network-wide changes, it would make sense to integrate second libp2p-based node into monerod and phase out levin at some point

> <@endor00:matrix.org> Right - but if you forgo security in exchange for convenience, you're kinda missing the whole point :D

making things perfect immediately is kind of the enemy of organized software development :)

(looks like it was 17 years ago, nearly to the day git.openldap.org/openldap/openldap/…a1b622ed2cceea7bda54187e04e15eb8f6f)

There's a difference between "not perfect" and "design that introduces vulnerabilities" though ;)

If you want to deliver something that doesn't expose your users to trivial attacks, you need to be able to talk to the rest of the network directly, via levin

> <@endor00:matrix.org> There's a difference between "not perfect" and "design that introduces vulnerabilities" though ;)

there is a difference between vulnerabilities or practical real world vulnerabilities

I suggest you dig in Monero's history and look at the DoS attacks from a few years ago, then

anyone else around here have experience with libp2p?

> <@endor00:matrix.org> I suggest you dig in Monero's history and look at the DoS attacks from a few years ago, then

I'll worry about this when I have the staged sync pipeline done :)

a modular comms library sounds nice, but I'd prefer OpenLDAP liblber to Google protobuf

Given that they were directly targeting the sync mechanism, you might want to do that before you're done - or at least, I personally don't like doing things twice

obviously I'm biased, but it's already tuned for zero-copy reads and other stuff

> <@endor00:matrix.org> Given that they were directly targeting the sync mechanism, you might want to do that before you're done - or at least, I personally don't like doing things twice

there are always points of trust for joining the network, like bootnodes in monerod's source code

And since we're trying to build a trustless network, adding more of them (when we already know how to do things without them) would be counterproductive

> <@endor00:matrix.org> And since we're trying to build a trustless network, adding more of them (when we already know how to do things without them) would be counterproductive

do you plan to port-scan random IPs or?

that seems viable for bitcoin's network. not sure we have enough nodes

ArtemVorotnikov[: I was talking about adding trust in the data processing during sync, not peer-finding

> <@endor00:matrix.org>

> I was talking about adding trust in the data processing during sync, not peer-finding

if the worst comes to pass, and attacks become practical, it's possible to add checkpoints and run staged sync block-by-block - since the worst part is history that won't impede the sync significantly

as I said, staged sync is mostly about ease of development, we got most of our performance gains elsewhere

the monero node stuff probably needs significant work to support seraphis, so even a proof-of-concept for better node design could be impactful

hyc: since proving that there is a consensus that a transaction happened, one requires a chain of blocks from genesis, such that there is a block that contains that transaction's hash. since each transaction has a unique hash, therefore the space complexity of the blockchain is O(number_of_transactions)

.merges

whoa

cvmn: space complexity isn't a thing

a block depends on the hash of its preceding block, and the hashes of the txns it contains.

that's what gets fed to a miner.

*hash of hashes of the txns. a miner never sees the actual txns