<r​brunner7:monero.social> Meeting in a bit more than 1 hour

<r​brunner7:monero.social> Meeting time. Hello! monero-project/meta #1265

<s​needlewoods_xmr:matrix.org> hey

<s​yntheticbird:monero.social> hi

<j​berman:monero.social> *waves*

<j​effro256:monero.social> Howdy

<r​brunner7:monero.social> What are the reports from last week?

<r​brunner7:monero.social> I worked jeffro256 's review comments into my PR, but today he found some little thing more to polish :) monero-project/monero #9939

<s​needlewoods_xmr:matrix.org> same procedure as every week

<s​needlewoods_xmr:matrix.org> just one thing worth to mention, I didn't see a good reason why the CLI `set_daemon` command had no option for daemon login and proxy, so I added both options ([commit](monero-project/monero 78e22f7))

<j​berman:monero.social> Almost exclusively worked on PR 81, the PR to refactor wallet2 refresh and fix fcmp++ reorg handling (and last blocker for stressnet). I worked on implementing using the existing `short_chain_history` approach to request blocks with some modifications (the goal with `short_chain_history` is to minimize trips to the daemon / data downloaded when handling reorgs). I detailed the mod<clipped messag

<j​berman:monero.social> ifications and rationale in the PR description, and in a follow-up comment. It's a bit complicated. I think it would be solid to get consensus on the PR's approach seraphis-migration/monero #81

<r​brunner7:monero.social> So pouring all this work into `wallet2` still is in the interest of working well for FCMP++, right?

<j​effro256:monero.social> Reviewing @j-berman's PR #81 today, have been working on a multisig fix upstream for the last few days. A wallet2 reorg handling issue with multisig exchange messages has been plaguing Haveno ever since Qubic started all those reorgs

<j​berman:monero.social> I'd say yes. The seraphis refresh mechanism has months of work to get it to production ready

<r​brunner7:monero.social> I see

<r​brunner7:monero.social> Oh, multisig data exchange *and* reorgs, sounds like fun :)

<r​brunner7:monero.social> I dimmly remember thinking while implementing the MMS years back "Well, there won't be any reorgs while this code here runs, hopefully"

<r​brunner7:monero.social> And for a long time that was almost a given. Not anymore right now, of course.

<r​brunner7:monero.social> Is there an ETA for the first FCMP++ based network starting to run?

<r​brunner7:monero.social> With the nodes of a handful of testers

<j​berman:monero.social> 7 days after we get PR 81 in

<r​brunner7:monero.social> Ok :)

<r​brunner7:monero.social> Looks a bit scary from the description alone already, not sure it's within my reach to review. Will try to have a look.

<j​berman:monero.social> "Modification 1" is the main thing I'm hoping to get consensus on

<j​berman:monero.social> I can try to explain it a little more here with shorter context

<j​berman:monero.social> First a primer on `short_chain_history` and requesting blocks using the `block_ids` param in getblocks.bin

<j​berman:monero.social> The idea is to include a short, ordered list of hashes that the wallet has already seen. The daemon will then traverse this list, starting from the highest block included in the request (i.e. most recent), and upon encountering a block id that is in the main chain, the daemon will then return a contiguous set of blocks starting from that block id

<j​berman:monero.social> If there was a reorg, then the daemon will include contiguous blocks starting from the *first* block included in the request that is *still* in the main chain

<j​berman:monero.social> This way the wallet will know if there was a reorg based on the daemon's response

<j​berman:monero.social> *end context on short_chain_history/block_ids`

<j​berman:monero.social> Now on to this PR: I modified the core approach in wallet2 to make sure that the `short_chain_history` in every request to getblocks.bin *always* reflects the wallet's known view of contiguous blocks, making sure that the list of `block_ids` **always** includes the oldest block that the wallet can handle reorgs back to

<j​berman:monero.social> ALSO importantly, the request will always include the genesis block in it as well, so if the daemon serves blocks contiguous from genesis, the wallet will know it cannot handle the reorg

<j​berman:monero.social> This way the daemon response will **always** serve the blocks that the wallet can handle reorging, OR the wallet will be able to tell that it cannot handle the reorg

<j​berman:monero.social> (another piece of context and why there is a max reorg depth that the wallet cannot handle reorging below: the FCMP++ tree that the wallet is building locally is "pruned" so that it doesn't take GB's of space. It has a max reorg depth so that the wallet can prune the tree effectively)

<j​berman:monero.social> So that is the necessary context to understand how the PR aims to maximize benefit of short_chain_history

<j​berman:monero.social> Following this so far?

<j​effro256:monero.social> wallet2 doesn't include the block ID for max reorg depth already?

<r​ucknium:monero.social> What if `max-reorg-depth` on a wallet is set very deep? Will the list of `block_id`s fit inside an RPC message?

<j​effro256:monero.social> Just cheked... ig not lol

<s​needlewoods_xmr:matrix.org> with "wallet cannot handle reorgs", you mean without `rescan_bc`?

<j​effro256:monero.social> That's definitely an improvement

<r​brunner7:monero.social> "Following this so far?" Not yet, I would say, lacking basic knowledge how the wallet synchs so far in detail, but I think your context written here will be valuable

<j​effro256:monero.social> The short chain history uses exponential backoff, so it will only be logarithmically bigger

<j​berman:monero.social> The short_chain_history works like this: start with 10 most recent blocks, then go step-wise *2. So first 10, then 12th, then 16th, then 24th, etc. So going back to the start of max reorg depth is log 2. Quick someone do the maths

<j​berman:monero.social> right it doesn't

<j​berman:monero.social> Ok jeffro is following, so I think I will just continue and I think it will eventually click for all at some point

<r​brunner7:monero.social> Just curious: You write in the PR description "Correctly handles deep reorgs". Where is the limit of the old i.e. current code?

<r​ucknium:monero.social> Makes sense. So you get more of the set of block ids fitting inside of a message, at the cost of more messages.

<j​berman:monero.social> m_max_reorg_depth

<j​berman:monero.social> Old code (as in code on release branch) I don't think is broken though AFAIK. It's fcmp++-stage fcmp++ integration that this is fixing tbc

<r​brunner7:monero.social> Ah, now I understand.

<j​berman:monero.social> So continuing rn on the core thing I want to get consensus on: right now, wallet2 fetches the **next set of blocks** asynchronously in parallel to synchronously processing the **previous** set of blocks

<j​berman:monero.social> process_parsed_blocks is where the block processing is done

<j​effro256:monero.social> If I understand correctly, `rescan_bc` might not be usable for soft resets after FCMP++ since we need to download all outputs since the starting refresh point to build the FCMP tree

<j​berman:monero.social> and that is where wallet2 sync m_blockchain, the local data structure keeping track of which blocks wallet2 has synced

<j​berman:monero.social> sorry, yes, this is what I meant

<j​berman:monero.social> will double check rescan_bc soft after this

<j​berman:monero.social> Continuing from here. Since process_parsed_blocks internally handles syncing m_blockchain (including handling reorgs), that presents a synchronicity conflict: how to make sure we're fetching the next set of contiguous blocks in parallel without needing to wait for `process_parsed_blocks` on the prev set of blocks to complete, using the expected short_chain_history that reflects th<clipped messag

<j​berman:monero.social> e latest chain state

<j​berman:monero.social> I solved this by maintaining a second struct `hashchain cur_chain` that I read in the control thread executing the request for the next set of blocks in parallel

<j​berman:monero.social> That struct is distinct from m_blockchain, but I duplicate the reorg /sync logic to make sure that cur_chain handles reorgs the exact same way

<j​berman:monero.social> This way I maintain wallet2's approach to fetch next set of blocks in parallel to processing the prev parsed blocks

<j​berman:monero.social> I wrote about an alternative relatively simple approach here that would present another change to wallet2's refresh structure, but would avoid the duplicated logic of reorg handling: seraphis-migration/monero #81#issuecomment-3263948108

<j​berman:monero.social> And that decision is the thing I'm hoping to get consensus on

<j​berman:monero.social> I don't think it needs consensus in this meeting, but hopefully that context helps to reach conensus faster

<j​berman:monero.social> And helps explain the PR's approach

<k​ayabanerve:matrix.org> 👋

<j​berman:monero.social> rescan_bc soft is still good, it clears m_transfers, m_blockchain, and m_tree_cache. It just doesn't clear things like notes and labels

<k​ayabanerve:matrix.org> Apologies for being late. My only update is how the fcmp++-stage branch merged the latest fcmp++ code and I'm very happy about that.

<r​brunner7:monero.social> "I don't think it needs consensus in this meeting" That would be asking for a bit much ....

<j​berman:monero.social> (faster prove time for large inputs will be in stressnet)

<r​brunner7:monero.social> Nice

<j​berman:monero.social> Sorry that explanation above ended up longer winded than I had hoped

<r​brunner7:monero.social> Ok, thanks for all the background info, probably helps understanding a lot

<r​brunner7:monero.social> Well, I think that's very often the most valuable info: general approach, intent, and overall structure of code

<j​berman:monero.social> *thumbs up*

<r​brunner7:monero.social> Ok, if that's it about that PR, do we have anything else to discuss today?

<j​effro256:monero.social> Wouldn't you still need to have a "copy" of the hashchain in this proposed appraoch to revert to in case there as an error in block processing?

<j​effro256:monero.social> Nah it was good info

<j​berman:monero.social> This is a good point to raise and is one of the complications to deal with as result of duplicated logic

<j​berman:monero.social> I initially missed it in my first pass, but this should solve that issue: seraphis-migration/monero #81#issuecomment-3263948108

<j​berman:monero.social> Resetting that locally scoped cur_chain to the state of m_blockchain in the event of an error should make sure the next loop continues from where it's supposed to

<r​brunner7:monero.social> "error in block processing" also includes interruptions of the communication with the daemon?

<j​effro256:monero.social> That's the proposed method I was referring to. So you would be syncing hashchain updates before making the next request to the daemon. Doesn't that mean you need to copy & mutate the hashchain before the processing in case the processing goes wrong?

<j​effro256:monero.social> Like I think you have to have it either way, no?

<j​berman:monero.social> yep, and also CLI errors upon encountering a receive and user needs to input their password

<j​effro256:monero.social> Unless you add new logic to rollback changes made before processing

<j​berman:monero.social> So first in the control thread, it rolls back the locally scoped cur_chain as needed for next loop

<j​berman:monero.social> Then next loop makes the request for that next set of blocks in parallel

<j​berman:monero.social> Then it continues processing prev parsed blocks on top of m_blockchain (which has not been updated yet for the rollback)

<j​berman:monero.social> process_parsed_blocks will then *also* handle rolling back m_blockchain internally (using duplicated reorg handling logic)

<j​effro256:monero.social> Is this the current state of #81?

<j​effro256:monero.social> The last 4 messages?

<j​berman:monero.social> if process_parsed_blocks errors, then the control thread should hit that catch statement, and cur_chain should reset back to m_blockchain

<j​berman:monero.social> And in effect, this will have been a wasted request for the next set of blocks, but no local state change should happen to the wallet

<j​berman:monero.social> yes

<r​brunner7:monero.social> Alright, discussion between people in the know about PR #81 can continue of course, but allow me to close the meeting proper. Thanks everybody for attending, read you again next week!

<j​berman:monero.social> Wait I had one other thing to raise!

<j​effro256:monero.social> Thanks everybody

<j​effro256:monero.social> jk

<j​berman:monero.social> actually a couple sorry

<r​brunner7:monero.social> Sorry, didn't suppose that ...

<j​berman:monero.social> Just going to mention in here, I'm pretty ok with ArticMine 's latest fee proposal to scale fee by tx byte size alone without considering verification time, considering tx size as a whole actually *does* increase a significant amount as n inputs increases beyond just membership proofs size

<j​berman:monero.social> pinging kayabanerve

<j​berman:monero.social> this kayabanerve

<j​berman:monero.social> the figures: seraphis-migration/monero #44#issuecomment-3150754862

<k​ayabanerve:matrix.org> I still support a fee by verification time too, to the point it's never worse re: fees to make multiple small TXs vs one large TX.

<k​ayabanerve:matrix.org> But I'll concede if I'm in the minority

<r​brunner7:monero.social> The discussion in the last MRL meeting gave me the impression that consensus about the new fees is still somewhere in the distance ...

<r​brunner7:monero.social> I myself don't have an opinion there, don't know enough about fees unfortunately

<j​berman:monero.social> the latest MRL meeting had a lot of discussion on block weight penalty, which is another discussion. I do think we are closer to a finalized weight calculation

<r​brunner7:monero.social> We may see in 2 days, hopefully

<j​berman:monero.social> I think the latter point still holds using artic's current proposed figures, but will double check it

<k​ayabanerve:matrix.org> It shouldn't, as the inputs are linear but the proof sizes favor large TXs.

<j​berman:monero.social> oh wait, I misread you as flipped

<k​ayabanerve:matrix.org> Unless it's superlinear to proof size still.

<k​ayabanerve:matrix.org> Then ack, I stated my criteria and can be happy however it's satisfied 👍

<j​berman:monero.social> you're saying you would prefer that users not have the incentive to make one large tx, versus multiple txs?

<k​ayabanerve:matrix.org> Coreecr

<k​ayabanerve:matrix.org> *Correct

<j​berman:monero.social> But verifying larger should always be faster than smaller too? So you're making 2 points there, A) that you would still support verification time, and separately B) that you would not prefer an incentive to make larger txs

<j​berman:monero.social> > But verifying larger should always be faster than smaller too

<j​berman:monero.social> In that 128-in fcmp verify should be slightly faster than 16x 8-in

<j​berman:monero.social> It seems like these 2 points are in conflict

<k​ayabanerve:matrix.org> Not if we weight verification time superlinearly as to achieve an incentive for many lower-verification-time TXs.

<k​ayabanerve:matrix.org> I won't complain if they're at least equal but would argue a 128-input TX should pay twice as much as 19 8-input TXs :C

<k​ayabanerve:matrix.org> I would also argue 128-input TXs shouldn't exist and point out you wanted booting a Monero node to require sampling ~200k points which would take around a minute D:

<k​ayabanerve:matrix.org> But that's not the community's vibe at this time :p So instead I made it so loading those 200k points take ~40x less time.

<j​berman:monero.social> seraphis-migration/monero #44#issuecomment-3187685572

<j​berman:monero.social> 16x 8-in is almost twice as large as 1x 128-in

<j​berman:monero.social> bandwidth is likely to be the main limiting factor to scaling long term

<k​ayabanerve:matrix.org> And should be <half the cost IMO.

<k​ayabanerve:matrix.org> The input limit will be the limiting factor to scaling if we ever introduce transaction uniformity, if every TX must be 100 KB due to the inputs.

<k​ayabanerve:matrix.org> From transaction uniformity to IVC (which could compress all inputs to just one), the input limit is an incredibly important factor.

<k​ayabanerve:matrix.org> I think it should be lower. It'll be 128. I think people shouldn't be encouraged to make TXs with 128 inputs.

<k​ayabanerve:matrix.org> *IVC would presumably also have a fixed input limit.

<j​berman:monero.social> The reality is the current hf and what's planned for it in particular and I think goal should be to optimize for that, not a future hf which may have distinct tech/research to achieve the goal of uniform txs

<k​ayabanerve:matrix.org> Except people keep bitching about the idea of reducing inputs and this would gently nudge them to build better wallet software themselves, reducing friction in any inevitable future where Monero development doesn't die off.

<k​ayabanerve:matrix.org> At some point, Monero will either be technically stagnant or will adopt a fixed-input limit (even if solely internally considered and not exposed to the user) for privacy or scaling, I'd bet on it.

<j​effro256:monero.social> I wonder if, without proper broadcasting of this rule, people making very large TXs will ever do the math and be deterred from making large TXs, or they'll just go "hmmm... that's a really big fee, oh well..."

<k​ayabanerve:matrix.org> I mean, I'd bet like $100, I'm not a millionaire.

<k​ayabanerve:matrix.org> jeffro256: p2pool is probably a large enough force that someone will build an optimized p2pool aggregator

<k​ayabanerve:matrix.org> It doesn't require users be smarter. It requires a few devs be smarter with the systems users use.

<j​effro256:monero.social> But aggregators do technically break the current UX of using Monero, so the users at least have to be aware of why it is being done

<k​ayabanerve:matrix.org> Any part of Monero 'breaks' the UX of Monero, fight me /s

<k​ayabanerve:matrix.org> Aggregators already exist. This does aggravate them, but in a softer way than what I originally pushed for, in hopes they'll adjust and it'll flow downstream so it never comes to the shove I wanted.

<k​ayabanerve:matrix.org> The fact users may not have noticed aggregators exist, or sweeping may be necessary, is solely due to the conditions which may or may not always exist. This narrows the chance they live in blissful ignorance, sure, but it sets a transition path and encourages uniformity now.

<j​berman:monero.social> I still think your line of reasoning is giving too much weight to how you imagine a future hf would be designed to accommodate tx uniformity

<k​ayabanerve:matrix.org> I understand I'm disagreed with and I'm not a dictator here

<k​ayabanerve:matrix.org> Feel free to move on :p

<j​berman:monero.social> I think a future hf that achieves tx uniformity in some solidly designed /agreed upon way is fine to then "shove" toward tx uniformity at that point

<j​berman:monero.social> rather than push for it in next hf

<j​berman:monero.social> the goal is still rough consensus on weights for this current hf. sounds like you still wouldn't agree with weights scaled by byte size alone and would NACK what's currently on the table

<j​berman:monero.social> which is an important NACK to take into account

<r​brunner7:monero.social> For what it's worth, my gut feeling is also to get FCMP++ out of the door in any reasonable, acceptable form, see how that works, and then adjust and improve starting from that

<r​brunner7:monero.social> It's already a giant step, a bit problematic to make it bigger still

<j​berman:monero.social> I think byte size has strong justification and has major benefit of being simple

<j​berman:monero.social> I'm content with closing discussion on this issue for now :)

<j​berman:monero.social> at least within this meeting that's still quasi ongoing

<j​berman:monero.social> I did want to raise one other thing, but I'll hold off on it

<r​brunner7:monero.social> So next week for that "one more thing"?

<j​berman:monero.social> or I'll just say it now and whoever wants can discuss

<j​berman:monero.social> 1 sec

<r​brunner7:monero.social> It will at least make it into the meeting log :)

<j​berman:monero.social> I raised a point in 81 that is unrelated to 81, that I will probably make a separate issue out of: seraphis-migration/monero #81#issuecomment-3197430438

<j​berman:monero.social> jeffro256 and ofrnxmr and I then had a discussion on it that is interesting

<j​berman:monero.social> my TL;DR: if a pool's tx FCMP++ uses a tree root that will no longer be part of the main chain e.g. daemon user pops blocks 40 blocks or there is a >= 10 block reorg, then said tx will no longer validate (unless of course, the tree becomes the main chain tree again), and as such I propose we kick the tx from the pool as a step in a better direction from current (the tx remaining i<clipped messag

<j​berman:monero.social> n the pool and preventing a user from spending the inputs that tx included)

<r​brunner7:monero.social> TL;DR if the tx is absolutely hopeless kick it?

<j​berman:monero.social> yes

<j​berman:monero.social> jeffro argues that since the tx could theoretically become valid again, it may make sense to keep the tx around in the pool

<r​brunner7:monero.social> Sounds good, with a watertight predicate "Is hopeless" of course

<j​berman:monero.social> (I think is his argument, may be putting words in his mouth sorry jeffro)

<r​brunner7:monero.social> Yeah, saw something of that argument. I think that may have merit, but not with a full week of expiry. I would guess if after 1 day there is no switch back of the chain it's over.

<b​oog900:monero.social> We could keep the tx but drop it if another tx comes in that double spends it but is valid?

<b​oog900:monero.social> or we could just keep both in the pool

<b​oog900:monero.social> why drop either :)

<r​brunner7:monero.social> Uh, tx id must be key in tons of related structures

<j​berman:monero.social> I think the latter is arguably the optimal long term solution, but more involved of a change that I don't exactly think is worth it for v1

<r​brunner7:monero.social> Keeping both might not be trivial

<b​oog900:monero.social> tbf this is an issue that needs fixing now not with FCMP

<b​oog900:monero.social> and it doesn't need to be a HF/consensus change

<j​berman:monero.social> that's fair too

<r​brunner7:monero.social> Alright, *now* I declare the meeting proper to be over. Thanks again, until next week.

<j​effro256:monero.social> I would only be okay with keeping both if the signable transaction hash is the same. But perhaps as a v1, I guess it makes sense to drop the transaction under a >=10 block reorg and allow a new transactions, rather than keeping that input completely locked for 3 days (current mempool expiry period)

<s​needlewoods_xmr:matrix.org> thanks everyone, until next time

<j​berman:monero.social> the problem with keeping the tx around (without other modifications) is that a deep reorg by a malicious actor could be even more problematic for honest users than it needs to be (honest users get their txs stuck and can't spend the inputs)

<b​oog900:monero.social> I think dropping txs in the pool after a reorg is fine as only txs in the pool before the reorg would be affected if the main chain comes back

<j​berman:monero.social> another contextual element: blocks that are reorged also have their txs placed back in the pool

<j​berman:monero.social> and I think it's a better step for v1 to drop those 2 if they can't be included in what is at the time the main chain

<j​berman:monero.social> those too* lol

<j​berman:monero.social> better step relative to the current of just keeping txs in the pool until expiry without changing logic to accommodate newly valid txs

<j​berman:monero.social> ______

<j​berman:monero.social> ^sorry ignore above