<vtnerd> "yeah thats a good question, it..." <- I could try my hand at an informal audit. I’m decently familiar with the stack and expected security properties. I also agree the UX of running your own light wallet server is vastly superior to needing to sync the wallet client-side every time in theory, with minimal downside

well, the difference is that with a full rpc wallet online, if an attacker can get admin access, essentially, they'd be able to spend monero, whereas with the LWS, they would not (because it only has hot viewkey)

i think

.merges

w0w merges

amazing

luigi1111w: do you have time to look at monero-project/monero #7821#issuecomment-896778611 ?

jberman[m]: or do you plan more work on it=

it*

I felt like 50 was longer than needed

This is what 3 looks like. Can see it would have drastically over-selected from the first few blocks, and under-selected after

<selsta> "jberman: or do you plan more..." <- I noticed it's failing a CI test in the release branch and haven't had time til now to look into that

why do charts appear to start at different places?

I guess it's just "off the charts"

jberman[m]: ci fail is unrelated

Right yam, I cut off the y-axis on that chart to make the impact versus current and observed clearer

can you do it with, say, 20?

also remind me what observed data is?

observed data is the age of every rct output, both decoys and reals, in every ring over the block range (2,300,000 - 2,412,241)

<luigi1111w> "can you do it with, say, 20?" <- I'll do 10, 20, 40, 50, 70, 100 to show

thanks

but the gamma distribution was from bitcoin, right?

no the one in the paper they ended up fitting the gamma to is from Monero when a lot of outputs could be linked on-chain

<gingeropolous> "well, the difference is that..." <- also here I meant versus the alternative of having a wallet sync client-side when you open it up (and one that keeps keys encrypted at rest), like monero-wallet-cli or cake wallet AFAIK

aight, so in user-images.githubusercontent.com/2…49d-1e4f-4d27-8a53-a02069392e06.png , observed is whats currently happening on the chain, "current decoy selection algo" is what *should* be happening on the chain, and the "current proposed fix" is the proposed fix, and your testing the proposed fix for closeness of fit against ... what.

?

closeness of fit to the observed

but shouldn't it be being compared to gamma? observed isn't "real" activity. its the outputs being selected by a gamma distribution thinger

Observed are mixins and real spends, composed in a 10:1 ratio.

sure. but the goal is to have the mixin selection match likely real outputs (gamma), not mixin selection created by a broken output picker (observed) ... or im not getting this right.

also, I ponder with gamma (and i really thought it was pulled from bitcoin), did the analysis that extracted gamma factor in pool payouts?

🤔 I'm just now thinking that there may be a way to test the hypothesis that there are wallets that are not using the gamma distribution. We have the distribution for each transaction, in a sense. The statistical test would be tricky to construct, though.

because the decoy selection algorithm factors in block density, it didn't seem to me that the gamma should be the exact correct thing to test fitness against. For example, if you have tons of dense blocks at different intervals, the expected distribution wouldn't necessarily be the gamma

Rather, it seems to me you would want to see how well observed matches up to your expected distribution. The expected distribution is how you apply the gamma distribution when factoring in block density, which in this case is what the lines "current decoy selection algo" and "current proposed fix" represent. does that make sense?

because monero, historically, didn't have that much activity. and a lot of those 0 mixin transactions were pools

gingeropolous: I think they ruled that out. Let me double check the paper...

they did say the ruled it out in the paper

u mean those data were excluded?

Rucknium[m]: I agree, this is one of the things I wanted to do after the fix too

I think. OOne sec...

yeah. it just seems weird to want something better (the fix) to be closer to the borked thing (observed) than the supposed real thing (gamma).

i mean, if gamma with block densities is complicated... well....

i dunno

the borked thing is the orange line, not the blue line

well blue line is partially borked

but orange line is what is definitely borked

"After accounting for the estimated impact of mining pools, which opt-out of privacy by publishing their transac- tions on webpages, there remain a substantial number of potentially privacy-sensitive transactions, more than a thousand per day." Moser et al. (2018).

what we don't know today is exactly what the gamma distribution should even look like. We don't know what to match it to, because we don't know exactly how we should account for the outputs earlier than the unlock time that the gamma factors in

In fact It's not 100% clear from my quick skim just now whether those mining pool transactions were excluded from the "countermeasures" section. Probably was.

Rucknium[m]: I agree, it's not clear if they were excluded from the gamma distribution upon further reading. Need to take a deeper look at the code I think

gingeropolous: Spoiler alert: using the gamma distribution is the thing that's borked. In my professional opinion as an applied statistician, the mixin selection algorithm needs an overhaul.

But figuring out an optimal overhaul will take time. jberman's patch is important to do now as a stopgap measure.

Ya, generally speaking, I think the goal of this fix to land on the sanest, do-no-harm approach that patches the known issue of earliest spends being ignored, while changing the least amount of stuff

speaking more precisely: while having the lowest impact on the rest of how the decoy selection algorithm performs

gotcha. yeah Rucknium[m], agreed that mixin selection needs overhaul. i was hoping we'd get to 100+ ringsizes sooner than later so this could be addressed just by upping the n

but thats suboptimal as well.

i guess it'd be nice to see gamma on those graphs. that woulda helped. or are you saying orange is gamma, and gamma is generally borked

no the gamma could be placed on the graph, I think it would be interesting too

it might get a little confusing though in how to square what's there with the gamma

<gingeropolous> "yeah. it just seems weird to..." <- Explaining the intuition of why it makes sense to get closer to the blue line: user-images.githubusercontent.com/2…49d-1e4f-4d27-8a53-a02069392e06.png

Looking at just the blue and orange line, notice the Star Trek logo shape that appears above the orange line. The fact that we are observing *many* more outputs than what the current decoy selection algo would produce in that age range, suggests that those are real outputs that the current algorithm is missing.

That's why getting closer to observed is sensible, so that the decoy selection algorithm would start picking up those outputs that it currently appears to be missing

And at the very least, moving away from the blue line, as my original proposed fix using a window of 1 block did, is the wrong direction to head: user-images.githubusercontent.com/2…110-ba55-47f2-87b9-a741dda83732.png

essentially the fix will change the observed, at which point you have a new (hopefully closer) target? haha

I'm curious what an overhaul will look like as well

I will be curious*

also, why that particular block range? do these distributions hold up across different epochs?

i mean, with the same tx algos etc

i mean, block 2300000 was just feb of 2021

luigi1111w: the idea of this fix is to get something out the door that does a *good enough* job to cover the earliest spent issue, *and* doesn't cause any potential harm. I can't imagine that getting marginally closer to the blue line causes harm (and may help), but moving further away from it could hurt

and, along these lines, are pool payouts excluded from the dataset... though... ... hrmm

jberman[m] yes I agree

<gingeropolous> "also, why that particular..." <- The starting point seemed like a wide enough range to capture enough useful data (albeit clearly not too deep of thought), and end point was just when I started doing analysis to keep it consistent. It takes a long time to produce the distributions

I also tested a more recent distribution (from when average_output_time shifted from 2 to 1, and the solution fit within this goal I had too "the idea of this fix is to get something out the door that does a good enough job to cover the earliest spent issue, and doesn't cause any potential harm. I can't imagine that getting marginally closer to the blue line causes harm (and may help), but moving further away from it could hurt"

I could plot different epochs too. I have some solid caching strategies set up in my local env to speed up analysis. Neptune gave a good basis from which to test epochs here: github.com/neptuneresearch/monero-ringmember-age-distribution

That's what I'll do next: test the different windows (1, 3, 10, 20, 40, 50, 70, 100) over different epochs, and I'll place the gamma on the charts too

my availability isn't super high though atm so this may end up going over to next week

Why do you need to cache? Low RAM?

no I have 32gb of RAM. I cache output heights so I can preload them all into memory into a data structure that has O(1) lookup time, and also cumulative rct distribution so I don't need to get that huge rct_offsets array every time

output height cache is for when iterating over every ring, to get the age of each output quickly

Ok. Keep in mind that random sampling can keep the resource use down while still being statistically valid. Although I suppose you might not know the line between statistically valid and invalid, especially in a context where we are dealing with potentially dependent time series data 🤔

just thought of a way to speed it up a whole bunch

Speaking of speeding up (or slowing down) I think I'll delay my line-by-line look at the proposed solution until you do the different windows simulation. I have a few other self-imposed deadlines coming up in short order that I want to check off the list.

I may try to replicate what you're doing in R

<vtnerd> "wallet2 become a do everything..." <- The interface is just an abstraction of the *current* state for having a lighter representation across the app. The new interface is not fixed, just like wallet2 isn't, and doesn't prohibit further segmentation in future.

at the small cost of having to copy the changes from wallet2 to wallet2_base of course, but you can say it about any interface.

vtnerd: I looked at wallet2_api and it looks great. My wallet2_base isn't supposed to replace the wallet2_api at all. Only abstract it for users of wallet2. If wallet2_api can be used instead of wallet2_base, then wallet2_api should be used. No doubt. I'm afraid that we're not there yet though, are we?

*Only abstract wallet2 for its users.

direct users that it.

<gingeropolous> "well, the difference is that..." <- In my case it would just be a restricted rpc (forget rpc, just a view only wallet i guess, running locally) - i will set rsync up on my node and desktop pc - then i run a 'start GUI' script which just does the rsync line -> opens the wallet (no 200 iq needed)

The definitive proof why the project is healthy:

git log --pretty=oneline --abbrev-commit

a32da4bc3 Merge pull request #7787

0878207e8 Merge the coolest pull request #7777

894e5b279 Merge pull request #7767

Regarding 7833 (and 7854)... Should I put my `n_choose_k()` function in `epee`'s `misc_language.h`, or should I create a new directory `src/math`, then move `rolling_median_t<T>` and `median<T>()` into that directory? The last commit pushed to 7833 (now closed) should my general idea. I feel a bit uncertain about how to proceed here.

shows* /should

Wallet running on your low powered node device, with passwordless ssh login setup then on your 'wallet device' e.g. pc:

1. rsync -a root@eeebox:/var/log/monero/wallets/general-fund/ /home/human/Monero/wallets/general-fund/

2. Open your fully sync'd wallet

For android devices, Termux gives you rsync but ive not actually tried it with a mobile wallet (but i cant see why it wouldnt work)

"I looked at wallet2_api and it looks great." wut

wallet2_api needs to not exist

we've talked about that before as well lol

it seems like quite a large wrapper

not only that, but it exists *because* wallet2's interface was not willing by the wallet2_api creators to be modified

in other words, it has the wrong reason for existing

so, the result is wrong

if a programmer avoids the problem then they'll probably end up infecting other places or making more issues elsewhere

if the rationale were that we needed a wallet2 instance manager, that's justifiable, but we wouldn't call it wallet2 api and it wouldn't do all this crazy redundant stuff

and it wouldn't be the case that, e.g. simplewallet uses wallet2 directly but the gui for some reason doesn't

i could go on :P

not to be too paranoid or anything but i really wonder why sometimes we can't make up our minds as a collective project about this

i think it's probably gonna end up as an achilles heel

being open to merging all kinds of changes is one thing but we should have some clear criteria of what definitely shouldn't be merged .. and i'm afraid wallet2_api (as an example) set a bad precedent that will continue to balloon

"monero-project/monero #872/commits" whole this PR ?

"monero-project/monero #728#issuecomment-197348058" probably this PR and this justification

hm perhaps we should move src/wallet2/api to monero-gui

and while we're at it maybe we can send epee with it too.. jk

idk much about this stuff... why doesn't the gui use rpc calls?

spinning up an rpc server instead of embedding code?

practically speaking, what is the overhead?

would simplewallet use an rpc server as well?

(as i think that through)

and, would the rpc server end up the only specced consumer of wallet2?

the lws server would be the other required consumer of the code kept in wallet2 like scanning

hmm TIL you should not name a header `math.h` if you want things to compile happily

and I doubt third party gui wallets would want to have to spin up an RPC server

but the concept of a singular interface, especially a network transport capable one, is very appealing

I think if wallet2 itself was decomposed into smaller pieces, it would be a lot easier to decide the best interface.

i think wallet2 already has a good interface

or at least we could say that it does for now

that is, with the exception, perhaps, of where certain type (e.g. struct) declarations are stored - since many of those will need to move once we do such a decomposition

the decomposition is a no-brainer tho

and who doesn't want wallet2 to be shorter and easier to reason about?

it may be the case that a lot of the things consumers use wallet2 or wallet2/api to - to your point - wont even be accessed via wallet2 or wallet2/api anymore after factoring and namespacing

wallet2/api for*

Did first successful p2pool test today. Sneak peak: wallets - imgur.com/a/G8UwXgY nodes - pastebin.com/D05aRS48 and pastebin.com/2wH4frcS

interesting

sech1: sexy!

So now I actually have 2 nodes mining on 2 different wallets and talking to each other. Still lots of things to do... Like checking that no one is cheating :D Also block pruning

excellent!

Seconded.

what am i looking at exactly?

2 nodes sharing block rewards

yes, this is how it looks in explorer: testnet.xmrchain.net/tx/a6a54ef7361…37c927953b958070f51fd417eab5808e269

I am going to post something in r/btc

r/btc or r/bitcoin?

r/btc

* I am sorry, I switched channels from Policy and didn't realize. There is a new bill introduced in Congress affecting anonymizing services and well as anonymous coins. It is unlikely to pass, though.

ohhh I see the connection now ok

So sorry. If you want to get a link to the fresh post, go to #monero-policy:monero.social

I now have a desktop shortcut in ubuntu that runs rsync then opens the monero gui synched up. (1 click) i guess this is only useful if its working on a mobile device with a wallet.. i will try cakewallet + termux/rsync with some simple 'launcher' app (if such a thing exists) in an android emulator (my phone isnt compatible :( )

<plowsof[m]> "I now have a desktop shortcut in" <- I have usage of termux scheduled tasks as well as termux widget usage in my repo: github.com/CryptoGrampy/android-termux-monero-node

Thank you very much crypto_grampy (exactly what i wanted)

What is going on with submodule miniupnp? Doesn't seem to update correctly with git.

git submodule update --init --force doesn't fix it either

"monero-project/monero #7650/files"

Saving this for later, Thank you.