00:47:13 <jberman[m]> <vtnerd> "yeah thats a good question, it..." <- I could try my hand at an informal audit. I’m decently familiar with the stack and expected security properties. I also agree the UX of running your own light wallet server is vastly superior to needing to sync the wallet client-side every time in theory, with minimal downside
01:04:09 <gingeropolous> well, the difference is that with a full rpc wallet online, if an attacker can get admin access, essentially, they'd be able to spend monero, whereas with the LWS, they would not (because it only has hot viewkey)
01:04:13 <gingeropolous> i think
02:09:53 <selsta> .merges
02:09:53 -xmr-pr- 7786 7792 7795 7800 7801 7802 7805 7809 7810 7811 7812 7816 7817 7818 7839
02:33:35 <selsta> w0w merges
02:34:34 <luigi1111w> amazing
02:35:13 <selsta> luigi1111w: do you have time to look at https://github.com/monero-project/monero/pull/7821#issuecomment-896778611 ?
02:35:29 <selsta> jberman[m]: or do you plan more work on it=
02:35:34 <selsta> it*
02:35:58 <luigi1111w> I felt like 50 was longer than needed
02:40:00 * jberman[m] uploaded an image: (45KiB) < https://libera.ems.host/_matrix/media/r0/download/matrix.org/rogqzRbDrZjpPLHOyrQUrWMF/window%20of%203.png >
02:40:40 <jberman[m]> This is what 3 looks like. Can see it would have drastically over-selected from the first few blocks, and under-selected after
02:43:27 <jberman[m]> <selsta> "jberman: or do you plan more..." <- I noticed it's failing a CI test in the release branch and haven't had time til now to look into that
02:43:53 <luigi1111w> why do charts appear to start at different places?
02:44:26 <luigi1111w> I guess it's just "off the charts"
02:44:29 <selsta> jberman[m]: ci fail is unrelated
02:45:29 <jberman[m]> Right yam, I cut off the y-axis on that chart to make the impact versus current and observed clearer
02:45:42 <luigi1111w> can you do it with, say, 20?
02:46:37 <luigi1111w> also remind me what observed data is?
02:53:23 <jberman[m]> observed data is the age of every rct output, both decoys and reals, in every ring over the block range (2,300,000 - 2,412,241)
02:53:40 <jberman[m]> <luigi1111w> "can you do it with, say, 20?" <- I'll do 10, 20, 40, 50, 70, 100 to show
02:54:21 <luigi1111w> thanks
02:58:31 <gingeropolous> but the gamma distribution was from bitcoin, right?
02:59:29 <jberman[m]> no the one in the paper they ended up fitting the gamma to is from Monero when a lot of outputs could be linked on-chain
03:02:43 <jberman[m]> <gingeropolous> "well, the difference is that..." <- also here I meant versus the alternative of having a wallet sync client-side when you open it up (and one that keeps keys encrypted at rest), like monero-wallet-cli or cake wallet AFAIK
03:03:56 <gingeropolous> aight, so in https://user-images.githubusercontent.com/26468430/129026276-5fd0749d-1e4f-4d27-8a53-a02069392e06.png , observed is whats currently happening on the chain, "current decoy selection algo" is what *should* be happening on the chain, and the "current proposed fix" is the proposed fix, and your testing the proposed fix for closeness of fit against ... what.
03:04:22 <gingeropolous> ?
03:04:58 <jberman[m]> closeness of fit to the observed
03:06:00 <gingeropolous> but shouldn't it be being compared to gamma? observed isn't "real" activity. its the outputs being selected by a gamma distribution thinger
03:08:38 <Rucknium[m]> Observed are mixins and real spends, composed in a 10:1 ratio.
03:12:24 <gingeropolous> sure. but the goal is to have the mixin selection match likely real outputs (gamma), not mixin selection created by a broken output picker (observed) ... or im not getting this right.
03:13:26 <gingeropolous> also, I ponder with gamma (and i really thought it was pulled from bitcoin), did the analysis that extracted gamma factor in pool payouts?
03:13:41 <Rucknium[m]> 🤔 I'm just now thinking that there may be a way to test the hypothesis that there are wallets that are not using the gamma distribution. We have the distribution for each transaction, in a sense. The statistical test would be tricky to construct, though.
03:14:24 <jberman[m]> because the decoy selection algorithm factors in block density, it didn't seem to me that the gamma should be the exact correct thing to test fitness against. For example, if you have tons of dense blocks at different intervals, the expected distribution wouldn't necessarily be the gamma
03:14:24 <jberman[m]> Rather, it seems to me you would want to see how well observed matches up to your expected distribution. The expected distribution is how you apply the gamma distribution when factoring in block density, which in this case is what the lines "current decoy selection algo" and "current proposed fix" represent. does that make sense?
03:14:26 <gingeropolous> because monero, historically, didn't have that much activity. and a lot of those 0 mixin transactions were pools
03:14:38 <Rucknium[m]> gingeropolous: I think they ruled that out. Let me double check the paper...
03:14:50 <jberman[m]> they did say the ruled it out in the paper
03:15:05 <gingeropolous> u mean those data were excluded?
03:15:11 <jberman[m]> Rucknium[m]: I agree, this is one of the things I wanted to do after the fix too
03:15:55 <Rucknium[m]> I think. OOne sec...
03:17:56 <gingeropolous> yeah. it just seems weird to want something better (the fix) to be closer to the borked thing (observed) than the supposed real thing (gamma).
03:18:14 <gingeropolous> i mean, if gamma with block densities is complicated... well.... 
03:18:28 <gingeropolous> i dunno
03:18:34 <jberman[m]> the borked thing is the orange line, not the blue line
03:18:48 <jberman[m]> well blue line is partially borked
03:19:00 <jberman[m]> but orange line is what is definitely borked
03:20:19 <Rucknium[m]> "After accounting for the estimated impact of mining pools, which opt-out of privacy by publishing their transac- tions on webpages, there remain a substantial number of potentially privacy-sensitive transactions, more than a thousand per day." Moser et al. (2018).
03:20:40 <jberman[m]> what we don't know today is exactly what the gamma distribution should even look like. We don't know what to match it to, because we don't know exactly how we should account for the outputs earlier than the unlock time that the gamma factors in
03:21:05 <Rucknium[m]> In fact It's not 100% clear from my quick skim just now whether those mining pool transactions were excluded from the "countermeasures" section. Probably was. 
03:22:05 <jberman[m]> Rucknium[m]: I agree, it's not clear if they were excluded from the gamma distribution upon further reading. Need to take a deeper look at the code I think
03:22:25 <Rucknium[m]> gingeropolous: Spoiler alert: using the gamma distribution is the thing that's borked. In my professional opinion as an applied statistician, the mixin selection algorithm needs an overhaul.
03:23:15 <Rucknium[m]> But figuring out an optimal overhaul will take time. jberman's patch is important to do now as a stopgap measure.
03:23:19 <jberman[m]> Ya, generally speaking, I think the goal of this fix to land on the sanest, do-no-harm approach that patches the known issue of earliest spends being ignored, while changing the least amount of stuff
03:24:08 <jberman[m]> speaking more precisely: while having the lowest impact on the rest of how the decoy selection algorithm performs
03:34:40 <gingeropolous> gotcha. yeah Rucknium[m], agreed that mixin selection needs overhaul. i was hoping we'd get to 100+ ringsizes sooner than later so this could be addressed just by upping the n
03:35:40 <gingeropolous> but thats suboptimal as well. 
03:38:26 <gingeropolous> i guess it'd be nice to see gamma on those graphs. that woulda helped. or are you saying orange is gamma, and gamma is generally borked
03:44:21 <jberman[m]> no the gamma could be placed on the graph, I think it would be interesting too
03:45:32 <jberman[m]> it might get a little confusing though in how to square what's there with the gamma
03:57:07 <jberman[m]> <gingeropolous> "yeah. it just seems weird to..." <- Explaining the intuition of why it makes sense to get closer to the blue line: https://user-images.githubusercontent.com/26468430/129026276-5fd0749d-1e4f-4d27-8a53-a02069392e06.png
03:57:07 <jberman[m]> Looking at just the blue and orange line, notice the Star Trek logo shape that appears above the orange line. The fact that we are observing *many* more outputs than what the current decoy selection algo would produce in that age range, suggests that those are real outputs that the current algorithm is missing.
03:57:07 <jberman[m]> That's why getting closer to observed is sensible, so that the decoy selection algorithm would start picking up those outputs that it currently appears to be missing
03:59:59 <jberman[m]> And at the very least, moving away from the blue line, as my original proposed fix using a window of 1 block did, is the wrong direction to head: https://user-images.githubusercontent.com/26468430/129025873-43395110-ba55-47f2-87b9-a741dda83732.png
04:03:54 <luigi1111w> essentially the fix will change the observed, at which point you have a new (hopefully closer) target? haha
04:04:15 <luigi1111w> I'm curious what an overhaul will look like as well
04:04:33 <luigi1111w> I will be curious*
04:05:36 <gingeropolous> also, why that particular block range? do these distributions hold up across different epochs?
04:05:52 <gingeropolous> i mean, with the same tx algos etc
04:06:54 <gingeropolous> i mean, block 2300000 was just feb of 2021
04:07:33 <jberman[m]> luigi1111w: the idea of this fix is to get something out the door that does a *good enough* job to cover the earliest spent issue, *and* doesn't cause any potential harm. I can't imagine that getting marginally closer to the blue line causes harm (and may help), but moving further away from it could hurt
04:07:38 <gingeropolous> and, along these lines, are pool payouts excluded from the dataset... though... ... hrmm
04:08:57 <luigi1111w> jberman[m] yes I agree
04:14:08 <jberman[m]> <gingeropolous> "also, why that particular..." <- The starting point seemed like a wide enough range to capture enough useful data (albeit clearly not too deep of thought), and end point was just when I started doing analysis to keep it consistent. It takes a long time to produce the distributions
04:14:08 <jberman[m]> I also tested a more recent distribution (from when average_output_time shifted from 2 to 1, and the solution fit within this goal I had too "the idea of this fix is to get something out the door that does a good enough job to cover the earliest spent issue, and doesn't cause any potential harm. I can't imagine that getting marginally closer to the blue line causes harm (and may help), but moving further away from it could hurt"
04:14:08 <jberman[m]> I could plot different epochs too. I have some solid caching strategies set up in my local env to speed up analysis. Neptune gave a good basis from which to test epochs here: https://github.com/neptuneresearch/monero-ringmember-age-distribution
04:15:36 <jberman[m]> That's what I'll do next: test the different windows (1, 3, 10, 20, 40, 50, 70, 100) over different epochs, and I'll place the gamma on the charts too
04:16:22 <jberman[m]> my availability isn't super high though atm so this may end up going over to next week
04:16:34 <Rucknium[m]> Why do you need to cache? Low RAM?
04:19:57 <jberman[m]> no I have 32gb of RAM. I cache output heights so I can preload them all into memory into a data structure that has O(1) lookup time, and also cumulative rct distribution so I don't need to get that huge rct_offsets array every time
04:21:55 <jberman[m]> output height cache is for when iterating over every ring, to get the age of each output quickly
04:23:07 <Rucknium[m]> Ok. Keep in mind that random sampling can keep the resource use down while still being statistically valid. Although I suppose you might not know the line between statistically valid and invalid, especially in a context where we are dealing with potentially dependent time series data 🤔
04:25:54 <jberman[m]> just thought of a way to speed it up a whole bunch
04:30:16 <Rucknium[m]> Speaking of speeding up (or slowing down) I think I'll delay my line-by-line look at the proposed solution until you do the different windows simulation. I have a few other self-imposed deadlines coming up in short order that I want to check off the list.
04:31:29 <Rucknium[m]> I may try to replicate what you're doing in R
06:55:41 <mj-xmr[m]> <vtnerd> "wallet2 become a do everything..." <- The interface is just an abstraction of the *current* state for having a lighter representation across the app. The new interface is not fixed, just like wallet2 isn't, and doesn't prohibit further segmentation in future.
07:15:43 <mj-xmr[m]> at the small cost of having to copy the changes from wallet2 to wallet2_base of course, but you can say it about any interface.
07:33:11 <mj-xmr[m]> vtnerd: I looked at wallet2_api and it looks great. My wallet2_base isn't supposed to replace the wallet2_api at all. Only abstract it for users of wallet2. If wallet2_api can be used instead of wallet2_base, then wallet2_api should be used. No doubt. I'm afraid that we're not there yet though, are we?
07:33:39 <mj-xmr[m]> *Only abstract wallet2 for its users.
07:33:46 <mj-xmr[m]> direct users that it.
08:21:45 <plowsof[m]> <gingeropolous> "well, the difference is that..." <- In my case it would just be a restricted rpc (forget rpc, just a view only wallet i guess, running locally) - i will set rsync up on my node and desktop pc - then i run a 'start GUI' script which just does the rsync line -> opens the wallet (no 200 iq needed)
09:33:35 <mj-xmr[m]> The definitive proof why the project is healthy:
09:34:00 <mj-xmr[m]> git log --pretty=oneline --abbrev-commit
09:34:04 <mj-xmr[m]> a32da4bc3 Merge pull request #7787
09:34:04 <mj-xmr[m]> 0878207e8 Merge the coolest pull request #7777
09:34:04 <mj-xmr[m]> 894e5b279 Merge pull request #7767
10:15:36 <UkoeHB> Regarding 7833 (and 7854)... Should I put my `n_choose_k()` function in `epee`'s `misc_language.h`, or should I create a new directory `src/math`, then move `rolling_median_t<T>` and `median<T>()` into that directory? The last commit pushed to 7833 (now closed) should my general idea. I feel a bit uncertain about how to proceed here.
10:15:52 <UkoeHB> shows* /should
12:30:44 <plowsof[m]> Wallet running on your low powered node device, with passwordless ssh login setup then on your 'wallet device' e.g. pc: 
12:30:44 <plowsof[m]> 1. rsync -a root@eeebox:/var/log/monero/wallets/general-fund/ /home/human/Monero/wallets/general-fund/
12:30:44 <plowsof[m]> 2. Open your fully sync'd wallet 
12:30:44 <plowsof[m]> For android devices, Termux gives you rsync but ive not actually tried it with a mobile wallet (but i cant see why it wouldnt work) 
15:15:54 <endogenic> "I looked at wallet2_api and it looks great." wut
15:16:20 <endogenic> wallet2_api needs to not exist
15:16:40 <endogenic> we've talked about that before as well lol 
15:20:31 <UkoeHB> it seems like quite a large wrapper
15:23:51 <endogenic> not only that, but it exists *because* wallet2's interface was not willing by the wallet2_api creators to be modified
15:23:56 <endogenic> in other words, it has the wrong reason for existing
15:24:00 <endogenic> so, the result is wrong
15:24:35 <endogenic> if a programmer avoids the problem then they'll probably end up infecting other places or making more issues elsewhere 
15:25:11 <endogenic> if the rationale were that we needed a wallet2 instance manager, that's justifiable, but we wouldn't call it wallet2 api and it wouldn't do all this crazy redundant stuff
15:25:23 <endogenic> and it wouldn't be the case that, e.g. simplewallet uses wallet2 directly but the gui for some reason doesn't
15:25:27 <endogenic> i could go on :P
15:26:21 <endogenic> not to be too paranoid or anything but i really wonder why sometimes we can't make up our minds as a collective project about this
15:27:17 <endogenic> i think it's probably gonna end up as an achilles heel 
15:28:17 <endogenic> being open to merging all kinds of changes is one thing but we should have some clear criteria of what definitely shouldn't be merged .. and i'm afraid wallet2_api (as an example) set a bad precedent that will continue to balloon
16:02:35 <wfaressuissia[m]> "https://github.com/monero-project/monero/pull/872/commits" whole this PR ?
16:07:12 <wfaressuissia[m]> "https://github.com/monero-project/monero/pull/728#issuecomment-197348058" probably this PR and this justification
16:17:53 <endogenic> hm perhaps we should move src/wallet2/api to monero-gui
16:18:38 <endogenic> and while we're at it maybe we can send epee with it too.. jk 
16:21:37 <UkoeHB> idk much about this stuff... why doesn't the gui use rpc calls?
16:24:49 <endogenic> spinning up an rpc server instead of embedding code?
16:25:22 <UkoeHB> practically speaking, what is the overhead?
16:26:03 <endogenic> would simplewallet use an rpc server as well?
16:26:15 <endogenic> (as i think that through)
16:27:23 <endogenic> and, would the rpc server end up the only specced consumer of wallet2?
16:28:56 <endogenic> the lws server would be the other required consumer of the code kept in wallet2 like scanning
16:29:42 <UkoeHB> hmm TIL you should not name a header `math.h` if you want things to compile happily
16:35:43 <endogenic> and I doubt third party gui wallets would want to have to spin up an RPC server
16:35:59 <endogenic> but the concept of a singular interface, especially a network transport capable one, is very appealing
16:56:02 <UkoeHB> I think if wallet2 itself was decomposed into smaller pieces, it would be a lot easier to decide the best interface.
16:57:30 <endogenic> i think wallet2 already has a good interface
16:57:38 <endogenic> or at least we could say that it does for now
16:58:22 <endogenic> that is, with the exception, perhaps, of where certain type (e.g. struct) declarations are stored - since many of those will need to move once we do such a decomposition 
16:58:51 <endogenic> the decomposition is a no-brainer tho
16:59:04 <endogenic> and who doesn't want wallet2 to be shorter and easier to reason about?
17:01:12 <endogenic> it may be the case that a lot of the things consumers use wallet2 or wallet2/api to - to your point - wont even be accessed via wallet2 or wallet2/api anymore after factoring and namespacing
17:01:35 <endogenic> wallet2/api for*
17:09:36 <sech1> Did first successful p2pool test today. Sneak peak: wallets - https://imgur.com/a/G8UwXgY nodes - https://pastebin.com/D05aRS48 and https://pastebin.com/2wH4frcS
17:18:25 <wowario> interesting
18:15:18 <binaryFate> sech1: sexy!
18:18:44 <sech1> So now I actually have 2 nodes mining on 2 different wallets and talking to each other. Still lots of things to do... Like checking that no one is cheating :D Also block pruning
18:27:13 <jtgrassie> excellent!
18:30:55 <moneromooo> Seconded.
18:32:08 <rottenwheel> what am i looking at exactly?
18:32:15 * rottenwheel wears the dumb hat
18:40:26 <binaryFate> 2 nodes sharing block rewards
18:41:43 <sech1> yes, this is how it looks in explorer: https://testnet.xmrchain.net/tx/a6a54ef7361e5bf399022dc4cace037c927953b958070f51fd417eab5808e269
18:52:10 <Rucknium[m]> I am going to post something in r/btc
19:06:20 <LyzaL> r/btc or r/bitcoin?
19:07:30 <Rucknium[m]> r/btc
19:09:18 <Rucknium[m]>  * I am sorry, I switched channels from Policy and didn't realize. There is a new bill introduced in Congress affecting anonymizing services and well as anonymous coins. It is unlikely to pass, though.
19:09:37 <LyzaL> ohhh I see the connection now ok
19:12:01 <Rucknium[m]> So sorry. If you want to get a link to the fresh post, go to #monero-policy:monero.social 
20:10:52 <plowsof[m]> I now have a desktop shortcut in ubuntu that runs rsync then opens the monero gui synched up. (1 click) i guess this is only useful if its working on a mobile device with a wallet.. i will try cakewallet + termux/rsync with some simple 'launcher' app (if such a thing exists) in an android emulator (my phone isnt compatible :( )  
20:39:38 <crypto_grampy[m]> <plowsof[m]> "I now have a desktop shortcut in" <- I have usage of termux scheduled tasks as well as termux widget usage in my repo: https://github.com/CryptoGrampy/android-termux-monero-node
20:46:40 <plowsof[m]> Thank you very much crypto_grampy (exactly what i wanted) 
22:19:16 <Guest598> What is going on with submodule miniupnp? Doesn't seem to update correctly with git.
22:21:07 <Guest598> git submodule update --init --force doesn't fix it either
22:23:23 <wfaressuissia[m]> "https://github.com/monero-project/monero/pull/7650/files"
22:26:56 <Guest598> Saving this for later, Thank you.