.merge+ 7793

Added

There's a new health report on:

The compilation time is steadily rising:

I've got 2 open PRs, which address this 6932 & 6934.

I'd have many more, but won't spam you with new ones, until the old ones get merged.

mj-xmr[m]: do you have an example of a PR that increases compile time according to your report?

or is it just measure variance?

selsta: it's hard to find the culprits here. I make a measurement after every set of merges, not after every merge. Besides it usually goes together. The variance is lower than the general trend.

I'll try to find an example PR in the history that could make things worse though, but I still think that the devs themselves shouldn't be worrying about it. It just needs a cleanup from time to time.

selsta: 7858 might be one good example, where a large header (`misc_language.h`) is added into an inlined code.

7616: another example, where `daemon/command_line_args.h` is added to `p2p.h`

It could be avoided, if the t_p2p constructor was inside a .cpp file.

I see no more possible culprits in the recent merged sets.

less_as_pod isn't used, and it'd save functional. I'm not sure what makes this file heavy though, might not be functional.

I opened a PR to move to C++14: monero-project/monero #7971

(This doesn't mean that we should rewrite current code, just that we can (optionally) use new features in new code. The only exception would be if it allows us to drop boost in some places)

Broad programming question: Would you say it is the case that it is more important to take into account code optimization for an interpreted language than a compiled language, especially one like C++ that (I've heard) has made great improvements in optimizing code in the compilation step?

hard to answer that

I'd say if you're using an interpreted language, then performance is clearly not a priority, so optimization is irrelevant

hyc: How can you say that? There are many cases where you need optimization in an interpreted language. R is one of them. There are a huge number of statistical techniques available in R and nowhere else. I suppose you could reimplement them in a compiled language, but that would be a huge time sink and subject to error.

meh. the point of interpreted languages is they're "easier" to write in. it means you value development time over runtime.

Rucknium: typically R / Python and such are great for researching various topics available there, but in the end, once the team agrees to 3 particular methods, it's off to reimplementing them in C++ and enjoying its speed.

good performance is not a trait of any interpreted languages, nor can it be, so it should just be off the table

There are embedded boards, which would just not handle Python and R's memory consumption.

This is silly. You can "accidentally" write interpreted code that is an order of magnitude slower than an alternative, just-as-simple syntax in the same langauge. So optimization matters.

that sounds more like knowing wtf you're doing matters

There are many cases when you might write something in R and it never makes sense to re-implement it in another language

expeecting an optimizer to make up for poor pgramming decisions is futile

Yeah. That was my question. Is it more important to know wtf you are doing optimization-wise in an interpreted language?

C++ is great at optimizing low level specifics of how your program is executed, so from that perspective yes -- manual optimization of small details is less important.

A question to question: what is your range of target architectures?

I would be careful not to take that line of thinking too far though. If the underlying algorithm you are implementing is poorly optimized, C++ probably can't fix it for you.

BusyBoredom: Agreed

Raspberry for example.

What is a target archtecture?

like linux? windows?

Or in other words: are small devices important for you?

Not raspberry. Usually a decent laptop or consumer desktop. Sometimes a server, of course. But most statistics done by scientists is done on a consumer desktop or laptop.

Then I wouldn't bother about C++.

Your data size matters.

I am surprised that you all are surprised that this is a relevant question.

I'm not.

Just shows the gulf between computer science and statistics once again.

I run calculations on very large financial data. I do some research in Python first, and then I'm done, I look for C++ implementation, since I will sooner or later want to loop the calculations in a Monte Carlo simulator or at least a grid optimizer to see the system's potential to adapt.

So in that case every cycle counts.

Memory consumption also matters, since there are some Raspberries in my computation network, that take over some burden.

indeed. my direct experience is with Bittorrent, the original software was pure python. both a CPU & memory pig, but it was released to the world and thrown into production as-is.

I kinda agree with hyc. In most workloads, most of the time is spent in a smallish number of "building block" routines. Implementing those in optimized asm/FORTRAN/C gets you most of the speedup you'll need if it's meant to be run repeatedly.

Yes, in some cases every cycle does count, and you may need to re-implement. That is not the case with most statistical analysis.

Rucknium[m]: Smells of overtiffing.

A lot of statistical analysis, especially in the natural and social sciences, is a one-and-done affair

\i worked on libtorrent/rtorrent in C++. reduced footprint by at least 1000x.

*overfitting

interpreted languages are good for prototyping. putting the results into production is IMO always a mistake

mj-xmr: Well, there _is_ a lot of bad science published in decent journals out there -- I'll give you that. You need to be careful, of course

Rucknium: does "the majority" of statistical analysis include Walk-Forward-Optimization (for time series) and/or Monte Carlo simulations?

moneromooo: I agree. The heavy lifting in R is in C, C++, or Fortan. The key to writing fast R code is to make sure it is the C/C++/Fortan code that is doing the heavy lifting, not the interpreted R code itself. That requires a few best practices, though.

Rucknium[m]: That's what I mean :(

Rucknium[m]: Agree to that. LibEigen is a great example.

FWIW, interpreted languages can be fucking annoying for prototyping time, due to the fact they tend to complain about simple typos only once you interpret that path, wasting so much time that'd have been caught much earlier with C.

^ That too.

so, if you've arranged for all the heavy lifting to be done in a compiled language, what's left to optimize in the interpreted language?

The data structure part.

And possibly caching.

ie, do not kneecap the C function :)

moneromooo: That's true. What is missing is the fact that the people who have domain-specific knowledge in science and/or statistics may not have the time to learn C++ extremely well. It's a big opportunity cost.

So working in teams, across disciplines, can be useful, but it can also be inefficient to try to communicate across disciplines. And it is most costly in terms of labor

Chapters 2 and 3 here explian how you can get orders-of-magnitude slowdown in R if you "speak with a C accent"

But, if you write R well, it can be extremely fast. There are benchmarks showing R can be faster in some basic tasks than Python by 7 times

Seems my messages from Matrix did not come through. Maybe it thought I was spamming:

^ Pandas is pythons' main statistics framework.

data.table is R's fast big data workhorse

Ah ok. Now it is through! I didn't actually paste it. it just arrived late 😬

Anyway, if you are interested in an deep (and optimistic) analysis of R as a programming language, see this interview

dunno why you're trying to justify use of your favorite tool to us

we don't get python guys in here trying to justify their use of numpy

I think you are saying interpreted languages don't have much value. They do. R is one of them

It is going to help overhaul the mixin selection algorithm

Unless you think the mixin selection algorithm has no problems. In which case I should just publish my attack

i am sick of being talked down to as a statistician

You want MRL to hold on to researchers? Maybe respect them

I am not going to publish my attack publicly, of course. But you see my point?

no. you miss my point.

interpreted languages have value for prototyping. it ends there.

when you go into production, you use a compiled language

what you're doing is research. decidedly non-production. so sure, your tool is the best choice for you.

nobody is going to install R to run their monero wallet.

Ok. I think we are in agreement. I suppose statistical analysis could fall under a definition of prototyping.

I run a Townforge wallet with R.

Just RPC it

And that's basically Monero

RPC the node, I mean. townforged

guessing that won't run well on my smartphone

R can run on a smartphone. There are iOS and Android apps for it. Just need a C and Fortan, and maybe C++ compiler for it.

And the point of TownforgeR is to have a browser interface. So yes you can interface with your wallet and the blockchain with a simple web browser, of course available on your smartphone

hyc: You seem to lack imagination or knowledge about this area.

you're welcome to believe what you like

It is no problem to lack knowledge. What can be a problem is to recognize the limits of your knowledge

I believe it because I've seen it. And because I've built it, even.

yes. you're arguing for a configuration that is of no use to anyone but yourself

#monero-dev-offtopic :D

ok that doesn't exist

yeah we've strayed far enough

selsta: We are going off topic, yes. First of all, Syksy built the foundation for TownforgeR, so it's useful t him at least. And other players have tested it and found it useful too. You are so arrogant it is breathtaking

I mean, there is plenty of arrogance in my own discipline, no doubt, so I am not a stranger to it.

you're saying you believe it's useful for end users to have R, fortran, C and C++ compilers installed on their smartphones so they can run this wallet?

when they could just run an app built on top of the existing libwallet, which btw was ported to android by me.

They don't need to do that. They just need to open a webpage on their browsers

And use the wallet that way

then it's not a wallet running on their phone. so it really isn't even relevant to the discussion of what they'd run on their phones.

I would LOVE someone to build that. I don't see any labor resources to do that though. moneromooo is occupied with the core protocol and the desktop-based 3D GUI. I wish that we could build the thing you mentioned, but it's about labor resources and not having them

The cryptocurrency space is starved for devs. We have to work with what we have, in terms of labor resources.

I am not necessarily a great programmer, but TownforgeR could be useful. I any case, working on TownforgeR started a chain reaction would led to me take a serious professional look at the mixin selection algorithm and now develop an important overhaul to it.

* In any case...

I don't think I will publish my attack, but I am not sure if anything will be done by Monero devs to address the statistical vulnerability of the mixin selection algorithm, so I may find myself in a position that I would have to get some movement on the issue.

do you have proposed mitigations?

what makes you think nothing will be done?

moneromooo : You were not clear in your Hackerone response if you believe it is safe to publish the attack. Do you think it is safe to publish, just so I know what my options are?

I sense a lack of willingness to understand the statistical issues involved.

I mean, plenty of researchers just publish attacks in the open, in journals. I care about protecting Monero user privacy, though, at an ethical level. So I don't want to do that, even if it would advance my "career".

For example, Moser et al. (2018) could have gone through Monero's VRP instead of publishing their attacks in the open.

AFAIK you wanted to work together with jberman[m] to find issues and mitigations?

isthmus, for one, thinks my attack is very serious. The attack can be rendered inert by an overhaul to the mixin selection algorithm, however. It will take a lot of time to develop that overhaul, however

selsta: Yes I am finalizing my CCS right now.

Maybe I am just used to a different style of argumentation, but I feel like my perspective has often been dismissed.

I could "calibrate" my sense of argumentation tactics to adjust to the CompSci way -- if that makes sense -- however.

I don't know what kind of response you're expecting. if Isthmus agrees with you that it's serious, that's good enough for me

In the past mitigations to statistical attacks are suggested from the research / MRL side and then implemented on the dev side.

hyc: Ok great. I am heartened to hear that.

but likewise, if I don't have the statistics background to understand it or devise a solution, then it's pointless for me to think about it further

hyc: Totally fair.

selsta: Yes, that is what I intend to do, assuming I now count as being from the MRL side.

Your kind of work would be MRL.

sarang often did python code in MRL and mooo ported it to C++.

yes. or java.

The research will take months, though, so I need a CCS to fund it, as I cannot work for free for months at a time -- bills need to be paid and there is an opportunity cost since I could be working on something else. The BCH community already funded me 18 BCH to do some statistical work on their CoinJoin protocol, CashFusion, so I could shift back there and do more work.

so yes, you can present a solution in your language of choice, and we (mooo probably) will implement it in C++

selsta: Right. In this stage of the overhaul, only 3 or so lines in C++ will need to be changed: The gamma distribution changed to something else, probably, and the two (or more) associated parameters values for shape, scale, etc. Hopefully there will be an even more extensive and optimized overhaul that will need more involved dev C++ work, though.

Jesus... you're just seeing slights where there are none...

hyc is saying that one one measurement axis (performance), interpreted loses to compiled. It's *one* measurement axis.

And he says that on another (prototyping), interpreted wins. That's all.

The H1 report is just "the fake out selection algorithm is squint, so the effective ring size is less than if it was perfect". We know that it's not perfect. It's engineering, nothing's perfect, models are just approximations, progress happens.

moneromooo: I understand that better now. I am not very used to having a close working relationship to dedicated programmers and computer scientists, so I suppose I am misinterpreting the content and tone of the discussion. I now recognize the need to "calibrate" as I said above

I'm sure someone can in turn do better than what you will, it'll just be diminishing returns.

moneromooo: So, under the VRP, can I publish the attack or not? I don't intend to, but it would be good to clarify.

Does the attack have an immediate impact of CIA (Confidentiality, Integrity, Availability) triad for Monero?

Whether you should, overall, depends on whether you'd be helping assholes more or helpers more.

<Rucknium[m]> "moneromooo: That's true. What is..." <- I have pretty good knowledge in C++ as well as lots of experience in R. You can call c++ routines from R with the RCPP package

SerHack: I do believe it has an effect on Confidentiality

ie, would it help assholes break people's privacy more, or researchers improve ring selection more. I think it'd help assholes more, but that is speculation.

Then there's a solid no from me.

moneromooo: I think to publish now would help assholes more, but is there a formal determination from the VRP process?

* c++ routines directly from R

I'm not really the one to talk to about process, I tend to reject rigid rules.

isthmus has said to me privately, after reviewing my submission, that it is "a fundamental breakthrough in analyzing Monero-style ledgers" that is "potentially catastrophic" and "might be honest-to-god RIP for Monero at current ring size and composition".

Dut in the context of H1, in general, either you do everying privately and get a bounty once it's all fixed, or you do it publicly and you don't.

The bounty is an incentive for you to keep it under wraps while things can get patched.

Well, having seen it, isthmus is full or it with that sentence ^_^

(sorry, nice person who does intereresting stuff, but still)

moneromooo: Thank you for clarifying that part. I don't care all that much about a bounty, but a bounty would be nice and it helps give form to the process: Stay in these lines, and a bounty is possible

What you worked out is just a good way to couch things in mathemetical terms so it can be reasoned upon, but that's really it.

Unless maybe I'm missing part of what you reported, but I don't really see it.

A good analysis, but nothing fundamentally different from what we knew before. Am I wrong ?

I am not sure isthmus is correct, either, but I have less than 2 months of work and thinking about this, while he has years.

moneromooo: Maybe you should get a second opinion

I'll have one when it's all done and disclosed ^_^

I mean, it's a *model*. It's known to be an approximation to a known, nice, mathematical function.

The attack is not fully developed since it's more important and urgent to allocate my time on fixing the problem than making the attack stronger.

If it were me, I'd just fucking fix it and move to the next thing.

moneromooo: So my submission is still under consideration? Your response on HackerOne made me think that it is now dormant.

IIRC I said "looks ok, please do it", no ?

Right. That's what I want to do. Fix it. But in order to do that it can be useful to explore how important it is to fix. That's the attack part.

It is important. It's the weakest part of monero.

So, the bottleneck, in engineering terms. Any improvement there is good.

moneromooo: Yes, but you didn't indicate that you would get any second opinion. I did not get that sense at all from your response. If you do get a second opinion, that is definitely what I want

I did not intend to get a second opinion. You said you got one already, and intended to get one from two other people with statistics/math background. That seems more than enough to me.

I see. Ok , yes I am working on it

And tbh what you reported seems clear enough to me.

The difficulty is not in assessing whether anything can be improved, it is in devising the improvement.

Maybe I *am* missing something in what you reported though.

Me: "Maybe you should get a second opinion"

You: "I'll have one when it's all done and disclosed ^_^"

^ This made me think that you were independently seeking a second opinion. I'm sorry if I misinterpreted.

Ah, no, I meant once you've PRed the changes you intend and they're public. Then we'll get people to assess your work and comment :)

moneromooo: I think it could be useful for you to talk to isthmus directly. It is not of paramount importance to really settle the question of the severity of the attack, however. Fixing the vulnerability is important. I am working very hard on that.

OK

moneromooo: But I don't think we should disclose the attack publicly, even at that time, since past txs on the blockchain would still be vulnerable

It's fine not to disclose it publicly in this case.

I am happy to share any part of the work, both the attack and the process for developing the solution, to any trusted Monero community member. But sharing it publicly is dicey, yes.

sounds good

So far moneromooo, isthmus, and jberman have seen the attack. luigi1111 has access to the report, but I do not think he/she/they has read it. I have used end-to-end encrypted means to share it, using PGP encryption to share with moneromooo & luigi and Matrix encrypted chat to share with isthmus and jberman.

To help gauge my understanding, do you think your work here is more significant, less significant, or about as significant as the Moser et al paper in term of improving the fake out selection ?

I would say less, though not by too much.

Or maybe between "less but still significant" and "less, though not by too much".

Yes it is less significant. Keep in mind that Moser et al. (2018) was able to de-anonymize 60-80% of all Monero transactions, in terms of finding the real spend.

If you see it as significantly more than that, thne maybe I have missed your point.

OK.

moneromooo: I would say talk to Isthmus about trying to get a better idea. He has spend more time thinking about these types of things, overall.

"de-anonymize ring sigs" fwiw. Still did not do anything for the one time sig layer. But yes.

I asked. But isthmus is online every few days.

I would say that my independent judgement is "less, but still significant", but isthmus, possibly because he has more knowledge, would probably classify it as "less, though not by too much.

The key thing is I have not fully examined the additional claims that he makes about my attack. I have not done so since it takes time away from working on the remedy.

I mean, it's not him just saying "I think this is really bad". It's him saying, "These are XYZ reasons why the attack is actually worse than you, Rucknium, present."

I touch on isthmus's ideas in section 3.4 on page 8 & 10 of my submission.

Well, then maybe I'll agree it's worse when it's shown how it is then. I think I've said all I want to say here, if not more. I'll stop, I've got better things to do than going in circles on this.

moneromooo: Sounds good. I will go back to writing my CCS proposal. I feel under a lot of stress recently, so I am sorry if I come off a bit heated. I have never done anything like this before, and it feels quite high stakes, with no "buffer" if I make an error.

Usually economists are protected from their own mistakes by virtue of no policymakers listening to them, or plausible deniability that the policymakers "didn't get it right." Not so here.

I have the sensation that other people's privacy, and therefore possibly their freedom and safety, are directly in my hand.

in that respect you have the right attitude. but you will not be the only person examining the problem and solution

Right. I am working to put together a review panel.

Hi people.I have a question about encrypted_seed function.Is it safe to store your money (huge amounts) on a wallet that is encrypted? i backed up the encrypted seed and base seed has all the money.i have read some stuff on github that its just adding some random numbers and stuff so im a little confused about my seed security

I dont have access to hardware wallet and i just used encrypt_seed on my main wallet to backup it.is it considered safe ?

If it's just adding random numbers, it sounds unsafe.

I mean, someone could guess them, right ?

Just try them all.

yeah but the stuff i read was from 2018 so i dont know if its uodated or not

i mean not like literally numbers.they wrote that just to explain it.

Anyway, seriously, it's not safe or unsafe by itself. It's a transformation of your seed.

If your seed is safe, then this transformation is safe. If your seed is unsafe but your password is safe *AND* implausible to brute force, then it is also safe.

if its considered safe to use i mean its the best option i could think of.storing your seed on plain sight.you can only restore with your password.and you can send even some xmrs to it to make it look like a used wallet.works like a decoy

Otherwise, it is not safe.

If your original seed is public, then it is safe if your encrypted seed password is really hard to guess. Like, 128 bits of randomness.

ie, "Hi people.I have a question about encrypted_seed function." will not do.

But enough bits used to derive N words for a known dictionary will be.

i think im just going to buy a ledger.does it works well with monero ? any extra steps or is it just plug and play?

It's meant to work. #monero will know details.