14:37:38 There was unusual fee behavior yesterday and today. There are four standard fee tiers: 20, 80, 320, and 4000 nanoneros per byte. On March 11, the share of "non-spam" txs paying 320 nanoneros/byte was 4.5%. It had been steady for a few days until then, too. Then on March 12 and today the share of those 320 fee txs increased to about 35%. 14:37:54 In the block tabulation, it looks like this set of 320 fee txs started at block 3103476 (2024-03-12 14:04:08 UTC) and ended after block 3104042 (2024-03-13 08:42:57 UTC). These 320 fee txs are almost all 1in/2out. 14:38:11 It could be a big centralized service turning on and off its fee priority. Or the suspected spammer could be changing its behavior. 14:38:29 The auto fee fix to the GUI/CLI wallet only increases fee from 20 to 80 nanoneros per byte when the mempool is congested. 14:44:14 Fee tabulation by block: https://paste.debian.net/hidden/fb7ec136/ 14:47:42 Ah, wait that tabulation was just for today. Here is yesterday and today: https://paste.debian.net/hidden/8d45e207/ 14:48:37 MRL meeting in this channel in about two hours 14:51:34 By the way, the numbers for the lowest fee tier (80) don't add match the blockchain data since I remove a portion of the low-fee 1in/2out txs as "spam". 14:52:31 I mean lowest fee tier (20) 17:00:44 Meeting time! https://github.com/monero-project/meta/issues/978 17:00:57 1) Greetings 17:01:22 Hello 17:01:33 hi 17:02:08 Hi 17:02:40 <0xfffc:matrix.org> Hi everyone 17:03:13 hi 17:03:46 2) Updates. What is everyone working on? 17:04:33 me: Analyzing the blockchain and mempool data from the last week and a half of the suspected spam incident. 17:05:29 I was working on finding a segfault lws (still not found), unit testing the lws rest api, and a little bit on figuring out how to enable multi-machine scanning in lws 17:06:13 I may punt on the last one, as its looking pretty complicated 17:06:32 until someone explictly needs it (its probably unlikely) 17:06:39 Like some sort of load balancing? 17:07:56 <0xfffc:matrix.org> Me: I updated RW lock (9181) last week, the lock is final at this point. Worked here and there on few GitHub issues. But main thing I am working on is performance benchmarks suite for XMR. I finished a prototype yesterday. ( https://github.com/0xFFFC0000/monero-perf/tree/main/private-testnet ). The prototype does have 3 wallet limitation. Now I am implementing it in C++ ( https:// 17:07:57 <0xfffc:matrix.org> github.com/0xFFFC0000/benchmark-project ). Numbers from these performance benchmarks can help us a lot in evaluating different PRs (including the 9181). 17:08:20 rbrunner: yes, send account scanning over socket to other machines for scanning 17:09:32 3) Discussion. We have the possible spam incident: https://bitinfocharts.com/comparison/monero-transactions.html#3m 17:11:01 I've been estimating the mean/median/percentile effective ring size of this is actually the prophesied black marble flooding. 17:12:10 The mean has been about 5.5 for the last few days. If the "spam" volume stays the same, it will decrease slowly now as more of the probability mass of the decoy selection algorithm moves into the spam time interval. 17:13:43 isthmus might try to re-run the scripts that we wrote to analyze the 2021 spam incident. The database workflow that we were using us broken, so it has to be repaired or a new workflow written. The 2021 analysis was mostly to provide evidence for or against the txs being produced by a single entity. 17:14:48 Is there a critical threshold for effective ring size that should not be crossed, assuming this is a black marble flood? What do we think? 17:16:50 Hard to say. Rings are only 1 of 3 privacy mechanisms, amounts stay invisible, stealth addresses hold, even if the effective rings size continues to go lower. 17:17:43 We lived for quite some time with a ringsize of 5, I think. Because that just was the ringsize back then, period. 17:17:50 I think we're already in the red zone. but going below 3 (historical ring size) would be critically bad 17:18:24 If the spam stays at its current level indefinitely (about 400kB blocks), the median effective ring size will decrease to 3 eventually. That may be low enough to allow chain reaction analysis. And we shouldn't forget about the extremes. 3 or 5.5 may be average, but some rings will pick up more spam tx outputs by chance. 17:21:41 Well, I think we are mostly aware that if we are not careful any "cure" we come up might turn out to be worse than the "disease" 17:21:51 Some of the countermeasure that I have seen: 1) Raise fees, 2) Raise ring size, 3) Change dynamic block size algorithm, 4) See if miners will voluntarily limit block size, 5) Launch decentralized counterspam to reduce the share of outputs owned by the spammer. 1-3 would require a hard fork AFAIK. 17:23:05 Blocksize growth seems pretty limited so far. You could speak of a "spam wave", but not, or not yet anyway, a "flood" 17:23:15 Do we want to try to figure out if a single node (or a small number) is broadcasting the txs? 17:24:22 I don't think that would be conclusive, the spammer could use many nodes 17:24:35 The suspected spammer is letting the block size algorithm's 100 block median fall down. Not sure if it's deliberate or accidental. That's limiting the block growth rate. 17:25:09 Yeah, so far it has been a pretty nice spammer ... 17:25:42 Do you think it would be conclusive if one node were found? You could confirm the hypothesis, but lack of evidence for a single node origin would not confirm that it is not spam. 17:26:11 that will work against them once eniugh nodes update to the fee patch, but I don't know if we see that yet in fee usage 17:26:58 Work against them? The auto fee adjustment is wallet-level, not node-level AFAIK. Users have to update. 17:27:35 Yes, the wallet decides the fee factor to apply 17:27:47 yes, I would conseder that conclusive. so it may be worthy to investigate 17:29:08 How do we know that the 20k average txs/day that we had previously were not poisoned by 10k or 15k txs from a spammer? 17:29:17 How would we go on to defeat Dandelion++? Some sort of coordinated listening? 17:29:53 dangerousfreedom: We do not know that. A slow rise in spam has been theorized before. 17:30:26 you're right, that's the wallet. if.tney keep stepping back, the block size can't grow and *if* they keep their tx's on the cheap fee level, more honest tx's will be included 17:30:27 Basically D++ is a statistical obfuscation technique. If the signal is huge, like with spam, then it can be heard above the noise. 17:31:27 The spammer can keep pushing up block size even if all users update to the auto fee fix. 17:32:09 But that would probably need some daemon instrumentation, to get some statistics, I guess. Or some third program that speaks the protocol. 17:32:42 Sharma, P. K., Gosain, D., & Diaz, C. 2022. "On the anonymity of peer-to-peer network anonymity schemes used by cryptocurrencies." https://moneroresearch.info/index.php?action=resource_RESOURCEVIEW_CORE&id=130 17:33:54 ^ This paper analyzed D++'s resistance to a specific form of statistical analysis. I thought this paper was unrealistic since they needed huge tx volumes to determine the node origin of txs. But now we have a huge tx volume, so it could work. I tried the paper's python code a while ago and it seemed to work ok. 17:35:07 Yes the main data you need is tx arrival times at many nodes in the network. Not quick or easy to set up. You would have to get volunteer nodes or nodes that try to place themselves strategically in the network. 17:36:15 they can, but at at a slower rate compared to a scenario where they increase their fees too, no? there's a chance the increased budget requirement could keep them back 17:36:43 Basically a lot of the analysis here just shows how good Monero is at privacy :) 17:36:48 One problem I have with D++ right mow is that node connections are unencrypted which means that ISPs have an advantage in being able to tell where transactions originated from if they decided to collude 17:37:04 Did you see my comments before the meeting? Recently there was a big increase in high fee txs. Then that stopped. 17:37:11 Dummy stems don't mean much against DPI 17:38:05 Yes, IRC the D++ paper says that it doesn't protect against ISP surveillance. 17:38:32 IIRC* 17:39:23 If we go up with the ringsize and stay in "reasonable" territory, say 20, will that help? And if yes how much? Just needs a bit more spam to overcome? 17:39:57 it's possible that the spammer is using Tor/I2P 17:40:00 rbrunner: I can run some scenarios 17:40:26 I didn't, will read the backlog. that's not good 17:40:46 That would be splendid, rucknium 17:41:01 "Not good" as in "making matters worse"? 17:41:28 #FreeOfrnxmr 17:41:37 Basically, when spam txs increase, effective ring size decays exponentially. So the largest drop happens when the spam volume is "small". 17:41:47 I have a few plots I can post soon 17:41:50 rbrunner: I think so, yes 17:42:44 If the spammer is willing to make high fee (320 nanoneros/byte) txs, then they have a large budget. 17:43:54 Maybe a dumb idea, but we could make the ringsize user-selectable again, with a minimum of 16. So people basically could choose their level of privacy ... or so, maybe that fails somewhere 17:44:26 I will do rbrunner's suggested analysis of simulating ring size increases. Other suggestions for analysis I could do? 17:45:37 re: changing minimum fee or dynamic block size algo, I would appreaciate input from ArticMine: 17:45:47 Selectable ring sizes leads to fingerprinting 17:46:02 Yup. It would be a question of the "lesser evil". 17:46:34 And it seems there are no really easy options on the table, sadly 17:47:24 Yeah. If we allow different ring sizes we should probably change the tx weight mechanism since a 2x increase in CLSAG verification time wouldn't lead to a 2x increase in weight 17:48:10 Custom ringsize is dumb 17:48:45 Isn't that partially at fault for wanacry payments getting traced or was that something else? 17:48:47 doing this would be a tradeoff. I'm more against than for (horrible optics and UX), but if for, you could prescribe a few ring sizes, not the way it worked in the old days where any amount could go. 17:50:11 It's silly, if the attacker owns 90% of decoys they still do that whether you have 15 or 500 decoys but with the later it's way more fingerprintable 17:50:21 Ok :) 17:50:32 All of this is solved with fcmps 17:50:37 2 million decoys let's go 17:51:07 rucknium: Maybe try to find out what the effects of "counterspam" would be? How much does the community have to spam to keep "effective ringsize up"? 17:51:32 I can do that 17:51:55 Counterspam is of course also bad, but well ... 17:52:07 sounds impossible, but could a FCMP upgrade be feasibly done separately from Seraphis? 17:53:05 Actually, why not try to counterspam and see if that causes a reaction of the "real" spammer 17:53:06 because that would be a trump card 17:53:17 Even if possible, that would probably take a year to go into effect. If we don't rush with inacceptable risks 17:53:22 Technically yes, but then you would basically have to remake seraphis lol 17:53:32 Biggest issue with counter spam is not knowing when to stop if there are multiple counter spammers 17:53:52 We have to consider that the *whole* reason for the exercise may be to drive us to do something unwise. 17:54:35 At this point it's most likely about chainalysis raising money 17:55:04 IMHO, probably the best short term countermeasure, if it is really a black marble flood, is to see if miners would set a limit on the block size that they will not go above. Only a majority of hashpower would be enough to reduce the 100 block median 17:55:09 That's also my "pet theory", as recently explained on Reddit 17:55:48 https://matrix.monero.social/_matrix/media/v1/download/monero.social/bGgVqJzYAtzSQXUdTKOcWunA 17:55:51 The worst part is, when chainalysis inevitably releases their "tracing monero" report the fud will be immense even though monero still protects amounts and receiver 17:56:26 So they can only get down to a 50/50? 17:56:45 ^ That long term projection is if all decoys are selected from the spam time interval. Since some decoys are being selected from before the spam start point, we are not there yet. 17:56:54 Not immediately clear to me why limiting blocksize is beneficial? 17:57:16 Not as a flood protection, but keeping the effective ringsize up 17:57:26 that's somithing to keep in mind, yes 17:57:42 I stopped it at effective ring size 2. I can extend the visualization. 17:58:13 I don't think it's needed 17:58:18 But that's just average. Some txs would have effective ring size 1 since they would select all spammer-controlled outputs by chance. 17:58:38 How many TX per day would that even be at effective ringsize 2m 17:58:50 How many TX per day would that even be at effective ringsize 2? 17:59:26 rbrunner: Since limiting block size limits the amount of spam that can be packed in. And keeps the number of real txs constant. I guess that last part assumes that users update to the auto fee fix. 17:59:55 monerobull: Looks like about 200,000 tx/day would be needed for that. 18:00:37 Yes, and that the spammer does not go higher with fees themselves? 18:00:59 👍 18:01:13 Yes, also assuming that. 18:02:32 All a bit unfortunate 18:02:50 a company like Chainanalysis would try to keep things under the radar. they wouldn't start spamming 7x the tx amount from one day to another, creating a 20mb txpool, harming UX and making everyone aware that something is going on 18:03:24 Well, who would then? 18:04:09 It's not that all their employees are competent for sure. Who knows who got the job :) 18:04:23 Selsta: they might have not expected the auto fee to be broken lol 18:04:31 yeah - someone incompetent 18:05:00 hey, yes, Chainanalysis alright, but they outsorced the job 18:05:56 if a sensible ring size increase (=doesn't disadvantage many potential users) could get us a significant advantage, that's one intervention that won't have catastrophic downsides and isn't a regression 18:06:43 How about a temporary rushed implementation of triptych/s 18:07:17 monermooo must still have a branch around somewhere with that 18:07:19 A ring size increase combined with parallel verification can work here 18:08:10 Also we can target the estimated tx size for full membership proofs 18:08:24 there's already a high degree of parallelism on incoming verify 18:09:24 That's an interesting argument: To "adjust" already today to the FCMP tx size 18:09:27 This would create a seamless transition to full membership proofs from a scaling perspective 18:10:22 How big of an increase would that be Artic? 18:10:28 We could also consider a 4x increase in the minimum node relay fee 18:10:52 +1 for higher fees 18:10:57 I wouldn't be opposed to a fee increase 18:11:12 Even without any idea how deep "their" pockets are? 18:11:32 How big is the current estimates for a 2 in 2 out full membership proofs Tx in bytes? 18:11:45 That's one of the dangers that I personally see: to rush into some unwise fee increase 18:12:02 i would be in favor of higher fees even if there isn't an attack, sub one cent is too low 18:12:29 is it too low? low tx fee was a seling point 18:12:34 I don't think we'll ever know how deep their pockets are until they stop 18:12:50 The consideration is that fees are node relay not consensus 18:12:57 kayabanerve, jberman : Do you have an estimate for 2in/2out FCMP tx size? 18:14:15 I'd be fine with the current "normal" tier fees of about 1.6 euro cents per standard tx. Going above 5 cents kills the "digital cash" aspect imo 18:14:25 I asked the above question in the last MoneroKon 18:14:48 My bank charges 10 cents per tx and I absolutely despise them for it 18:15:30 That is my idea 18:15:41 Besides, it's only cheap in fiat terms at current fiat evaluation 18:16:20 By the way I can now speak freely after the end of the trial 18:17:30 If there were a way to have a purchasing power oracle for a unit of XMR, the exchange rate changes wouldn't be a big problem. Maybe it could be linked to network hashpower since hashpower is linked to XMR purchasing power. That makes assumptions about how RandomX hashpower will change in the future. 18:18:15 There is a justification for a 4x increase in the minimum node relay fee while keeping the current quadratic fee scaling 18:18:56 There is tx per day and block size 18:19:20 In the absence of spam 18:20:07 tying anything to oracles is a huge can of worms and very large attack surface, even just to hash rate 18:20:09 By the way I am not convinced this is a spam attacj 18:20:30 Fee level is _okay_, panic raising fees is stupid. When price pumps eventually will we panic drop fees? Hardfork every time? 18:20:32 Attack 18:21:45 Also fee increase does not fix the underlying security problem, assuming there is one. Just makes it more expensive, which would still be ridiculously cheap for large scale actors 18:22:06 And to be fair 140k tx/day is nothing. Increasing ring size is most sensible thing to do, if do anything 18:22:16 We need to take a very close look at the impact of the Binance delisting on DEXs, instant exchanges, peer to peer trading and even CEX customer behavior 18:22:23 note that this works only against DoS, not black marbles. I know you don't think at all this is a black marble scenario, and I think you're wrong on that 18:22:54 IIRC this paper says basically you can use congestion pricing to form a fee oracle just from user bids for block space: Huberman, G., Leshno, J. D., & Moallemi, C. (2021). "Monopoly without a Monopolist: An Economic Analysis of the Bitcoin Payment System." https://moneroresearch.info/index.php?action=resource_RESOURCEVIEW_CORE&id=78 18:23:19 It does increase the cost 18:23:20 But that would require accepting a lot of congestion I think. 18:23:32 ... I am suggesting this combine with a ring size increase 18:24:04 So a multi faceted approach 18:24:43 After the Mordinal incident, analysis of P2Pool outputs, and the 2021 spam incident, I have thought for a while that current ring size provides a safety margin that is too low. 18:26:43 We are 25 minutes past the hour. Should we continue to monitor the suspected spam and continue discussing next Wednesday? 18:27:22 I have my doubts. That being said there are solutions that can deter a black marble attack until we move to full membership proofs 18:27:36 thanks. if this paper implies congestion in the sense of tx volume, there are live working examples for that 18:28:59 Yes but Monero has an adaptive block size 18:29:02 We can revive Mordinals so people will start spamming their NFT transactions 18:29:11 Problem solved 18:29:22 So Bitcoin solutions will not work 18:29:26 Black marble flooding was one of the first theorized attacks against Monero privacy. It was analyzed in the first MRL research bulletin. So...plenty of time to develop a solution. almost 10 years 18:29:48 Lol when tx_extra weight clawback? 18:30:29 ArticMine: IIRC the paper suggested modifying bitcoin's block size limit to adjust to congestion. It was supposed to provide a constant miner revenue from txs in purchasing power terms. 18:30:36 The best solution is to increase the ring size. Ideally to infinity with full membership proofs 18:30:47 Limit tx extra to 256 bytes so each Mordinal will create a dozen tx 18:31:13 If the ring size were to increase, what would be a reasonable hard fork schedule? 18:31:43 Hard fork is +6 months after binary release 18:31:55 Not earlier 18:32:06 Which is not relevant to Monero with its adaptive block size and tail emission 18:33:03 This is in the paper: "To summarize, this analysis suggests the following simple adaptations to the current protocol. First, a smaller block size K is preferable. Second, an adjustment of the block rate to μ = λ/(Kρ∗) in response to demand λ. This keeps congestion constant at ρ∗, yielding a stable, desired level of revenue." 18:33:19 Yes, it is probably not feasible because there are a lot of economic assumptions. 18:33:23 The trouble with Bitcoin like coins is the need to replace falling block rewards with Tx fee 18:33:23 This premise has never been shown to work 18:34:33 We only need to be concerned with pricing out spam while keeping the ham affordable 18:38:01 "~5.5kb" (https://github.com/monero-project/research-lab/issues/100#issuecomment-1609536076) 18:38:03 Tangentially related (and not proposing we do anything now): but does anyone feel that the emissions is currently too low compared to other chains which makes XMR comparably less profitable to mine ? IMO security budget could be higher 18:38:06 Once I have an estimate on the full membership proofs Tx size I can get to work on the modifications to the scaling algorithm and fees 18:39:11 So 6000 bytes 18:40:00 IMHO changing anything about the emission would reduce the purchasing power of the block reward more than keeping it the same. It would reduce confidence 18:40:15 I will review the GitHub link 18:40:42 I have thought about it, but I'm vastly deterred from touching a monetary policy we have good consensus on 18:40:58 Yes I'm of the same opinion. Wish it was higher from the get go 18:41:27 This is very workable with the additional flexibility on increasing the minimum node relay fee 18:41:29 Doing it now is a bad look 18:41:35 If we consider the spammer with unlimited resources then increasing the ring size wont stop him, neither limiting the block size nor letting the fees go up (like btc). I also cant see what could be done. Maybe a solution that encopasses all of that (increased ring size, smaller blocks, higher fees). The question would be then, what users want the most? Privacy, low fees, constant blocksizes... I personally think that Monero has a pretty damn good privacy already. I wouldnt care about smaller blocks and higher fees. I actually think it is a greater selling point to know what will be the maximum blockchain size in 5 years from now then saying that the fees will remain this low (in XMR terms) in five years. I think money should be scarce (not only in the supp ly but also in the resources required to operate it) as well as acceptable. So it is a personal trade-off. But anyway, at this point, I dont think any changes are necessary, if things get bad in the future, I'm of the opinion that we should reduce the block size so txs with higher fees have higher priorities and the spammer would need to pay more and therefore benefit the miners. 18:43:25 No company has an unlimited budget. Also a flood XMR could likely be illegal and at best a public nightmare 18:43:49 Increasing ring size makes it exponentially harder to get a deterministic hit FWIW 18:45:22 an attacker doesn't care about legality, it actually may be approved by a government 18:46:19 In the current money printing system everything is legal and unlimited 18:48:19 I think we can end the meeting. We'll keep monitoring. Read you next week. 18:48:43 hopefully next week we will have more information how the spam evolved 18:50:05 this reminds me of Artic bringing up Nielsen's Law (https://en.wikipedia.org/wiki/Jakob_Nielsen_(usability_consultant)#Nielsen's_law) in his recent interview. would it make sense to define constraints in the (dynamic) block size according to it, or something like it? this could be one way to gain predictability without sticking to static parameters. 18:50:09 MoneroKon CFP deadline is this Friday… if you would like to do a talk or host a workshop, submit proposal here: https://apply.monerokon.org 18:54:59 My take is a reference tx size of 6000 bytes. This will allow for a 2.5x increase in ring size. When combined with a 4x increase in the minimum node relay fee we are looking at a 10x increase in the cost of a potential spam attack while only having a 4x increase in the cost of ham 18:54:59 We can also implement the ideas of capping the growth to Nielsen's Law l proposed in the last MoneroKon 18:54:59 This can be implemented as an interim solution until full membership proofs are available. 18:56:04 The beauty here is that when full membership proofs are available it will be seamless from a scaling point of vie 18:56:39 View 18:59:13 as an aside, even without FCMP, we're looking at 2500-3000 byte Seraphis transactions (depending on variant), assuming ring size = 128 18:59:56 The use of graphics processors for parallel processing of verification is required here particularly for devices such as the Monero Nodo that has a powerful and idle graphics processor 19:01:08 This can be accommodated with the current reference tx size of 3000 bytes. 19:01:38 So if it is available it is an option 19:02:36 I am proposing doubling the reference tx size to 6000 bytes 19:04:31 So ring 40 under the current proofs, ring 512 under Seraphis without FCMP 19:04:59 if an attacker establishes a communication channel with a miner/miners who use modified node software, can't they lower the cost imposed on them by the minimum relay fee? 19:05:59 Yes 19:06:06 I prefer to treat miners as rationally selfish actors who instasell every spendable reward they earned 19:06:52 This is the danger with going too high on the minimum node relay fee 19:08:11 without Seraphis we also don't have privacy perserving light wallets for mobile. that means too large ring sizes can make monero unusably slow on mobile phones. 19:10:15 are you suggesting that with a "moderate" minimum relay fee bump, like 4x, it's less likely that such channels are established? 19:10:27