08:24:47 .merges 08:24:47 -xmr-pr- 9287 9345 9357 9380 9414 9416 9421 9425 9426 9429 9430 08:24:56 so when do we plan a new release? 08:45:46 Ask in today's meeting? 09:33:58 Quick reminder that we will have a meeting here today 18:00 UTC, not anymore in #no-wallet-left-behind:monero.social. Announcement: https://github.com/monero-project/meta/issues/1053 11:16:55 sech1: luigi is away after wednesday for a bit 11:17:00 so we'd have to tag before that 11:20:32 And gui? I want to release p2pool 4.1 before that 11:31:06 I guess the time frame is the same for gui 12:28:55 We should keep this room for current repo stuff. Todays meeting probably an exception, but stuff that isnt on the repo should be in a different room. 12:28:57 i suggest renaming and readderessing nwlb to monero-next/upgrade/evolution, and doing the meetings there. -dev for current repo work, -next/upgrade/evolution for in progress dev work that is not on the repo, and mrl for ideas/research 12:33:37 > We should keep this room for current repo stuff. 12:33:51 Why would that be better? What issue, what potential problem would that solve? 12:34:34 nwlb is the place of discussion fkr the topics, it should be the place for the meeting 12:34:59 Problem is that nwlb is named incorrectly, its not just a wallet room 12:35:29 And doesnt have monero- addressing 12:35:37 If we move FCMP discussions between meetings to here as well, things are ok then? 12:36:27 We are hell-bent to introduce and hardfork to FCMP *as soon as possible*, right? 12:36:47 no, fcmp has 0 prevalence on monero-project/monero. Fcmp and seraphis should be in monero-next/upgrades/evolution (aka nwlb) 12:37:14 Its not finished, not even in review stage 12:37:54 Why should GitHub repositories and their content or content that is missing there decide what we should or should not discuss here? I am pretty confused about the approach. 12:38:28 future protocol upgrades already have their own place of discussion, its just lost due to the naming if the room 12:38:39 The same reason nwlb even exist, rbrunner. 12:38:51 Is this room overflowing with long and complicated discussions? 12:39:20 The logic of "move meeting to -dev" woukd also apply to "delete nwlb and move all discussion about seraphis and fcmo to -dev" 12:39:35 Nwlb is 12:39:45 Apart from deleting what has historical value - yes, let's move!$ 12:40:22 All Monero development 12:40:36 I disagree 12:41:38 its not monero development until its on the repo. Seraphis is 2+n years away 12:41:59 Fcmp likely >12 months before hitting the repo 12:42:31 And that's why we are not welcome to meet here weekly and discuss FCMP? 12:42:54 Yes 12:43:03 Strange 12:43:55 Where is the benefit, the win, the improvement only to discuss things here that are "in the repo". I ask again, is the room overflowing with too much stuff? 12:44:03 nwlb should be a room dedicated to upgrades that arent on repo yet 12:44:23 (nwlb should be renamed and readdressed) 12:44:44 The benefit is the same as "why we have nwlb" or "why we have mrl" 12:45:16 We dont have mrl meetings here for the same reason we shouldn't have seraphis or from meetings here 12:45:30 Seraphis or fcmp* 12:45:54 Re overflowing: no -dev is not, but mrl and nwlb are 12:48:44 I can't follow. The MRL and the MWLB rooms are overflowing with too much stuff? Like, right now, this week, last week, last month? 12:49:34 Should we do mrl meetings in -dev? 12:50:03 If I say no, I have lost? 12:50:41 Yes 12:50:46 :) 12:52:01 Thing is: It seems, at least to me, that "research" and "development" are quite easy to make into two heaps. If you want to part "development" further into sub-heaps, you arrive, IMHO, exactly at the situation that prompted the resurrection of general-purpose dev meetings of yesteryears: Something will always fall through the cracks. 12:54:37 -dev meetings can still resume, but should focus on releases, triaging issues and prs, dev reports and potentially dev ccs discussion 12:54:41 If you ask me, we will see pretty fast whether we are on the wrong track: If we have, after broadening the scope of the meeting, so many things to discuss that people lose overview and 1 hour is hopelessly too short, it's clearly "back to the drawing board". If we fare ok with a single meeting, well, "simple is simply simpler" 12:55:38 If you have so many things to discuss in a dev meeting as you see it before your inner eye, why aren't there such meetings? 12:56:08 because i was banned 12:56:14 :) 12:56:18 Oh, plot twist 12:57:40 i had no comments to make / no observations to observe. The above is how i think dev meetings are best served 12:57:52 they dont have to run an hour either 12:58:41 Well, I am curious whether others will read and leave their opinion. I think I get your general idea now, but I still don't see enough improvement compared with a dead-simple meeting discussing all things Monero dev, whether on repo or off repo or whatever 12:59:09 And with the meeting right here. The channel will bear about 300 new lines from the meeting. 13:00:58 I am not sure you are fully awary how shy a kind of animals our real devs are, the ones actually writing C++ and Rust code. With too many meetings and too many rooms you don't win but they just run away and hide. 13:02:51 And after all discussions are not at all restricted to this room, or 5 other dev related rooms at that. Proper place for dev related discussion are, to a large extent, GitHub issues. 13:06:09 Mrl, upgrades/next, dev 13:06:09 3 rooms, 3 tiers of development, 3 distinct crowds, 3 purposes 13:21:54 "And that's why we are not welcome to meet here weekly and discuss FCMP?" , "Yes" :o 13:58:23 FWIW I consider this canonically on topic. 16:36:50 sech1: can you release that today? 16:38:26 yes, I'm waiting for CI to finish building the binaries 16:38:55 https://github.com/SChernykh/p2pool/actions/runs/10354764833 16:48:56 <0​xfffc:monero.social> My vote is to have the regular meeting in another room. Not that it is unrelated to here. No. It is just this room is extremely crowded. And there is always Q/A going on. 17:05:49 Meeting here (at least for today ...) in a bit less than 1 hour 17:14:09 Arguably having the meetings in the 'core' rooms will increase visibility and participation 17:14:31 Having the meetings scattered across many rooms will probably reduce participation 17:18:29 plus, discussion related to the major refactoring being done to allow for future code additions is arguably pretty on topic 17:24:48 10 Monero devs with around 20 opinions :) 17:36:14 one day in the coming years i am getting involved in at least one of these: 17:36:15 - some crypto development(xmr? bch?) 17:36:17 - some kernel or firmware development(linux? freebsd?) 17:36:19 - some compiler development(sbcl?) 17:36:21 the idea is trying a new method of programming to tackle with problems which require a total new mindset, different than what i had till now. 17:36:27 one day in the coming years i am getting involved in at least one of these: 17:36:29 - some crypto development(xmr? bch?) 17:36:31 - some kernel or firmware development(linux? freebsd?) 17:36:33 - some compiler development(sbcl?) 17:36:35 the idea is trying a new method of programming to tackle with problems which require a total new mindset, different than what i had till now. 17:37:31 to be precise, in exactly one of these 17:44:10 Arguably having the meetings in the 'core' rooms will increase visibility and participation << nwlb would become a "core" room 17:44:40 its current name and status is foggy as best 17:45:13 Its like "where we put everything that we dont know where to put, because its neither -dev or -mrl" 17:47:02 the same argument for creating nwlb is the same argument for expanding it to encapsule more than seraphis wallets (which it has already done, by becoming the come of all things seraphis, fcmp, and any other not-yet production items) 17:48:38 The only thing that needs to change is the room name and address, from nwlb to monero-pickaname 17:55:29 How about #monero-devel ? We'll make fedora and debian people both happy. 17:56:12 Windows people can stay in #monero :D And #monero-gui for Mac users. 17:56:38 What does "devel" stand for? Just "development" abbreviated a bit differently? 17:57:13 Yes. Headers are in -dev packages for debian and -devel packages for fedora. Or vice versa, I always forget. 17:57:42 So all clarity successfully eliminated :) 17:57:47 Thank you. 17:58:59 Just to clear something up for me, some version of fcmp *will* most probably end up in monero, barring unforeseen problem, correct ? 17:59:14 Yes. 17:59:19 Thanks. 17:59:39 After audits, testing, and scaling is sorted 18:00:07 Meeting time. Hello! https://github.com/monero-project/meta/issues/1053 18:00:19 *waves* 18:00:22 hey 18:00:32 Hello. 18:00:48 Hi 18:00:58 Hi 18:01:02 Hello 18:01:07 <0​xfffc:monero.social> Hi everyone 18:01:14 hi 18:01:24 Is there an agenda for this meeting? 18:01:30 hello 18:01:35 No 18:01:42 howdy 18:01:52 Before we come to the reports: We have a surprising amount of ideas floating around how many dev meetings with what kind of foci in which rooms we should hold. 18:02:07 I propose to keep discussion about this *out of this meeting* 18:02:18 I think we should use the time to discuss dev related things. 18:02:45 Meta-discussion about meetings should go, IMHO, in a new issue in -meta on GitHub 18:03:00 Since were in dev, lets start with pressing dev matters. 18:03:01 1. We have to tag by wednesday 18:03:06 If people are ok with that, I can open such an issue later. 18:03:49 What is there to report from last week? 18:04:01 Is everybody ready to tag, or are there any issues or PR's that need review or implementation asap 18:04:32 Can you please hold back for 5 minutes more, so we can get the usual overview who did what during the last week? Thanks. 18:04:36 I'll release p2pool v4.1 today which needs to be included in monero-gui v0.18.3.4 18:05:30 added missing functions to the API, now just TODO's and QUESTIONs to fix, I think I can see the finish line, but it will probably still take a while 18:05:59 Sneedlewoods, this for what? 18:06:18 Seraphis? 18:06:47 me: finalizing a draft PR + writeup for the fcmp++ integration, shooting to submit that PR by tomorrow. The PR so far includes a Rust FFI to use @kayabaNerve's fcmp++ crate, code to migrate existing cryptonote outputs into a merkle tree for fcmp++'s, growing the tree as the node syncs, an algo to trim the tree, and changes to the `cryptonote::transaction` class for fcmp++. I plan 18:06:49 to also implement tx construction & verification, and consensus changes in that PR 18:06:57 file df_draft_fcmp_v1.pdf too big to download (1184797 > allowed size: 1000000) 18:06:59 df_draft_fcmp_v1.pdf 18:07:03 This week I continued reading the code and theory of FCMP. I still don't understand the details of the circuits in the `prove` and `verify` functions and the exact details to fetch and store the tree in the database but I will improve my understanding with time since I'm much more familiar with the code and terms used. I believe it would be better to try to do something concrete n 18:07:05 ow. Here is a draft of my understanding so far of some concepts. There may be big errors and imprecisions and it is still very shallow. Take it with a big grain of salt. I will improve it as I also improve my understanding. 18:07:34 @jberman: What will you PR against? 18:07:52 me: polishing the carrot doc and soliciting auditors (I have one meeting planned thus far) 18:07:56 master in core Monero 18:08:31 Also started to prepare my incoherent notes of things I found during the API work all over the place and plan to create a github issue for that. Not sure if I should put it into monero-project issues though, because it's a bunch and I don't think it makes sense to create a github issue for everything I found suspicous. 18:08:33 So much about "future stuff" :) 18:08:56 <0​xfffc:monero.social> Me: since Wednesday I was reviewing different PR. The big one is reproducible build PR 8929. Which review is almost complete. 18:09:02 off topic rbrunner, dont bait me 18:09:18 Will that go into a separate fcmp branch for the time being? 🤔 Can't be marged on master till it's ready, no? 18:09:32 You asked nicely to leave that discussion out of this meeting 18:09:48 Alright, understood. 18:09:52 It could be merged but not activated 18:10:31 Similar to some of koe's seraphis preparedness prs 18:10:48 @rottenwheel I'm thinking it could remain a WIP draft PR until it's ready 18:12:14 Me: worked on socks v5 - adding ipv6 support to proxy requests. Also worked on LWS frontend API project 18:13:05 So soon people will be able to run their own LWS instances, are you nearing completion? 18:13:47 They already can 18:13:49 sorry ofrn, I'm slow, I think here you can get enough information what I'm working on in general https://github.com/monero-project/monero/pull/9368#issuecomment-2168451558 18:14:17 They are already able to run their own LWS instances, there just isn't many wallets supporting it because there's a decent amount of work to tx construction in particular 18:14:36 I see. 18:14:47 The LWS API is json, and is pretty straightforward I think 18:15:28 <0​xfffc:monero.social> One important thing is the PR 9404 [1] needs reviewer. The PR has been running on stressnet with different scenarios. And no issues has been reported. With this PR, we have instantaneous start up. 18:15:29 <0​xfffc:monero.social> 1. https://github.com/monero-project/monero/pull/9404 18:15:30 i've already done all that work 18:15:35 vtnerd and rbrunnner 18:15:49 thank you endogenic 18:15:51 i've told you this a few times, but i havent ever been contavted by vtnerd 18:16:01 I told him a few times, but somehow we didn't end up being able to collaborate yet 18:16:06 The problem is that I haven't published it yet 18:16:12 endogenic, show and tell 18:16:13 The only reason I haven't done that is because I don't have help 18:16:49 you say show it but what happens is that everyone else is just going to use my stuff before I get to actually launch my own product - this is just because I don't have any funding or any help from anyone 18:16:56 so it sounds a little bit like a Catch-22, but I swear that I have done all of the work already 18:17:03 it's directly from wallet2 18:17:21 i'm sure we can figure something out, but I literally am programming right now and I'm so close to releasing this stuff but I've just had to make it better because of the fact that all this other work is happening meaning I have to progress even further before I can release 18:17:28 so if we had worked together, then I think it would've been released already 18:17:33 it's just not fair to me either 18:17:34 Alright, looks like we are through with the reports. ofrnxmr , we have a pressing issue, tagging a new release? 18:17:37 it's actually been really difficult 18:17:41 to see this 18:17:58 geez so cold? 18:18:16 yes. We have to tag the new release by wednesday 18:18:19 Is it in a private GitHub repo, or something completely offline? 18:18:31 me? vtnerd? 18:18:34 private repos 18:18:36 yes 18:18:39 long history 18:18:42 many commits 18:19:05 Pardon my ignorance, why do we *have* to tag? Would a long pause result if we don't? 18:19:40 i've been homeless at times.. and many other things no one would believe lol 18:19:49 Im not sure about the reasoning dor 18.3.4, id have thought it should be 18.4.0 (?) Major minor patch, this isnt just a bugfix release. 18:19:58 Then just make it public, it's the least friction. Theres probably no shot at keeping it private, it's unlikely people will support private wallet code 18:20:10 I don't intend to keep it private at all 18:20:14 it's not intended to be private 18:20:21 it's just intended to be ready before it's released publicly 18:20:38 But it puts me at a major disadvantage to release all my work when everyone else has so much funding and help 18:20:45 especially because my work is actually pretty interesting 18:20:53 I have a huge write up.. 18:20:55 and selsta would be the one to ask about "why" wednesday, im just relaying the message 18:20:59 Ok, do we have @selsta around, they may have the best overview what is needed in order to tag. 18:21:31 Endogenic has been delivering groundbreaking code since before the 2000s! 18:21:36 vtnerd would you be open to collaborating with me privately on the lws support for 2-3 weeks max? then it all goes public 18:21:47 Is anybody here right now directly involved in that release? With contributing PRs that may still not be re ready? 18:21:48 He just hasn't been able because the life threats and lack of funding guys! 18:21:49 rottenwheel please, man, you're going to eat your words 18:22:07 you have no idea 18:22:13 what i have been through.. 18:22:30 endo, please ignore 18:22:31 lws support is done tbh 18:22:33 on my side 18:22:51 rottenwheel, @endogenic, maybe this special topic can wait until after this very meeting? 18:22:53 i just need to update ny client for vtnerds new api 18:22:59 as you know he added spenda and recvs 18:23:01 i think it both a negative and positive we have duplicated efforts around endogenics area of work/expertise. neg being the dupl efforts, positive being we have more than 1 developer who understands the problem and can pick apart each others implementation. but i think we should leave it at that for this meeting unless it is on the agenda 18:23:04 this was one of the things I was going to announce 18:23:12 create a private repo and invite vtnerd as collaborator? 18:23:32 Whatever does not derail this meeting. Please? 18:23:34 The only thing is that vtnerd might have confidentiality agreements, or he might not be willing to enter into a confidentiality agreement with me 18:23:38 that's why I was really trying to just launch everything 18:23:59 to be honest, honest with you, it's going to be trivial for me to update my client to work with vtnerds new api format 18:24:09 it's just that I was having a little bit of difficulty getting something compiling and stuff like this just takes more time 18:24:17 I don't have admin or any other special rights in here, by the way. If people insist on sabotaging this meeting, they will probably succeed. 18:24:19 there's no communication between me and him and so I can't just ask him about stuff 18:24:41 in other words, it's not like I'm paying him for his time so I don't feel comfortable asking him 18:24:50 **Regarding the release** << are there any prs that anyone is trying to get into the release or issues that need to be resolved? 18:24:54 we've asked a lot from him already to be honest 18:25:04 Plowsof has matrix ops 18:25:07 vtnerd: sorry I meant to address your comments on https://github.com/monero-project/monero/pull/9135 earlier this week; thanks for the review. In your opinion after looking at the PR's code, once those issues are addressed, do you think it worth including in the next release? 18:26:06 yes I'm around 18:26:09 jeffro256: I recall it being pretty much ready for inclusion 18:26:30 This is running on stressnet, but were unsure if it caused an increase in reorgs (?) Rucknium: 18:26:31 Btw that PR is running on the stressnet currently 18:27:26 The reorg increase _may_ have been due to a very poorly connected miner though 18:27:32 I was looking into this too, I *think* there were just more reorgs this time around, but I'm actively looking into it 18:27:38 ofrnxmr: jeffro256 These was an increase in re-orgs around the time that we put that PR into release, but other code was put in release, too. It's not a controlled experiment. 18:27:47 r​brunner7: nothing is needed anymore on the -cli side, but if something is ready and well reviewed we can include it with the next round of merges 18:27:56 And miner behavior isn't a controlled variable, either 18:27:58 There were already mass reorg messages before, but they didn't feel *as* common. Of course, there's been a lot of variables changed with the new fork 18:28:00 Sorry, Murphy's law. My cable modem just had the idea to reboot now. 18:28:44 Alright, seems I am back on track. 18:28:47 Sorry I keep duplicating people's thoughts delayed by 15 seconds lol 18:29:26 I will compile a stressnet node without the PR and observer reorg behavior 18:29:47 At any rate, it appears that the nodes are syncing just fine even with the increased messages 18:29:54 @jeffro: I also saw the reorgs with a stressnet daemon without your PR ... 18:30:00 (Reorgs have calmed down a lot on both new and old stressnet) 18:30:30 New stressnet isn't being spammed yet FWIW. 18:30:51 Ok, looks like we have the release and the tagging sorted, yes? At least as far as meeting participants are concerned. 18:31:08 gui is tagged separately? 18:31:10 Rbrunner, spackle was having almost every block reorged. I hypothesise unrelate to jeffro's pr, and related to a poorly connected miner 18:31:23 I still need to have p2pool v4 in the next gui release 18:31:30 sech1: yes separately 18:31:30 You had aksed "why wednesday", im not sure thats been answered 18:32:57 I can only imagine luigi will be away for some time afterwards 18:33:32 proprietary 18:33:42 Everything can run towards release without them after tagging? Surprises me a bit, but I don't know the details 18:34:23 rbrunner7: yes, apart from the auto-updater 18:34:36 GitHub compiles, or people compile repeatable builds, and binaryfate signs and uploads? 18:34:45 yes 18:35:00 Ok, nice. 18:35:20 Ok, hopefully news about the wednesday deadline will reach everybody 18:35:27 <0​xfffc:monero.social> There is this one too 👆🏻. 18:35:51 https://github.com/monero-project/monero/pull/9404 << for people on irc 18:36:09 (0xfffc's reference) 18:36:20 You hope to have that reviewed and included until Wednesday? 18:37:05 9404 is a bit risky to include in this release in my opinion 18:37:10 https://github.com/monero-project/monero/pull/9376 18:37:24 This one as well 18:37:26 <0​xfffc:monero.social> I am not hopeful about the “included” part. 18:37:51 <0​xfffc:monero.social> I believe is safe to include. IMHO. 18:38:12 shooting to review 9404 this week. I can shoot to have it done by wed, but doesn't seem required for this release to me 18:38:21 would be nice if 9376 gets a second review 18:38:30 agreed 18:38:34 including deep internal changes in the daemon last minute isn't good 18:39:47 I could have a look tomorrow at 9376 18:40:54 9376 has been running on stressnet for a while now and works as expected. Much. Much faster started speed when txpool is large 18:41:37 Yeah, if there is something wrong at all then maybe an unintended side effect, influence on something not yet thought about. 18:42:53 Ok. Do we have some other subject to discuss now? Is the dev here that, with their question, made the ball roll towards a broader dev-meeting? Something with binary stuff in JSON maybe? 18:43:09 Yes 18:43:50 Do you have a pointer? 18:43:53 rbrunners serialization / strings issue (he has a pr on stressnet) imho needs to be investigated and property implemented asap 18:44:31 Block sync size + the serialization issue = problems that we dont want on mainnet 18:44:45 Yes, needs a taker, a dev with some free time on their hand. Which may be rare right now ... 18:44:46 rbunner7: well, I have extended monero-wallet-rpc with two endpoint for binary blob key image export/import, don't know if that fits here, and it would be cool if it could be included 18:44:59 <0​xfffc:monero.social> ( Very important ) 18:45:07 I think the connection is with cuprate 18:45:14 hinto hinto suggested the binary JSON-RPC deprecation 18:45:32 We need to implement a dynamic BSS and/or need to fix the serialization / strings limitations 18:45:45 sorry, didn't want to disturb :/ 18:45:50 BSS 18:45:52 ? 18:46:14 How many blocks to sync in one gulp 18:46:18 bss= block sync size 18:46:24 hello, yes the issue could maybe use more discussion from other devs although I think any approach within reason will be reasonable: https://github.com/monero-project/monero/issues/9422 18:47:16 Ah, ok, there is already an issue as a nice place to discuss. So just needs more eyes, and people opining :) 18:47:20 I'm willing to make the changes to `monerod` although it may be faster if someone more familiar were to do it, at the very least I can review 18:47:57 Do we run into nasty coordinating proplems if we change that with clients that use these endpoints? 18:49:35 jberman suggested deprecating the endpoints within 12 months, I'd assume they'd still work afterwards but are not recommended in documentation 18:49:49 ((We also need to stop using words interchangeably, ie "block" and "ban". Dns blocklist, banlist, block size, block time (ban), block time (blocks) 18:49:49 block-sync-size really means "batch size")) 18:50:01 For the record, the issue that needs a competent dev for implementing a good longterm solution, about that "BSS" issue: https://github.com/spackle-xmr/monero/pull/12 18:50:47 ^^ this is high priority imo 18:51:52 <0​xfffc:monero.social> I can take this. The bandwidth efficient propagation will be delayed though 18:52:02 If nobody finds time near- or mid-term, we could probably just go with what I did for a first quick fix, just rising those limits. After some testing whether it really does not blow up somewhere. 18:53:17 Which limit type was actually hit in the stressnet case? Objects, fields, or strings? 18:53:24 Strings. 18:53:33 We still have a max packet size of 100mb. If we were to play it safe, we should mimic stressnet and change bss from 20 > 5 and implement rbrunners suggestion. 18:53:35 but neither of those is a long term solution 18:53:50 At least. But I did not rise the limits one-by-one. 18:53:58 <0​xfffc:monero.social> ( multiple limitations at play here ) 18:54:22 strings was the error that 0xfffc's debugger showed 18:54:44 Yes, I also saw it myself in the debugger, in some nice live scenario. 18:56:25 The problem with bss = 20 is the packet size limit 18:57:19 0xfffc: IMHO it makes sense for you to try to solve the BSS issue. The efficient tx propagation proposal has some privacy issues that the dev & researcher community may want to discuss more. 18:57:23 Simply raising serialization limits can eventually also hit the packet size limit. 18:57:25 dynamic bss is best solution to the static bss=20 imo 18:57:40 And the current stressnet will last for about two months longer 18:58:10 This is the efficient tx propagation issue: https://github.com/monero-project/monero/issues/9334 18:58:16 <0​xfffc:monero.social> I am all ears. If community thinks this is more important. I will switch immediately. 18:58:48 Unrestricted rpc will fail to connect when txpool > 100mb 18:59:21 Restricted batches rpc calls into 100tx/call. We need to fix this for unrestricted rpc jberman @jberman:monero.social: 18:59:27 Hmm, gut feeling tells me that this efficient tx propagation thing might take quite a while to get merge-ready, maybe really a good idea to attack that BSS issue first. 19:00:38 And well, it may take a while until we have the next spam wave, or organic traffic in stressnet amounts 19:01:04 <0​xfffc:monero.social> Sure. Then I am switching to fixing serialization limit right now. ( will go back to bandwidth efficient propagation after that ) 19:01:38 bss or serialization? 19:01:40 Splendid! 19:02:18 <0​xfffc:monero.social> Serialization will be easier to attack at first IMHO. What do you guys think? Which one we should start with? 19:02:53 Serialization only helps up to 5mb blocks 19:03:10 <0​xfffc:monero.social> Keep in mind the multiple limitations 👆🏻 19:03:17 Bss can be reduced to 5 (like stressnet) to add a stop-gap measure 19:03:35 <0​xfffc:monero.social> My impression was we hit the serialization first. 19:03:48 But dynamic bss is a better solution (which would allow <=100mb blocks) 19:03:50 <0​xfffc:monero.social> So it is better to start from the bottleneck. 19:04:04 In my understanding things are linked. With a good algorithm to dynamically adjust BSS to traffic and block sizes the serialization limits are not that critical, and we can stay with relatively low "sanity checks" 19:04:05 Yes. We hit serialization at 1.5mb blocks 19:04:21 after Fixing serialition, we hit packet at 5mb blocks 19:04:44 If dynamic bss, we hit serialization at 30mb blocks 19:05:01 After fixing serialization + dynamic bss, we hit packet at 100mb blocks 19:05:11 Alright, 0xfffc , I think things will clear up as soon as you wrap your head around the hole story :) 19:05:20 *whole story 19:05:29 <0​xfffc:monero.social> ( So in every case we hit serialization limitation. ) 19:05:46 No 19:06:06 <0​xfffc:monero.social> But rbrunner7 thinks the dynamic bss is more important than serialization limit. 19:06:07 The final boss is packet 19:06:36 <0​xfffc:monero.social> That is the monster at the end of story :)) 19:06:52 this likely needs testing, but just changing `std::numeric_limits::max()` to something sane like 1000 may be fine here: https://github.com/monero-project/monero/blob/caa62bc9ea1c5f2ffe3ffa440ad230e1de509bfd/src/rpc/core_rpc_server.cpp#L660 19:07:03 right, bss gets us further (to 30mb blocks) than serialization (5mb blocks) 19:07:48 it shouldn't break wallet2, but could break other clients that rely on it 19:07:49 This was my suggestion (1000 = 100kb(max block size) * 1000 = below the packet size limit) 19:08:36 <0​xfffc:monero.social> Important point: all of those limitations need to be fixed. Let me review each of them. We debugged it once. 19:08:37 <0​xfffc:monero.social> Start from the compromise of (simpler + more important) 19:08:55 "this" = changing the max tx pull to <1000 19:09:02 ofrnxmr: FCMP is not 12m away. We're looking at a PR for a lot of the framework within the next few weeks. Academia is also near complete, and a variety of the libraries are about to head to auditing. 19:09:18 100mb packet probably doesn't need to be touched 19:09:31 To mainnet? 19:09:36 Kaya 19:09:43 The moment that PR is open, it's monero-dev. 19:09:57 <0​xfffc:monero.social> I agree. If we have good dynamic bss algorithm. 100mb should be more than important. 19:10:45 <0​xfffc:monero.social> s/important/enough/ 19:11:15 IMO 0xfffc your time would be really well spent reviewing vtnerd's serialization library changes since those changes would make the limits less important since the serializer is inherently more efficient, and parsing rogue messages is less of a risk 19:12:00 jeffro256: Happy to help with auditors :) Feel free to reach out. 19:12:02 <0​xfffc:monero.social> Great suggestion. I think I have missed those PRs. What was the number? 19:12:06 I think all fcmp integration code (not including reproducible build-related) in the monero repo can be completed within 4 months 19:13:00 Re: RPC method deprecation, given how FCMP++ will change the definition of the output distribution (it's a new pool superseding all others) and TX creation code, do we want to take advantage of the amount of changes to argue hard breaks on other methods? Deprecate prior to HF, remove entirely with the bin that HFs? 19:13:17 Here the PR for the overall tracking: https://github.com/monero-project/monero/pull/8867. Here's the currently opened piecemeal PR: https://github.com/monero-project/monero/pull/8970 19:13:36 I'd hope for mainnet within a year, ofrnxmr. 19:13:45 Sorry for not being present earlier and responding to what's relevant on my end now. 19:14:00 It's a pretty huge task, but replacing the DOM serialization, especially for large funky messages, will help DoS mitigation a lot 19:14:26 Isnt this 7999 redux? 19:14:34 <0​xfffc:monero.social> Great. Well before the time I joined monero community. I was not aware of these. Thanks for mentioning. 19:16:02 <0​xfffc:monero.social> Just the final question. I am between two tasks right now. Fixing out serialization limit, or reviewing vtnerd serialization PR. 19:16:03 <0​xfffc:monero.social> Which one I should tackle? 19:16:15 https://github.com/monero-project/monero/pull/7999#issuecomment-938098462 19:17:53 Full wallets shouldn't need an RPC route for anything like getting the output distribution at all (only light wallets should need to fetch decoy paths), but generally ya the get_output_distribution endpoint can be set to be deprecated in theory when fcmp's are required, and that is a better deprecation timeline we can aim for 19:18:54 Jbernan, thoughts on 7999 vs 8867? 19:19:59 jberman: we still need decoy requests for paths, as you say. How does that enable deprecating getting the output distribution? 19:20:57 My comment was how if we're already making notable changes necessary for practical operation, we can also make breaking changes and provide a single migration guide (instead of two releases with such notable changes). 19:22:12 I really don't see the point in removing the `get_output_distribution` RPC method (even the "broken" JSON one) since it *works* now for the custom JSON parser, and doesn't really affect other code in the codebase 19:22:30 The broken JSON one is a sin, it should be removed ASAP. 19:22:32 Obviously I see the value of *also* having a JSON compliant endpoint 19:23:30 I disagree. Current services depend on it, and removing it for the sake of removing is just annoying to people building things in Monero 19:23:43 We should not have an endpoint which is fundamentally broken to its definition. 19:23:49 8867 seems like it would be fine compared to 7999 for those reasons. If it solves the issue presented in the PR, then fine with me. Would be curious to see how it performs relative to 7999 too 19:23:53 Alright. We are well past the hour. Allow me to close the meeting proper while the detailed discussions about serialization and other things go on of course. Thanks everybody for attending, read you again in 1 week, place to be decided. 19:23:58 It claims to be JSON. It is not. It at least needs to be made hex. 19:24:03 <0​xfffc:monero.social> I have feeling we have opened a can of worms. 19:24:05 <0​xfffc:monero.social> ( 7999 vs 8867 vs fixing serialization limit ) 19:24:40 7999 vs 8867 can was opened 2+yrs ago 19:24:59 Also, it's unclear how to parse it with a custom parser. It isn't spec compliant? Is its binary guaranteed to not to have instances of "? Are some sequences unrepresentable? 19:25:30 thank you everyone, seems there are enough topics to justify a dev-meeting 19:25:35 If it's not spec compliant, it's not properly documented, and it's quite possibly incomplete, it needs revamping. 19:25:59 <0​xfffc:monero.social> I belie we can use the serialization as standalone library. Why not write a benchmark suite, and winner takes all. 19:26:12 SNeedlewoods @sneedlewoods:monero.social: i was never against a dev meeting, ftr 19:26:37 hinto's proposal I believe to keep the code so it continues to function as is, and mentioning it's deprecated in documentation seems fine to me 19:26:40 <0​xfffc:monero.social> Not gonna lie. -dev meeting was more interactive :)) 19:26:50 If FCMP++ breaks the endpoint semantics anyway, can't it stay until post HF, then be removed afterwards? 19:27:36 A benchmark shouldn't be the ONLY factor, since readability and maintainability is also important for open-source health 19:27:38 I see, ya, merging timelines with fcmp's make sense on that front 19:28:11 FCMP++ doesn't inherently break `get_output_distribution` semantics though 19:28:27 <0​xfffc:monero.social> At least it is an objective starting point though. 19:29:03 I think you'll find that both of them are *significantly* faster than the core codebase, especially in the worst case 19:29:22 <0​xfffc:monero.social> ( just a suggestion. Not saying it is the way to move forward ) 19:30:17 <0​xfffc:monero.social> Quick question. The interface is same, or they have completely different interface? 19:30:22 jeffro256: okay, in that case I think it makes sense for it to remain indefinitely, be clearly deprecated in documentation, and have a sane `_v2` version 19:31:18 <0​xfffc:monero.social> ( quick question. The release number should be 18.3.4 or 18.4.0? ) 19:31:36 <0​xfffc:monero.social> ( apologies. Too many “quick questions” ) 19:31:37 18.3.4 afaik 19:32:11 Historically it _should_ be 18.4.0, id think 19:32:37 We usually the major.minor.patch for bugfix releases 19:33:17 The interfaces differ in a lot of ways 19:34:33 It would be really hard to lay out all the differences right here right now, but I'd be happy to DM 19:35:22 <0​xfffc:monero.social> Will dm you. If we want a benchmark suite to do a performance comparison. It should be a fair one. Thanks. 19:35:28 You now need to use a new pool for a new distribution. I wouldn't be surprised if a lot of people hardcoded the pool to 0. 19:37:14 0xfffc: the interfaces are different. 7999 only supports p2p, whereas 8867 supports p2p, json, and msgpack via c++ interface (msgpack is written in LWS already but not in 8867). 7999 changes macros for existing code, whereas 8867 is compatible with existing macros with the exception of trailing semicolons that was used ina few places of the RPC code 19:37:36 I'm not sure how much work it is to add different formats to 7999, Id have to look at it agajn 19:38:37 Why a new pool? Are you saying that because the new output definitions are not backwards compatible with CLSAG? 19:39:49 I _think_ it requires unique template expansions to foreach format, whereas 8867 allows for a single base type instead, to cut down on template expansion and executable size (with a corresponding runtime hit to vtable usage). 8867 can expand to each unique format too if the runtime hit seems too big 19:40:14 Because we're now including all transparent amount outputs? 19:40:15 the new pool is the set of all outputs unified (includes pre-rct and post-rct) 19:40:17 It's the superset of all prior pools? 19:43:17 Oh I see what's you're saying now. That fact still doesn't break the backwards behavior of `get_output_distribution`, though, even if it's less useful, so I still think it's worth keeping the old version around in some form 19:43:43 I'm not saying get rid of it. 19:43:50 > The design is intended to maximize privacy of the source of a transaction by broadcasting it over an anonymity network, while relying on IPv4 for the remainder of messages to make surrounding node attacks (via sybil) more difficult. 19:43:51 is that true? IPv6 support isn't completed? 19:44:09 I'm saying we're changing how it's almost certainly called, removing usage of getting output info in TX construction, and adding new routes to use during TX construction. 19:44:17 > The design is intended to maximize privacy of the source of a transaction by broadcasting it over an anonymity network, while relying on IPv4 for the remainder of messages to make surrounding node attacks (via sybil) more difficult. 19:44:19 https://github.com/monero-project/monero/blob/master/docs/ANONYMITY_NETWORKS.md 19:44:21 is that true? IPv6 support isn't completed? 19:44:26 Yes, there 19:44:29 With so many semantic changes to practical use, why would we not also get the syntax changes out of the way? 19:44:52 (fixing a broken, undocumented route which we still have yet to settle if it's complete or not) 19:45:23 Else we have two nontrivial RPC updates with two migration guides. I just want to unify them into a single event. 19:46:09 The only breaking change I've discussed so far is fixing the ones which claim to be JSON and aren't. I'm not proposing removing methods at this time. 19:46:54 If you want to make a well-defined spec for the encoding they do, if you even can, update their documentation, and remove all claims they're JSON, you're welcome to do so to get me to drop this topic BTW. 19:47:11 Else we can just make them explicitly binary methods or use hex. 19:49:16 Oh yes I understand, I wasn't trying to suggest that you were suggesting removing any instance of `get_output_distribution`, but I think trying to fix this *one* problem with the JSON encoding is NOT going to be backwards compatible (I could be wrong about this premise). Thus, I support making a *new* endpoint (or at least a new way to call it) with these new semantics and syntax, 19:49:17 while still keeping the old one to support currently deployed services 19:50:02 It won't be. We should make the breaking change (its removal) at the same time as we make all other significant changes. 19:50:28 I don't support endlessly supporting a broken endpoint when migration should be trivial (oh no, a call to decode hex, the overhead, the horror) 19:50:39 sorry for interrupting again, but why not deprecate them all and introduce /v2 and give a timeline to remove /v1 endpoints? 19:50:56 I'm calling for the removal timeline to be the FCMP++ HF. 19:51:20 I don't care when they're deprecated, I'd just rather make all notable changes at one moment. 19:51:41 Making the HF the deprecation, so people change to the FCMP RPC methods AND stop using the deprecated method, then removing it however long after, would also be fine. 19:52:01 A further update on 7999 - it replaced the binary RPC too, but not json 19:52:24 We can deprecate even earlier, helping out Cuprate, but then we twice need people to update their RPC callers (once for breaking changes, once for practical, not once for both). 19:53:03 As long as the method is deprecated *during the FCMP HF binary release*, so users can only update once at that specific point, I'm happy with timeline. 19:53:14 "endlessly supporting a broken endpoint" in this case means just not touching the code, and is pretty trivial. breaking changes across the ecosystem I think is less trivial 19:53:30 deprecating in documentation I think is fine 19:53:44 jberman: You're not only proposing tech debt, you're wrong. 19:53:49 (and intro'ing a new _v2 endpoint) 19:53:57 Cuprate has to implement this to achieve parity 19:54:09 Cuprate having to implement a broken method which is fundamentally undocumented is not trivial. 19:54:22 They can just not bother. Then we have two different RPC specs. 19:54:23 hardenedsteel: the daemon supports ipv6, except when using a proxy. I'm working on ipv6 support now for that 19:54:57 How many people using this endpoint are going to switch to cuprate? 19:55:05 Do we want to constantly discuss how some impls only have some routes and open that door, or do we want to remove *a broken endpoint* and have the ability to clean up our own code in that regard? 19:55:26 That doesn't change any alt impl must support this or must maintain its own list of supported methods vtnerd 19:55:39 should i update the docs then? https://matrix.to/#/#monero-dev:monero.social/$QhZBR4drH0jBNHKd03sNJJx2dSE8f7A5ftutn1NOcyI 19:55:45 I don't even know who uses this method now so it's so cursed. I'd guess its only used internally to the project? 19:55:58 Which means no one should care if the project stops using it, and later removes it. 19:56:19 lws + openmonero use it as well 19:56:25 Please guys deprecate binary json for cuprate sanity 19:56:32 this is a cry for help 19:56:55 Even if it wasn't for Cuprate I'd be up in arms over this. Broken methods shouldn't be retroactively declared valid with a lifetime promise of support. 19:56:57 LWS doesn't use that endpoint, it uses zmq 19:58:03 OpenMonero will have to make changes for FCMP++. While there, they can stop using a broken method and move to a working one. Then we can deprecate an undocumented standards and remove their custom parsers. 19:58:07 There are definitely other encoding schemes, but as a note on hex, using it here would make the RPC call (an already heavy call at ~5MB) about 2x heavier 19:58:22 Then use bin 19:58:39 Why have this as broken JSON in the first place? 19:59:17 OpenMonero isnt really doing active development afaik, so if broken it will most likely just stop working until someone outside the project issues a pr 19:59:32 They'll already break with FCMP++s. 19:59:50 So that argument doesn't inherently hold so long as we sync their timelines, as I'm advocating, IMO. 20:00:04 IIRC it is used in other places outside of `get_output_distribution` too ;( 20:00:35 get_txpool_backlog iirc 20:00:51 Wow, we have multiple broken, undocumented methods. We should fix them... 20:01:00 tbf it is documented 20:01:06 @vtnerd this isn't calling the JSON endpoint https://github.com/vtnerd/monero-lws/blob/24e6c2f5f3840553c44a5a0f519c5ca7c267f887/src/rest_server.cpp#L533 ? 20:01:09 and iirc txpool backlog have a boolean to toggle binary 20:01:19 SyntheticBird: what's the rules? It isn't JSON 20:01:42 Is the encoded binary guaranteed to not have "\"", enabling use of " \"" for termination? 20:02:07 its literally raw binary so ig there are no guarantees 20:02:12 How is that guarantee enforced when "\"" is a valid single byte <128? Is it using a base64 encoding and "\"" > 64 in ASCII? 20:02:48 Wow, it's almost like it's undocumented how to properly handle it and likely incomplete (there's presumably some binary sequence it'll just error on) 20:03:22 its epee encoding 20:03:31 You can't have raw binary in JSON and claim it's documented/a valid encoding. 20:03:40 this is a fine argument to me 20:03:54 i meant documented in the common sense (we have an entry on doc website) 20:04:02 but yes its completely ridiculous 20:04:10 It doesn't matter how you define it. It's still its own thing with questionable parser rules. Is it hit string start, switch to epee read, read till epee exhausted, then switch back to JSON? 20:04:28 probably yes 20:04:38 The fact no one can immediately answer how it handles the context switch shows this isn't documented, even if the fact it's broken is documented. 20:04:48 jberman @jberman:monero.social: that line is calling the zmq endpoint, which has its own unique serialization code. Merging this with the http rpc serialization is one of my nice to haves of the 8867 patch (eventually). Not all the endpoints can be merged to same code iirc 20:04:56 The question is how exactly it works and the exact spec it isn't broken under. 20:05:51 I'd ping jeffro256 on if you can read epee until exhausted like that, without issue. I don't recall if the outer object declares the amount of fields it contains. 20:07:17 It's a bit-wise copy of the type into a string and then the that sting is transformed using this: 20:07:19 https://github.com/monero-project/monero/blob/caa62bc9ea1c5f2ffe3ffa440ad230e1de509bfd/contrib/epee/src/parserse_base_utils.cpp#L42 20:07:51 it's not epee bin 20:07:59 Oh, so it's a custom base 250 20:08:16 that's even funnier (my bad for saying it was epee) 20:08:18 ... bass 248? 20:08:21 *base 248 20:08:55 kayabanerve @kayabanerve:monero.social: epee binary has an explicit element count for arrays 20:09:03 Or is it just escaping and it is suboptimally using all 256 characters 🤔 20:09:25 try casting a type into a string in rust like cpp challenge. 20:09:27 difficulty: impossible 20:09:29 vtnerd: I am aware it's self describing :) I was just unsure for the outer most object. 20:09:51 If the outer most object is considered an array, explicitly lengthed, ACK 20:11:09 Anyways. Remove all this, don't claim non-JSON is JSON, simplify lives of other people. We do force people to move to a new .bin but we're already practically forcing rewriting RPC callers for the new FCMP++ semantics/TX construction. 20:11:11 Or keep this forever as a headache to us to spare the headaches of... us, except it obviously doesn't spare us, and MyMonero. 20:11:30 I've spoken abundantly and will drop out now :p 20:13:43 so... are there any arguments against deprecating binary json ? 20:14:59 get_output_distribution as it currently exists will have no value to any wallets trying to create txs with fcmp's. As it currently exists, it can be deprecated once fcmp's are live, and replaced with a sane JSON endpoint with 0 issue to live services. We can implement a _v2 endpoint today that functions today and that could be the replacement for the get_output_distribution endpoi 20:15:01 nt that is dropped once fcmp's are live. The v2 endpoint could also be designed to cleanly support fetching the unified distr for fcmp's 20:15:26 Meeting done? Thanks everyone 20:15:37 and about get_txpool_backlog? 20:15:58 should it be deprecated too ? 20:18:03 I'm fine with a 12 month deprecation timeline for it. the argument not to support broken endpoints I think is sound 20:18:35 Sounds fine (another w for the crabs) 20:18:45 OpenMonero is using the binary endpoint, so it looks like it won't be affected 20:19:45 is openmonero not dead? 20:21:18 vthor_ this is a secret, never say it is dead again 20:21:37 :S 20:22:13 I was searching for it but didn't find a sign of life in the last 5years? 20:22:38 https://github.com/monero-project/monero/blob/master/contrib%2Fepee%2Fsrc%2Fparserse_base_utils.cpp#L42 20:23:12 That's the escape function, it's simply doing a backslash on key characters, allowing it to technically to be parsed, if still invalid json 20:23:41 vtnerd: autch 20:33:13 I planned to take whole monero source appart and try to remove all dependencies and modularize it, so when you want to use the code or parts you could do that without the nee to chew the whole source and drag it into the own source, but now I start to doubt a bit, how it seem a fast moving target. Am I wrong? 20:34:36 good luck, see you in 12 months in mental asylum 20:35:33 (you are right, but you'll suffer) 20:36:53 modularizing a huge cpp codebase with technical debt (that is actively moving) is absolutely insane 20:40:21 It will be easier to rewrite in Rust tbh 20:41:05 Cuprate already have it. Next step is Swift node (i'm racist of Go) 20:41:30 :D 20:42:52 sech1: rewrite in another language will be easier when you understand all, and so I thing first step would be to transform the code to better understand each piece, get it sortet out, and then reimplement in another language. 20:44:26 that's the neat part, you don't transform the code 20:44:29 s​yntheticbird: well I'm insane, so I could do insane stuff, too. What you don't like on go, was really thinking with endgoal rust/zig/go 20:44:59 oh forgot about zig 20:45:23 hm, I learn mostly through throwing the wrench into the gears and look where the parts fly :D 20:47:28 crash test approach, love it 20:51:02 I mean I do a long time security related stuff, but I'm no cryptographer, I use crypto how it's given. But after what I read in the source code, a lot of source is bloat and makes the things only more complicated, when I see alone this epee stuff, I mean it Qt/C++ so when I can do Simply QString.toHex() and have the same result then jumping through three methods and land then in an epee lib, my head starts to spin, and then I wonder 20:51:04 why not in Qt/C++ or in c (is this performance plus it realy worth it?) 22:17:29 There's no need to add qt dependency, especially since hex conversion is fairly simple. It's a bit complicated in our code because we convert to/from struct directly to/from hex - there's no intermediate step like copying to qstring 22:38:49 .merges 22:38:49 -xmr-pr- 9287 9345 9357 9380 9414 9416 9421 9425 9426 9429 9430 22:39:41 .merge+ 9384 9385 22:39:41 Added 22:41:04 .merge+ 9423 22:41:04 Added 22:47:52 Miniupnp needed? 22:48:07 https://github.com/monero-project/monero/pull/9367 22:54:08 Just use my Rust rewrite vthor_, only half /s 22:54:36 I already modularized the ZK proofs, transaction structs, wallet code, etc. 22:54:37 So each section should have minimal dependencies and clear API boundaries. 22:57:26 this one to lower the amount of log files created default 22:57:27 https://github.com/monero-project/monero/pull/9428 23:12:20 k​ayabanerve: that sounds cool, this is cuprate, or? Thought it was only substitute for monerod, so I never looked inside yet 23:16:21 Cuprate uses parts of it, but monero-serai is separate: https://github.com/serai-dex/serai/tree/develop/networks/monero 23:30:00 Thank you boog, I never looked there inside. That looks promising :) So there is monero-serai, but there is written: "monero-serai originally included wallet functionality. That has been moved to monero-wallet.", does that mean/serai-dex/serai/tree/develop/networks/monero/wallet? 23:40:28 yeah, `monero-wallet` is the crate name