08:57:11 https://twitter.com/_patrickogrady/status/1554662880242348032 08:57:26 this Solana attack is insane, especially if it's the result of an unsafe ed25519 lib 08:57:32 thank goodness we did the SUPERCOP exercise 09:25:29 Whts the value of H used in Commitment aG+bH give in x,y form in hexadecimal 09:57:07 It's in rctOps.h IIRC. 09:58:32 Oh nvm it's not. I looked. 09:59:44 Ah, in rctTypes.h. 11:05:56 According to it H is hash of G 14:18:13 Not checked, but it seems strange that’s only an hash.. G and H are EC points, so maybe an hash is applied to G serialization (here it is first additional stuff) to honor NUMS, then a function has to return the second coordinate of H (and here it is second additional stuff) 15:17:31 meeting 1.75hr 16:14:01 h 16:14:43 * 17:00:02 meeting time https://github.com/monero-project/meta/issues/724 17:00:02 1. greetings 17:00:02 hello 17:00:28 Hi! 17:00:45 hey 17:01:53 Hi 17:02:11 hello 17:02:30 hi 17:04:13 2. updates, what has everyone been working on? 17:05:04 This week I implemented bulletproofs in Rust and I will start scanning the blockchain for the bp transactions this weekend. Although my Python code works, it is tooo slow. So I thought that making another implementation in Rust would be more efficient and serai (kayabaNerve) can also profit from that. 17:05:41 Also, I finished my first reading about Seraphis and have some questions about it. But I will let them for the end of the meeting. 17:05:58 OSPEAD, with a focus on forecasting and evaluating out-of-sample risk. 17:06:15 reviewing trezor's hf PR, then planning to move over to reviewing the serialization overhaul (7999/vtnerd's alternative) 17:07:25 me: still working on legacy balance recovery for my seraphis lib (will probably be doing less hours this month, on vacation) 17:08:18 I've been working on the MAGIC grant, I've finished processing all the datasets and now hyperparameter tuning the models. If anyone has free time over the next few weeks I'd love for some constructive feedback on the paper. 17:10:18 3. discussion, anybody have comments/questions? 17:10:29 xmrack[m]: Nice! I will try to in the next 2-3 weeks. 17:10:56 xmrack: Are you ready to share some preliminary results here? 17:11:44 Yea, one second 17:12:19 I have a 100% assembly implementation of the X25519 key exchange using mulx/adcx/adox instructions. Will post some benchmarks later this week. That's probably as fast as output scanning can get with current CPUs. 17:12:31 Hello :) 17:12:39 tevador: very cool, thanks for taking that up 17:12:54 tevador: what's the difference to the supercop lib we use? 17:12:59 I implemented BP+ verification in Rust as well, sorry for being a bit late to comment 17:13:35 selsta: supercop doesn't use mulx, only legacy mul. And this is X25519, which only works on the X coordinate. 17:14:37 tevador: Fantastic. Would the code apply to Seraphis outputs (enotes) too? 17:14:56 I estimate ~30% speed-up for x86 and up to 3x speed-up for ARM64, which also has a full assembly implementation (not by me). 17:15:25 It appears that the AdaBoost random forrest is performing the best when tested on mainnet transactions, with a 19% accuray. The gradient boosted classifier also achieves similar accuracies but with very difffernt feature importances. The results may vary as my mainnet validation dataset only includes ~72 ring... (full message at https://libera.ems.host/_matrix/media/r0/download/libera.chat/61a172d1a3baf69a251633477a118328deeb21f1) 17:15:27 Rucknium[m]: the plan is to use X25519 with Seraphis 17:17:43 tevador: 🎉 17:17:55 dangerousfreedom: what are your questions about seraphis? 17:18:17 xmrack: In the machine learning field, is there a typical explanation for why two ML algorithms would have quite different feature importance rankings? 17:19:00 UkoeHB: 1) To make sure I understood, if you could boil down Seraphis to two concepts, what would them be? No ring signatures anymore (usage of linking-tags instead) and separation of this part (membership) from the rest of the transaction? 17:19:00 2) What is state of the implementation? Do we have a minimum wallet or minimum node? 17:19:00 3) Is there already a template of a transaction with all the inputs and outputs like we would see at the json representation of a tx? 17:20:01 The membership proof is still for a specified subset. It's key images which are replaced by linking tags. 17:20:30 1) separation of 'membership proof' and 17:20:33 whoops 17:20:52 "key image" is a colloquial term for a linking tag; it's the same thing 17:21:11 Yeah 17:21:47 1) separation of 'membership proof' and 'ownership proof' into two pieces; more powerful user key structures are possible with the new key image construction 17:21:57 While that is true in purpose, and it's a bit awkward to be changing terminology, I do appreciate highlighting their vastly different implementation. 17:21:58 Actually, the linking tag would be the stealth_address, right? 17:21:59 Rucknium[m]: Yes, each model uses different algorithms which separate the features to the best of their ability and correlate patterns for each of the ring positions. The way they go about doing this could easily cause a difference in feature importance across multiple models. This is mainly why I'm training a few different ones along with a neural network. 17:22:02 These would be the members? 17:22:09 I basically gave up on the linking tag terminology lol 17:22:58 Haha 17:23:06 Eh. We should encourage it 17:23:13 But yes, does take more time to do so :p 17:25:59 the kind of linking tag we have is a key image, so I think it's fine 17:26:10 xmrack: I think medium-term it would be good to estimate confidence intervals for the F1-score on the mainnet transactions since the sample size is small. The typical approach would be bootstrapping as long as the F1-score satisfies the technical regularity condition of being a _pivotal statistic_. 17:26:40 I do have a side comment on Seraphis dev/impl, further inspired by a recent thing I noticed from BP+. BP+ implemented new generators and a new transcript. While I have no idea why (as under batch verification, there should be no benefit to new generators of the same quantity/method, especially since these aren't even batched together). The new transcript is... negative however. It's the same methodology as the previous one 17:26:40 (with all the idiocy it brought), yet with a DST. While I don't mind the DST, it broke type safety because Monero keys aren't type safe :/ 17:26:55 2) the core library is mostly done, I just need to finish integrating legacy enotes and add a coinbase tx type; the core library should be well-featured enough to build a wallet from 17:26:59 Hopefully there is a standard and mathematically-proven way to do that with ML. If there is not, I will be Very Upset with the machine learning field :P 17:27:26 It also performed unnecessary operations in its initial parameters, which I can't explain. While yes, I know the BP+ code is settled, and this is -dev commentary, I do believe there is the discussion with Seraphis about ensuring we don't continue such oddities. 17:27:53 Rucknium[m]: Off the top of my head I’m not sure. But I’ll look into this 17:28:23 While UkoeHB is a great developer, and I know they have fixed a lot of issues like this in their work on a new library and new wallet I want to highlight the value of independent implementation for full review, both of theory and practice. 17:28:26 3) there is no serialization finalized - that kind of thing is up to future developers, but the main tx type is here https://github.com/UkoeHB/monero/blob/seraphis_lib/src/seraphis/txtype_squashed_v1.h 17:29:27 there are several areas where serialization can be optimized compared to the C++ representation, I'll probably have to write some things down for that 17:29:56 I think this will bring much fun because it does not derive from the CryptoNote tx type that now permeates the code from bottom to top 17:31:38 Ok, nice! Thank you for the answers! 17:31:54 UkoeHB as a quick remninder (which you probably know), the cryptonote serialization is separate from the work that jberman[m] is referencing, and is a bit more compact in binary size 17:32:00 kayabanerve[m]: I already forked the BP+ code to use the seraphis transcript system + generator factory https://github.com/UkoeHB/monero/blob/seraphis_lib/src/seraphis/bulletproofs_plus2.h 17:32:18 vtnerd: I mean serialization in general (of any kind) 17:32:25 all I have is a C++ struct 17:33:05 kayabanerve[m]: I believe switching to seraphis transcripts actually sped things up a bit since blake2b is a bit faster than keccak 17:33:13 for seraphis? a brand new one? probably makes sense to start new but man 17:33:23 yes 17:33:32 I have another point too. Maybe in resonance with what kayabaNerve said. 17:34:13 I'm not happy with the non-canonical points and scalars stored in the blockchain. Although I didn't find any bug that could definitely lead to inflation, these points and scalars can lead to wrong signatures and fungibility issues that will cause trouble for different nodes and wallets implementations, which are inevitable as the project grows. Moreover, I believe we should change the mindset that we have now which is 17:34:13 'trust in the code' to something like 'trust in the math' which is a more robust argument as math proofs are usually less prone to errors than implementations of these math proofs. 17:34:19 I know :p I'm very happy you covered both of these. I'm discussing items of this level of eccentricity in general though. While you're also working hard against it, I'm highlighting how we're individuals in our own ecosystems, and how certainty there... 17:34:41 its probably a great chance to break away from cruft, and finally get some "librarization" going for downstream projects, but hopefully it won't turn into too big of a project 17:34:52 I refuse to write this kind of disgusting mess in my library lmao https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/ringct/rctTypes.h#L335 17:35:11 The hero we need 17:35:16 yeah and `/src/ringct` is one of the better folders 17:35:40 well kaya the problem has been doing librization meant total pain with compatiability 17:35:44 Iirc you also made it so we don't need to write commitments into the bp to do verification which is great 17:36:05 but since thats being blown up it does make it easier in one sense, but the older code still needs to be present for blockchain verificaiton :/ 17:36:41 anyway I;ll stop hijacking this, ukoehb if you need seralization suggestions thats one of my few specialities 17:36:58 the interface I developed could also do tagless (like cryptonote) 17:37:06 vtnerd: This is about librarization. It's about implementation preferences of developers or decisions relevant to an ecosystem which make it hard to move out. I'm commenting independent implementation is one of the highest forms of review and enables commenting on the portability of decisions as well 17:37:06 so most compact 17:37:10 vtnerd: you may want to spearhead that effort, since I don't plan to do anything with serialization 17:37:35 once legacy stuff and the coinbase tx type is ready, I am moving on (after finishing the paper) 17:37:41 I am a bit confused. Why should existing serialization be unfit for sure? 17:37:54 ah - you'll have enough with trying to get seraphis integrated, etc. ? 17:37:59 While portability largely isn't an issue, I'd highlight the rct::key type ambiguity and custom hash to point as examples. I'm mainly concerned about the eccentric preferences which have pervaded monero's history though 17:37:59 it is a massive effort already 17:38:12 After all we serialize all kinds of things already, no? 17:38:24 the existing serialziation for cryptonote is decent actually 17:38:54 its just that the interface I wrote could handle epee/json/(most) cryptonote/msgpack 17:39:01 whereas now there are multiple in the codebase 17:39:50 there are several "tricks" in the existing tx that can't be duplicated easily with a generic library, since it reuses array length values across fields, etc., that is extremely custom 17:40:11 I think it is impossible to go from rct::key -> point with certainty. The other way round is possible. I dont know if it happens in the code. 17:40:14 and as ukoehb will point out, that code is a bit gross and hard to follow due to the custom nature 17:40:17 Making the protocol itself more strict can improve development decentralization. Hopefully there could be less reliance on the actual wallet2 code and more reliance on a protocol specification. (This goes for decoy selection and other forms of defects in tx uniformity IMHO too.) 17:41:18 With every additional wish here right now we cooly add one more person year of work :) 17:44:11 I will finish my CCS by the end of this month and I would like to continue working for the betterment of Monero as I have built in the past 6 months my tools and knowledge about Monero to do so. Specially regarding the security and cryptography behind, as I said, we should really make sure to have only canonical scalars and points stored in the bc. I will propose a new CCS this month and would like to know how I can help. 17:44:11 :) 17:44:33 if ukoehb does rewrites similar to multisig it will help out big time, regardless of any strict spec being written 17:44:57 From my side, I would like to continue working on my monero-inflation-checker (as I still have BP+ to check and also I believe I can improve the content and make more implementations in Rust) and help to implement the necessary functions for Seraphis/Jamtis. Do you think I could help you, jberman, tevador and someone else to implement the necessary stuff for Seraphis/Jamtis? How is the work structured so far? Where is the 17:44:57 TODO list of the project ? :p 17:45:20 fluffy had one somewhere a long time ago 17:45:35 doing what ukoehb is finally doing with txes was on that list 17:47:13 As I mentioned, hightlight every place in the code that now references cryptonote::transaction and mark it for possible rewrite. 17:47:16 I don't think an updated one exists 17:47:28 Then sit back and marvel at about half of all code marked ... :) 17:47:52 yeah idk anything about how tx serialization is handled 17:48:10 yeah thats the one concern I have with the seraphis proposals, its going to be a beast to integrate and will likely take longer than expected 17:48:23 figuring out the core library already takes up my entire attention 17:48:36 which isn't to say that Im against it, just that it will likely be a slog 17:48:49 I don't say UkoeHB did not do good work, maybe there really is no way around this hard brake, but boy will this bring work with it. 17:48:57 *break 17:49:01 oh no trying to get seraphis shoehorned into all parts is already a huge task 17:49:16 yeah billable hours for all! 17:49:31 For all our dozens of devs? Hurray! 17:49:42 vtnerd: it should be possible to completely deprecate wallet2 aside from wallet file migration, so hopefully that helps 17:49:49 yeah it always sounds good until you realize it just means you become overworked 17:49:57 :) 17:50:15 yes, we can move stuff to the side but it always have to be around somewhere sadly 17:50:33 But well, maybe the introductin of RingCT looked similar at start, and we could do it after all 17:50:47 or maybe we just make wallet2 forcibly sweep all coins, and then start fresh 17:51:04 Hiring freezes and layoffs have started in the tech sector. Could be a good time to do some talent scouting. 17:51:15 vtnerd: it won't be necessary, the new library can do full legacy balance recovery + spend legacy coins 17:51:18 That's the spirit. 17:51:39 or will, when I'm done working 17:52:06 ok you are ambitious. clear out your calendar lol 17:52:09 I think that alone will be a very big plus. Thank you for that. 17:52:29 Is there any discussion about changing the crypto libraries (and therefore the serialization)? What if we took a LibSodium or OpenSSL libraries? 17:55:30 dangerousfreedom: no the monero crypto library is too deeply embedded 17:55:56 Maybe that would also be quite risky and would need very, very good testing. 17:56:05 we need the raw crypto library for optimized implementations anyway 17:57:16 I believe we should consider enforcing some encodings at least to be compatible canonical points and scalars for Seraphis. As in my opinion this presents a far bigger risk than quantum computing. 17:58:12 (If we are only planning to change the library for post-quantum cryptography schemes someday) 17:58:53 vtnerd: I think it would be even more work if legacy stuff wasn't fully integrated, because there are balance recovery overlap problems around the transition. 17:59:19 you'd end up in a situation where wallets need an instantiation of both wallet2 and spwallet 17:59:48 dangerousfreedom: yes everything is already enforced to some degree 18:00:32 prime subgroup isn't enforced except where necessary for efficiency reasons, but crypto point/scalar serialization should be enforced everywhere 18:01:40 ok that's the end of the hour, so I'll call it here; thanks for attending everyone; I'm happy to answer any further questions about seraphis in this channel (as usual) 18:02:08 Thank you :) 18:04:47 dangerousfreedom the serialization for crypto points just uses ed25519 standard form. every ed25519 library uses that same format afaik, and it works just fine (albeit somewhat slow to unpack the x-coord) 18:05:09 the serialization we have to is tracking multiple public_keys, integers, etc 18:06:16 monero has used a custom implementation that is strictly ordered, such that no "tag" or "value" information is serialization - all fields must be in a specific order 18:07:06 so ASN.1, msgpack, and all kinds of binary formats are "less acceptable" because they bloat the binary size a bit more with field type/name information 18:09:17 Im not sure of a comparable standard serialization format off the top of my head, although I will search for one to help with seraphis/ukoehb 18:09:35 but we'll probaly just continue with custom, because a strictly ordered format breaks golang, javascript, and a whole bunch of other languages that assume unordered fields in their framework 18:10:19 vtnerd: Oh, thank you for the detailed explanation. 18:12:51 vtnerd: by crypto serialization I mean whether a point can be decoded from 32 bytes or if a scalar is in reduced form (sc_check()) 18:16:49 ok, so the plan was to re-use `cryptonote::serialization` for the fields? because I thought that was part of the discussion as well 18:16:55 there's nothing wrong with using it, just that so few other projects can unpack because its so linked into monero atm 18:17:43 no I'm only talking about how crypto stuff is represented in 32-byte buffers, nothing to do with saving to file or whatever 18:18:21 there are invalid 32-byte sequences which we want to detect 18:20:12 ok, I vastly misunderstood then. you think a custom serialization for this will be better? if it could unpack/detect without doing the full mult_by_group_size ? 18:20:35 I think you're the expert and should decide lol 18:21:07 all I do is add checks like this https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/tx_validators.cpp#L192 18:21:44 Storing points multiplied by 1/8 does th trick. 18:22:24 yes all the proof points and scalars are validated in the proof verifiers 18:22:30 oh then you do a mul8 immediately after unpack? which also does a clear 18:22:53 e.g. https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/sp_composition_proof.cpp#L275 18:23:57 vtnerd: no you do mul8 in the verification code so the verifier knows it is valid 18:23:58 Voilà :) 18:26:00 like here: https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/sp_composition_proof.cpp#L303 18:26:00 and here: https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/grootle.cpp#L200 18:26:00 and here: https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/bulletproofs_plus2.cpp#L953 18:28:23 "Storing points multiplied by 1/8..." <- I just want to make sure that it is true :) I mean, all the points stored are really multiplied by 1/8 and the miners reject the blocks otherwise. 18:28:57 And if it is a scalar, then it has to be canonical 18:29:05 Using these sc_checks 18:30:28 I hope it's true, we do that in, er, BP I think. sarang approved, so it's got to be right. 18:30:44 As for sc_check, it's very fast IIRC, so a check on load is fine. 18:30:59 well ok, I see what you want. I will dig into ristretto and decaf for some ideas .. 18:31:23 vtnerd: this stuff doesn't impose a requirement on the wire format 18:31:43 maybe I am misusing the word serialization (or causing confusion) 18:32:34 ristretto still requires that you point-deserialize (ge_frombytes_vartime() equivalent) to check if a point is a valid representation 18:32:36 There is a big difference between writing it in the code and verifying what is actually in the blockchain. Everyone can modify the software and create custom transactions with whatever he wants inside. 18:32:38 or at least I think I see 18:32:39 if there is a compressed canonical form, then the issue is handled immediately ... but damn lots of stress on that code being correct 18:32:46 I just think it is a really bad idea to allow that. 18:37:14 * There is a big difference between writing it in the generation code and then verify what is actually in the blockchain. Everyone can modify the software and create custom transactions with whatever he wants inside. 18:54:01 I'll once again comment ristretto is faster to decode than a decompression mul8 :p 18:54:43 So if our comment becomes to save everything inv 8, we can optimize proving AND verifying by ristretto 19:01:34 ristretto is just for the compression/decompresion stage ... ? or did I misunderstand how that works 19:02:02 because I thought it was ed25519_group->ristretto_group->compressed_format 19:02:26 and then the reverse. but there wasn't necessarily anything being done in the compression group 19:02:41 kayabanerve[m] ^ 19:03:00 btw, you need a strictly ordered serialization format for transactions at least for hashing purposes 19:03:19 ah yeah good point, cryptonote solved both issues 19:03:19 Encode/decode 19:03:48 Where encode decode includes compression 19:04:00 yeah, we should likely switch to that format then because there probably isn't a penalty 19:04:23 There is compared to raw. There just isn't compared to torsion safe raw 19:04:30 And be making it default... 19:04:38 I put all the hashing of tx content in-code instead of hashing the serialized version 19:04:46 in order to use the transcript system 19:04:58 We get the safety and ease of use of not dealing with this 19:05:02 so it does an in-place hash instead of serialize then hash? 19:05:23 hmmm probably faster I guess 19:05:25 I have nits with xmr serialization but overall it's fine, and not something I'll suggest changing 19:05:32 no https://github.com/UkoeHB/monero/blob/24ef2d6d8742abddf8e090faaba518aa42a04ba7/src/seraphis/txtype_squashed_v1.cpp#L185 19:05:52 it's faster if you already have a parsed transaction 19:05:55 If we did one thing, I'd like to remove duplicated lengths and replace some variants with fixed bytes 19:06:29 s/variants/varints/ 19:07:31 Relating to tevador's comment, protobuf isn't ordered which is stupid 19:08:28 er maybe yes if that's what you meant by in-place 19:09:31 this kind of hashing is probably fine for a Fiat-Shamir transform, but not for getting the TXID 19:17:17 tevador: Care to clarify? 19:34:54 if we had a non-ordered serialization format, then e.g. a transaction with 10 fields can have 10! ~ 3 million different representations