Psylocybin42, please notice in the room message thing: "No fork help, you’re on your own"

rucknium thank you

.merge+ 9023

Added

gingeropolous, Ah okay, sorry about that!

hi all, does --restricted-rpc make the rpc on 18081 be the same as the restricted rpc on 18089, or is there more to it?

<o​frnxmr:monero.social> Yea

do people really fork cn that often that it needs to be in the message?

obv the answer is yes because it's there, I'm just very surprised

<j​effro256:monero.social> vtnerd: for the new reader serialization interface, am I correct to say that it is the reader class' job, when deserializing objects, to keep the entire blob in memory and keep track of the point of each field at each level of recursion?

<v​tnerd:monero.social> Jeffro256: Yes. A streaming interface is almost possible too, but requires some changes to the constraints logic.

<v​tnerd:monero.social> The fields are tracked on the stack currently. This only fails for protobuf arrays, where a field appears multiple times within an object. Not sure how to handle that at the moment tbh, writing is easier than reading protobuf.

<v​tnerd:monero.social> I think we can persuade people against protobuf perhaps, as msgpack is just as fast and self describing and closely related to json

<j​effro256:monero.social> Since we're going for resource limiting and general DDoS protection, I personally think we should go with the approach which doesn't require extra allocations. In my serialization PR, I had a class which was provided for each object type which handled field ordering on behalf of that type. What was nice about that is that, since most "object"-like types we serialize are actually c<clipped message>

<j​effro256:monero.social> ++ structs, we can pass compile-time objects that handle field ordering on behalf of a type without any allocations, and the reader class itself isn't required to keep the state of serialization all the way through an object

<j​effro256:monero.social> If we wanted to go deeper in this respect, I've been working on a compile-time O(1) perfect hashing lib that'll let us dispatch deserialization to struct types with no allocations and only a few character compares. I haven't integrated it so idk if the extra complexity is work it, but that can be a later problem

<j​effro256:monero.social> s/work/worth

<v​tnerd:monero.social> There aren't any allocations in the current design I posted?

<j​effro256:monero.social> > The fields are tracked on the stack currently

<j​effro256:monero.social> how so?

<v​tnerd:monero.social> Variadic templates + bool

<j​effro256:monero.social> If you have 100 levels, deep the reader has to have an array which adds 100 elements to keep track of where fields are in the bytes, yeah?

<j​effro256:monero.social> Even if the field names themselves are kept on the stack

<v​tnerd:monero.social> I'm not certain what you mean. The reader does everything in one pass.

<v​tnerd:monero.social> There isn't a temporary object for tracking location

<v​tnerd:monero.social> Some writers have to buffer, protobuf is a good example. You need the inner byte size length of objects before you can write the first byte of the object

<j​effro256:monero.social> Okay, I might've been misinterpreted something a while back and now I can't remember what I was thinking of

<v​tnerd:monero.social> Maybe skipping a field? That's somewhat tricky because you need to remember depth an integer suffices for that afaik

<j​effro256:monero.social> Okay yeah for the `epee_reader`, it has a field `skip_stack_`

<j​effro256:monero.social> jinx

<j​effro256:monero.social> But you wouldn't necessarily need that if you had the interface as streaming, yeah?

<j​effro256:monero.social> JSON works w/o it since you have the SAX API

<v​tnerd:monero.social> As push parser? No you don't need it because you continue parsing on the stack. I could do that here too probably, I doubt this heap allocation is _required_, just easier

<v​tnerd:monero.social> Yeah id have to look again, but I think it's the result of the epee format. The msgpack interface for example just needs a single integer

<v​tnerd:monero.social> And I'm not certain how rapidjson is tracking things internally. Presumably recursive/stack, but I dunno

<v​tnerd:monero.social> So I think your point is that with a pull parser, skipping a field requires some method of driving the parse temporarily, whereas with pull/sax there's no change. That's a fair point, and is probably why msgpack exists the way it does - it's trivial to write a pull parser for it, and a major implementation is a pull parser

<v​tnerd:monero.social> Fwiw that stack in epee is in the class itself, only one instance is needed at a time. I could probably convert to stack based, but as I said it wouldn't be as compact most likely

<j​effro256:monero.social> What if we made the `key` method return a `span` to the bytes of the key? Then we allow the object driving the deserialization to route the field's deserialization based on the key value?

<j​effro256:monero.social> It would make the object deserialization logic a bit more complicated, but you've already generalized the object deserialization logic in `wire_read::object`

<v​tnerd:monero.social> Finding the end of the span would then require the skip logic to run for each key before returning. So it wouldnt necessarily get rid of the vector in epee_reader, and would turn every object into a two pass read

<j​effro256:monero.social> I see what you're saying, but that assume that objects would use the same stack-linear-field-search type logic, which isn't necessarily the case. For example, most of the "objects" we serialize are structs, of which the field names are known at compile time. We can make deserialization logic which leverages this fact so that no skipping is needed at all

<j​effro256:monero.social> Then for dynamic maps of type K->V, we also don't need to skip

<v​tnerd:monero.social> This is what the current design does? Skipping is needed for forward compatibility, an unknown field is in the bytes

<v​tnerd:monero.social> We could make it more strict, and fail at every new field, but that's now how the parser handled things historically

<v​tnerd:monero.social> *new => unknown

<j​effro256:monero.social> Well yeah in that case you can skip without having to store any information, you just parse once, and leave it alone

<j​effro256:monero.social> So i guess you'd have to add some kind of `skip` method into the main interface as well, but you wouldn't need to store any dynamic information about it

<j​effro256:monero.social> (Assuming that you don't want to make a subclass for formats with self-describing field names)

<v​tnerd:monero.social> the dynamic information is only needed in the epee format, and only exists in the reader itself (so only one is needed at a time for the entire parse). I could probably remove it, but the implementation would likely look a little funky.

<v​tnerd:monero.social> and only max_depth() are allowed

<v​tnerd:monero.social> now that you mentioned this, I'm going to work on more tests for epee and json, I think they could use some more

<v​tnerd:monero.social> *max_depth() should be the max size of the vector when skipping

<j​effro256:monero.social> Okay now that I'm thinking about it, I do remember that you need some kind of stack for reading epee format, since the the lengths are mentioned in the beginning of the objects / arrays and there's no stop sequence like in JSON. Let me check that the `skip_stack_` is actually doing what I think it's doing

<j​effro256:monero.social> And if you're being forward compatible, the objects driving deserialization can't know how many fields are in the data

<v​tnerd:monero.social> Correct. There's no stop sequence, and "inside of array" and "inside of object" look different on the wire. AND inside of array has no type information. Whereas msgpack requires a tag before every array element (making everything uniform so that depth/location is less important)

sounds like asn.1 but more ambiguous

<j​effro256:monero.social> Interestting, so is there a tag before every field in an object that says "this is an object field"

<v​tnerd:monero.social> not exactly, but close. it says "this is an integer" or "this is a string" where every odd number element is the key

<j​effro256:monero.social> wen ASN.1 deserializer in wire module

<j​effro256:monero.social> ?

<v​tnerd:monero.social> the big difference is arrays which always "this is an integer" or "this is a string" before each element. it helps cut down on size a bit

<v​tnerd:monero.social> lol its probably doable, but doubt anyone downstream would want it