00:42:38 Psylocybin42, please notice in the room message thing: "No fork help, you’re on your own" 01:20:10 rucknium thank you 01:20:14 .merge+ 9023 01:20:14 Added 07:59:55 gingeropolous, Ah okay, sorry about that! 10:29:13 hi all, does --restricted-rpc make the rpc on 18081 be the same as the restricted rpc on 18089, or is there more to it? 10:32:21 Yea 13:14:46 do people really fork cn that often that it needs to be in the message? 13:15:15 obv the answer is yes because it's there, I'm just very surprised 16:12:38 vtnerd: for the new reader serialization interface, am I correct to say that it is the reader class' job, when deserializing objects, to keep the entire blob in memory and keep track of the point of each field at each level of recursion? 16:46:56 Jeffro256: Yes. A streaming interface is almost possible too, but requires some changes to the constraints logic. 16:46:56 The fields are tracked on the stack currently. This only fails for protobuf arrays, where a field appears multiple times within an object. Not sure how to handle that at the moment tbh, writing is easier than reading protobuf. 16:47:54 I think we can persuade people against protobuf perhaps, as msgpack is just as fast and self describing and closely related to json 16:51:48 Since we're going for resource limiting and general DDoS protection, I personally think we should go with the approach which doesn't require extra allocations. In my serialization PR, I had a class which was provided for each object type which handled field ordering on behalf of that type. What was nice about that is that, since most "object"-like types we serialize are actually c 16:51:48 ++ structs, we can pass compile-time objects that handle field ordering on behalf of a type without any allocations, and the reader class itself isn't required to keep the state of serialization all the way through an object 16:54:41 If we wanted to go deeper in this respect, I've been working on a compile-time O(1) perfect hashing lib that'll let us dispatch deserialization to struct types with no allocations and only a few character compares. I haven't integrated it so idk if the extra complexity is work it, but that can be a later problem 16:54:54 s/work/worth 16:55:10 There aren't any allocations in the current design I posted? 16:55:15 > The fields are tracked on the stack currently 16:55:19 how so? 16:55:45 Variadic templates + bool 16:56:02 If you have 100 levels, deep the reader has to have an array which adds 100 elements to keep track of where fields are in the bytes, yeah? 16:56:13 Even if the field names themselves are kept on the stack 16:57:38 I'm not certain what you mean. The reader does everything in one pass. 16:57:52 There isn't a temporary object for tracking location 16:58:44 Some writers have to buffer, protobuf is a good example. You need the inner byte size length of objects before you can write the first byte of the object 17:05:14 Okay, I might've been misinterpreted something a while back and now I can't remember what I was thinking of 17:28:58 Maybe skipping a field? That's somewhat tricky because you need to remember depth an integer suffices for that afaik 17:29:03 Okay yeah for the `epee_reader`, it has a field `skip_stack_` 17:29:09 jinx 17:31:07 But you wouldn't necessarily need that if you had the interface as streaming, yeah? 17:31:53 JSON works w/o it since you have the SAX API 17:33:09 As push parser? No you don't need it because you continue parsing on the stack. I could do that here too probably, I doubt this heap allocation is _required_, just easier 17:42:18 Yeah id have to look again, but I think it's the result of the epee format. The msgpack interface for example just needs a single integer 17:44:32 And I'm not certain how rapidjson is tracking things internally. Presumably recursive/stack, but I dunno 17:55:34 So I think your point is that with a pull parser, skipping a field requires some method of driving the parse temporarily, whereas with pull/sax there's no change. That's a fair point, and is probably why msgpack exists the way it does - it's trivial to write a pull parser for it, and a major implementation is a pull parser 17:56:13 Fwiw that stack in epee is in the class itself, only one instance is needed at a time. I could probably convert to stack based, but as I said it wouldn't be as compact most likely 18:07:27 What if we made the `key` method return a `span` to the bytes of the key? Then we allow the object driving the deserialization to route the field's deserialization based on the key value? 18:10:32 It would make the object deserialization logic a bit more complicated, but you've already generalized the object deserialization logic in `wire_read::object` 18:15:44 Finding the end of the span would then require the skip logic to run for each key before returning. So it wouldnt necessarily get rid of the vector in epee_reader, and would turn every object into a two pass read 18:18:44 I see what you're saying, but that assume that objects would use the same stack-linear-field-search type logic, which isn't necessarily the case. For example, most of the "objects" we serialize are structs, of which the field names are known at compile time. We can make deserialization logic which leverages this fact so that no skipping is needed at all 18:19:19 Then for dynamic maps of type K->V, we also don't need to skip 18:21:00 This is what the current design does? Skipping is needed for forward compatibility, an unknown field is in the bytes 18:21:53 We could make it more strict, and fail at every new field, but that's now how the parser handled things historically 18:22:12 *new => unknown 18:23:12 Well yeah in that case you can skip without having to store any information, you just parse once, and leave it alone 18:24:09 So i guess you'd have to add some kind of `skip` method into the main interface as well, but you wouldn't need to store any dynamic information about it 18:25:40 (Assuming that you don't want to make a subclass for formats with self-describing field names) 18:26:20 the dynamic information is only needed in the epee format, and only exists in the reader itself (so only one is needed at a time for the entire parse). I could probably remove it, but the implementation would likely look a little funky. 18:26:52 and only max_depth() are allowed 18:27:28 now that you mentioned this, I'm going to work on more tests for epee and json, I think they could use some more 18:28:02 *max_depth() should be the max size of the vector when skipping 18:31:54 Okay now that I'm thinking about it, I do remember that you need some kind of stack for reading epee format, since the the lengths are mentioned in the beginning of the objects / arrays and there's no stop sequence like in JSON. Let me check that the `skip_stack_` is actually doing what I think it's doing 18:33:56 And if you're being forward compatible, the objects driving deserialization can't know how many fields are in the data 18:39:08 Correct. There's no stop sequence, and "inside of array" and "inside of object" look different on the wire. AND inside of array has no type information. Whereas msgpack requires a tag before every array element (making everything uniform so that depth/location is less important) 18:39:42 sounds like asn.1 but more ambiguous 18:42:27 Interestting, so is there a tag before every field in an object that says "this is an object field" 18:43:26 not exactly, but close. it says "this is an integer" or "this is a string" where every odd number element is the key 18:43:41 wen ASN.1 deserializer in wire module 18:43:49 ? 18:43:54 the big difference is arrays which always "this is an integer" or "this is a string" before each element. it helps cut down on size a bit 18:44:32 lol its probably doable, but doubt anyone downstream would want it