<k​ayabanerve:matrix.org> vtnerd: Do you have an opinion on monero-project/monero #10118? I know you've spent time with epee's encoder and also may also benefit from this feature.

huh... batch JSON-RPC, hmmmm

how would that interact with restricted limits?

I think it's better to introduce new RPCs for the use cases where people would need batch requests. Batch requests would probably fetch much more data than needed in each case, and put more load on the server

an example is docs.getmonero.org/rpc-library/monerod-rpc/#get_block

* which undocumented can take hashes/heights

* docs.getmonero.org/rpc-library/monerod-rpc/#get_block_header_by_hash

which has hash or hashes

to select one or multiple blocks at once

hashes was not documented but I was using it on my library, until it got documented recently

quite certain other methods have similar batch requests

<k​ayabanerve:matrix.org> DataHoarder: `get_block` can already take multiple items?

* get_block_header_by_hash

<k​ayabanerve:matrix.org> That solves why I was asking for batch requests.

lemme see get_block

<k​ayabanerve:matrix.org> You first specified `get_block`

bad copy paste

that's why *

<k​ayabanerve:matrix.org> sech1: I mean, we're claiming to offer a JSON-RPC 2.0 server when we don't :/ We also still have a response size limit.

it can only get single hash or height

<b​oog900:monero.social> its not even JSON 😭

block headers can be fetched by hashes list (and returns a list)

<k​ayabanerve:matrix.org> Because the parser won't decode some valid JSON, because we sometimes inline weird binary blobs, or for yet more mysterious reasons boog900?

<k​ayabanerve:matrix.org> DataHoarder: I need the transaction hashes but not the transactions.

that's on get_block indeed

get_block doesn't contain the transactions, just hashes

<b​oog900:monero.social> the binary blobs, atm its a custom format that _can_ be used as JSON most of the time

block header only has miner tx id + tx count :(

though you can probably reuse the same HTTP/TCP connection to send the multiple queries after each finishes

<b​oog900:monero.social> although it is planed to be fixed at some point

get_block.bin tbh does it for you here

<s​yntheticbird:monero.social> fun fact: `"--"` is a valid number for monerod json parser (its zero)

14:53:45 <m-relay> <k​ayabanerve:matrix.org> That solves why I was asking for batch requests.

so if requesting a list of JSON-RPC requests, these are processed sequentially right?

It'd be the same than not closing the TCP/HTTP connection and issuing a couple in sequence, ofc, you take roundtrip latency

<k​ayabanerve:matrix.org> The JSON-RPC 2.0 request allows handling them in an unordered fashion with unordered responses.

<k​ayabanerve:matrix.org> The simplest implementation for Monero would be to accept the entire request, write '[' as a response, then handle them sequentially, writing each individual response and the necessary commas, before finally writing `]'

alternatively do you just want the *transaction ids* ?

and tie to block height

<k​ayabanerve:matrix.org> I want the block.

<k​ayabanerve:matrix.org> The block is the block header and the list of transactions within the block.

<k​ayabanerve:matrix.org> If you tell me I have a way to fetch the transaction IDs alone, I'm curious, but it alone is not a sufficient solution for me.

yeah, I was mostly referring to > 14:56:04 <m-relay> <k​ayabanerve:matrix.org> DataHoarder: I need the transaction hashes but not the transactions.

for that second ... docs.getmonero.org/rpc-library/monerod-rpc/#get_outs and request every 2 global output indices

<k​ayabanerve:matrix.org> ...

you'll hit some dupes but given at minimum you need 2 outputs + coinbase id :)

and allows multiple!

<k​ayabanerve:matrix.org> I hate how reasinable that is in the current environment.

(and gets txid)

<k​ayabanerve:matrix.org> *reasonable

you get coinbase id on block headers

which also allow ranges

<k​ayabanerve:matrix.org> Right, but that will just be the outputs, not even the transactions.

via docs.getmonero.org/rpc-library/monerod-rpc/#get_o_indexesbin

that gets you indices of each coinbase outputs... then get outs for the missing indices :D

each output says the txid, not the transaction yeah

+ height

get_blocks_by_height.bin is probably the most reasonable method here

but it's not JSON-RPC

<k​ayabanerve:matrix.org> Yes, I have two separate goals:

<k​ayabanerve:matrix.org> 1) Efficiently fetch blocks, defined as the block header and list of transactions within it

<k​ayabanerve:matrix.org> 2) Efficiently fetch 'scannable blocks' (block, transactions, output indexes)

<k​ayabanerve:matrix.org> The issue with `get_blocks.bin` is it doesn't return the `prunable_hash`. `get_blocks_by_height.bin` doesn't support pruning at all and won't work with a pruned node.

<b​oog900:monero.social> monero-project/monero #10121 :D

return prunable_hash?

for the transaction?

<k​ayabanerve:matrix.org> So I have to use `get_blocks.bin`, then re-request all transactions for their prunable hashes (pointless)

get_block also just returns the in-wire tx ids

which don't have the prunable hash afaik?

<k​ayabanerve:matrix.org> OR I have to call `get_block` (JSON-RPC) and then `/get_transactions` (JSON but not JSON-RPC)

yeah, only way that I know to get transaction other hashes (prune related stuff) is via get_transactions

<k​ayabanerve:matrix.org> Yes, but I can use `/get_blocks.bin` and `/get_transactions` for the prunable hashes, downloading every transaction twice, or `get_block` and `/get_transactions`, downloading every transaction once BUT suffering from non-batch fetching of the blocks.

get_block returns the "block" which doesn't include transactions, just a list of them. so it seems you always need to hit get_transactions

does get_blocks.bin also return txs here?

<k​ayabanerve:matrix.org> `get_blocks.bin` includes all the transactions, yet sets `prunable_hash` to zeroes

aha! then that's special!

yeah it's the "complete entry"

<b​oog900:monero.social> monero-project/monero #10120

<k​ayabanerve:matrix.org> Hence why the discussion is `/get_blocks.bin` vs `get_block` + `/get_transactions`

nice timed issue :)

<k​ayabanerve:matrix.org> complete*

<k​ayabanerve:matrix.org> *prunable hash may or may not be included

whatever the person who wrote that believes to be complete :)

<k​ayabanerve:matrix.org> It is EXCEPT the RPC explicitly sets it to zero.

<k​ayabanerve:matrix.org> The DB code fetches it without issue AFAICT.

<k​ayabanerve:matrix.org> As for the timing of the issues, I've been ranting to boog900 about my rewrite of monero-oxide's RPC for a few days now.

<b​oog900:monero.social> its been real journey

<k​ayabanerve:matrix.org> It turns out writing a Monero RPC client is extremely difficult/impossible to do performantly unless you don't verify pruned transactions and shove yourself into the `get_blocks.bin` API, which is really a blockchain synchronization manager, not a way to fetch blocks?

<k​ayabanerve:matrix.org> If you request two historical blocks with monero-oxide's RPC, we will download 100 MB because Monero will give us a 100 MB response.

^ aye on that. for everything that I do with large sets of blocks I just end up with a local block id -> get_block cache

<k​ayabanerve:matrix.org> monero-project/monero #9901 is the issue for that

usually my problem is getting block ids efficiently as they depend on each other

<b​oog900:monero.social> JSON (too slow) -> `get_blocks_by_height.bin` (no pruning) -> `get_blocks.bin` (no prunable hash) -> now 🥳

<s​yntheticbird:monero.social> how about we took you three, close you up in a room for a week so that you can come up with a new core RPC layout?

<s​yntheticbird:monero.social> (unironically)

<k​ayabanerve:matrix.org> It's been merged for almost a year, it just still hasn't been included in a release. Finally the next one however

<k​ayabanerve:matrix.org> boog900: And we're back to JSON.

why not get_blocks_by_height.bin? that allows specific heights

you need to handle the case that heights change

<k​ayabanerve:matrix.org> Because it requires a full node and I want to support pruned nodes.

but allows an efficient request of desired heights

right, that doesn't run properly with pruning

I think the lack of JSON-RPC batching (even in unrestricted mode) is just a "fix" for everything else that is wrong

<k​ayabanerve:matrix.org> Actually, using `get_blocks_by_height.bin` may be optimal boog900. If we have a full node, then we get the full transactions which is needed as `get_blocks.bin` omits `prunable_hash`. If we don't have a full node, they'll be silently omitted and we won't download them twice. We can then use `/gettransactions` for pruned transactions.

<s​yntheticbird:monero.social> how about we took you three, close you up in a room for a week so that you can come up with a new core RPC layout?

just use gRPC and make everyone cry (and extreme bloat)

<k​ayabanerve:matrix.org> But it's only optimal so long as the current behavior is codified into its API.

just make sure to handle cases where the block id you fetched is different than expected :)

<k​ayabanerve:matrix.org> that's a load bearing bad design

<h​elene:unredacted.org> SyntheticBird: are you sure that you don't just wanna see three people locked up in a room?

<k​ayabanerve:matrix.org> xkcd.com/1172

<b​oog900:monero.social> full nodes will be returning txs unpruned which isn't great but yeah it is probably best for now

oh now that we are complaining about RPC, send_raw_transaction sends a transaction. It has a parameter "do_not_relay". If set to true it prevents relaying

... unless the node is bootstrapping

<k​ayabanerve:matrix.org> But again, only for as long `get_blocks_by_height.bin` silently omits pruned TXs.

then it instantly sends it out regardless if do_not_relay is false

<b​oog900:monero.social> someone needs to rewrite the RPC in Rust

<b​oog900:monero.social> /s

but only the exact same RPC behavior.

<k​ayabanerve:matrix.org> No, someone needs to rewrite the RPC in Rust but with the exact same mindset as the developer of epee

<k​ayabanerve:matrix.org> A whole new undocumented codebase with its own collection of bugs

<s​yntheticbird:monero.social> and easter eggs

I'm trying to decode non-coinbase transactions now ... yeah I feel that

<h​elene:unredacted.org> gRPC is terrible, JSON is slow and unstructured... clearly, the solution would be XML-RPC

nah, an IRC network where clients connect to

distributed in channels

<s​yntheticbird:monero.social> discord bots rpc when?

What would be nice is an automated version of RPC in binary. Currently it's a manual dupe.

<h​elene:unredacted.org> discord bot RPC over firebase

<s​yntheticbird:monero.social> epic

like decoding that for what I wanted, the outputs, was so convoluted

<s​yntheticbird:monero.social> the unrestricted endpoints would be only accessible through twitch chat

I heavily considered parsing it *backwards*

and taking just the outputs out lol

twitch chat is IRC

<s​yntheticbird:monero.social> fair

Maybe I should implement a struct annotation epee marshaler/unmarshaler for my go stuff ...

I wrote one for SWF files including bitmaps/dynamic checks, it can't be THAT much worse

:D crates.io/crates/monero-epee/0.1.0 > The best specification is available here.

(the link 404s)

<h​elene:unredacted.org> i feel the problem with the RPC api and co. is that it's designed for usages where you shouldn't be trusting the node, because it may be malicious

<b​oog900:monero.social> I think the issue is there was no design

<b​oog900:monero.social> just bodge after bodge

<h​elene:unredacted.org> i mean, that's also part of it

<k​ayabanerve:matrix.org> Hélène: The monero-oxide interface API clearly documents its behavior, and we do provide sanity/consistency checks over the data from the RPC.

<k​ayabanerve:matrix.org> Additionally, we provide (but don't implement) an API which allows you to select decoys entirely via a local database, without using a Monero daemon.

<k​ayabanerve:matrix.org> I've done my best to be very thorough and clear in what the inputs, outputs are

<p​eteryangtime:matrix.org> Is there anyone who is asking for xmr and onion developers?

<p​eteryangtime:matrix.org> 😁

<h​elene:unredacted.org> well, it makes sense that its design would be reasonable, considering you have examples of real consumers (Serai), whereas monerod didn't and just copied bitcoin core :P

<k​ayabanerve:matrix.org> Eh. The designs aren't directly related.

<k​ayabanerve:matrix.org> Serai can use a trusted full node if it wants

<k​ayabanerve:matrix.org> DataHoarder: Ugh, did I get the link wrong? It should be to jeffro256's Rust epee lib which has a markdown file detailing epee

commit probably gone?

links to github.com/jeffro256/serde_epee/tre…58c88ceb2531de17PORTABLE_STORAGE.md

oh

missing slash :)

<k​ayabanerve:matrix.org> FWIW, the code in that crate there is now monero-oxide/monero-oxide #90

yeah, added slash, works now

<k​ayabanerve:matrix.org> It's been further improved. It now supports full traversal of the object while maintaining a lack of allocating.

<k​ayabanerve:matrix.org> I want to do the same for JSON as well but boog900 won't let me >:(

<k​ayabanerve:matrix.org> - wants to build a monero node

<k​ayabanerve:matrix.org> - doesn't want to rewrite basic functionality from scratch

<k​ayabanerve:matrix.org> Make it make sense.

3 KiB stack for the intermediate state?

<k​ayabanerve:matrix.org> Now 1 KB.

<k​ayabanerve:matrix.org> It was only 3 KB when I supported non-single-pass traversal, allowing you to enter _and exit_ objects.

<k​ayabanerve:matrix.org> Now you can only enter and restart.

yeah to reduce heap allocs I went as far as remove some io.Reader interfaces on go stuff (or bringing their sha3 hasher out of tree to prevent it from using interfaces)

now I can do the full p2pool consensus / verification on stack

<k​ayabanerve:matrix.org> It's a neat lib, even if just for the circlejerk of not allocating while offering a key-value API.

<v​tnerd:monero.social> I'm not sure pipelining would help lws much, but making zmq msgpack _finally_ would. Or else lws could switch from zmq to http binary I guess

<v​tnerd:monero.social> The problem with the array would be the responses over http would have to be sent in one shot. That array mode is probably better for a custom TCP protocol instead of http 1.1

hi

the CI tests of my DoT PR seem to fail, specifically the ones that resolve stuff over DNS

well, Quad9's regular DNS server

but i tested the unit tests on my computer, and they all passed

<k​ayabanerve:matrix.org> I mean, is there a real distinction between a bunch of small objects and one large object? The RPC server puts both in memory before finally sending it down the wire vtnerd

<v​tnerd:monero.social> I’m not sure how that relates to my comments. The pipelining thing should work better with a custom tcp protocol because you can send responses as they are completed

<k​ayabanerve:matrix.org> You said the problem would be. I'm saying I'm unsure it's a problem as it wouldn't be _worse_, right?

<k​ayabanerve:matrix.org> Or did you mean the problem preventing it from benefiting lws?

<h​elene:unredacted.org> Cindy: don't hardcode quad9, you might have better luck with FDN's DNS resolver (also available over DoT) or Wikimedia's

the CI hardcodes quad9

over regular DNS

that's what i'm talking about, and it's failing my PR

Cindy: we can change it to 1.1.1.1

you can do it in your PR, but ideally as a separate commit

i'll change the DNS_PUBLIC env var in the CI workflow to 1.1.1.1

and see if that passes or fails the DNS resolver tests

also btw, there is a global settings in monerod or the wallet porgrams to store the option to use DoT?

i don't want to use the environment variables because it seems way less flexible

monerod does not really have global settings

you can use a config file

i just mean arguments

command arguments

jeffro256: please also open 10123 against release

<o​frnxmr:xmr.mx> [@jeffro256:monero.social](https://matrix.to/#/@jeffro256:monero.social) i caught that a few weeks ago, but could only get it to segfault on one device. I typed up the issue bur never submit it. Glad to see the segfault could be reproduced

i switched to 1.1.1.1 (over TCP) and the CI still failed

i don't actually know why it can resolve domains in my machine, but not on the CI

.merges

Merge queue empty

<j​effro256:monero.social> @selsta: the bug fixed in 10123 isn't present in the release branch because 8703 wasn't PR'ed to the release branch

<v​tnerd:monero.social> Adding multiple requests wouldn't be worse, but may have minimal gains when used over http 1.1.

<k​ayabanerve:matrix.org> Fair. For me, it's about minimizing latency by making one trip, not one hundred, without making multiple connections.

it turns out that the TLS CA certificate bundle that was hardcoded in the code

the path to the bundle (/etc/ssl/cert.pem) didn't exist in the CI's filesystem

so unbound couldn't initialize properly and caused the CI to fail

so i had to switch it out with a path given by OpenSSL's X509_get_default_cert_file

<o​frnxmr:xmr.mx> <C​indy> the path to the bundle (/etc/ssl/cert.pem) didn't exist in the CI's filesystem

<o​frnxmr:xmr.mx> wont this also not exist on, say, windows?

true

the hardcoded path was a dumb mistake

i should have used OpenSSL's functions

and not guess for ONE platform

<v​tnerd:monero.social> If you've got that much data to transfer, typically it's better to overlap requests with local processing instead of pipelining. That's the typical case for monero - request next set of blocks while doing work on current set. There may be other cases where pipelining is more beneficial, but those seem to be less prevalent

<v​tnerd:monero.social> If http 1.1 allowed real pipelining, then this changes because you could requests multiple blocks (but kind of need them in order, so it's still tricky)

<k​ayabanerve:matrix.org> While an app SHOULD avoid observing this latency by so pipelining, that doesn't change my offering this amount of latency is unnecessary and I can do better by not having to make sequential HTTP requests.

<k​ayabanerve:matrix.org> But heard you're unconvinced and uninterested :)