hyc: i found the issue, randomx tests are failing

[84] Hash test 2a (compiler) ... Assertion failed: (equalsHex(hash, "639183aae1bf4c9a35884cb46b09cad9175f04efd7684e7262a0ac1c2f0b4e3f")), function operator(), file tests.cpp, line 966.

sech1: ^^

selsta I've got macos 12.0.1

i installed 13.0 so that i can test if everything monero related works, and this randomx issue showed up

is there an env var to disable randomx jit?

yes\

MONERO_RANDOMX_UMASK

MONERO_RANDOMX_UMASK=8

that's what i found

so jit is broken on macos 13?

yep

it's beta so maybe macOS itself is broken but so far everything else works

try doing make test in the randomx source tree?

that's where i got test 84 failing

i get a different hash every single time i run ./randomx-tests

that's not good....

2a sometimes passes but then it fails at 2b, other times it fails at 2a directly, it's inconsistent

try editing src/virtual_memory.cpp and make sure USE_PTHREAD_JIT_WP stays undefined

same issue

I wonder if CPU cache invalidation isn't happening.

might try explicitly calling sys_icache_invalidate() instead of __builtin__clear_cache

I feel like we've had this conversation before...

checking

then again, gcc should automatically be emitting that when __builtin__clear_cache is used

but still, worth a try

yeah I looked up this same question november 2020

nov 29 2020

xmrig calls it explicitly instead of the __builtin so it's worth a try anyway

didn't help

probably will need sech1 to get onto a Mac with OS13 to debug. sounds like an OS bug tho

I'll try to get it set up

i'll also check if xmrig has the same issue

does xmrig --bench=1M return the right hash?

you might need to also specify a --seed

"right hash" == same hash sum each time

fish: Job 1, './xmrig --bench=1M --seed "test"' terminated by signal SIGSEGV (Address boundary error)

got enough RAM?

32GB

no idea what tripped there

hm, bench defaults to seed of all zero, so try without --seed

same

I'm starting to wonder if I should just install the old OS again

and never had this problem on os12?

no

sounds like it

monero wallet sync also crashes

use after free or address boundary error

can't believe a use after free wouldn't be caught on every other os too

monero-wallet-cli(29852,0x16eaaf000) malloc: Incorrect checksum for freed object 0x137464610: probably modified after being freed.

Corrupt value: 0x0

... or a heap overrun

could run wallet sync on x86-64 with valgrind

if it always dies in same place, can compare stack traces

doesn't always die in the same place

it's... super weird

but par for the course... along with the random connection drops and other crap

hard to imagine how they can screw up a BSD based OS so badly

the weird thing is apart from monero and randomx i didn't find any issues yet, web browsers also work fine (i assume they use jit)

hm, probably, yeah

trying to use the binaries generated with depends now

in case it's some compiler error, no idea

hm, the depends build will use the OSX 11.0 SDK and its compiler

ok same

and that binary works on OS 12

so I don't see a compiler bug being likely

so i'll rollback my laptop and try to install macOS 13 on that macmini we have

selsta it really looks like cache invalidation problem. Could you run xmrig under gdb and get a callstack?

i don't think gdb really works on macos. prob need to use lldb

<selsta> "we don't use lmdb for the wallet..." <- only ringdb uses lmdb

Is it normal for zmq-pub to miss publishing mem-pool tx ?

That's unusual

The receiving script: github.com/trasherdk/node-test-snip…aster/monero/src/node-zmq-grampy.js

It shouldn't miss publishing, but keep in mind that it doesn't dump everything in mempool when you first connect. It only publishes new transactions

On stagenet, sender show txid:6226f5d745bfaa158a70ba7b3e0d8b41e71e40b3cafb9449a0e839a45d3981eb

receiver show both mem-pool: 2022-07-07 15:03:42 0.500000000000 6226f5d745bfaa158a70ba7b3e0d8b41e71e40b3cafb9449a0e839a45d3981eb 0000000000000000 0.000000000000 57Es8x:0.500000000000 0

and receiving: Height 1130735, txid <6226f5d745bfaa158a70ba7b3e0d8b41e71e40b3cafb9449a0e839a45d3981eb>, 0.500000000000, idx 0/0

But nothing received from zmq-pub

did you submit this tx to your node? zmq-pub might skip publishing transactions which are in dandelion stem phase

Yes. My node and both wallets are on the same host.

it definitely filters something: github.com/monero-project/monero/bl…note_core/cryptonote_core.cpp#L1107

The Node script is on a different host.

relay_category::legacy, not sure which transactions it is

yes, it matches only for dandelion fluff transactions

so if the node adds transaction to mempool during stem phase, it's not sent vid zmq-pub

*via

which makes sense because this is an interface for miners (dandelion transactions should be mined only after stem phase has finished)

but it should still publish them after they switch to fluff

but it doesn't and it can be considered a bug

Nah. It never arrived on the sup end. How can I ensure I get all tx's ?

you can remove "matches_category(tx_relay, relay_category::legacy)" from that line if you don't mind rebuilding monerod

sech1: run xmrig benchmark under lldb?

The wallet confirm prompt was probably more than a minute after transfer command

a minute sounds like Dandelion++ delay

selsta if you can, yes

TrasherDK[m]: did you disable dns?

disable-dns-checkpoints=1

enable-dns-blocklist=1

I meant wallet --no-dns

Probably not. Checking...

that was one thing that caused slow tx generation but not sure if it's related to what you are writong

writing

Okay, dns disabled and sessions restarted. Let's see how that goes.

This one never came on zmq-sup: 8fa62730c2c82cc45703471b085c46e771ebc244c98287497857b4bf82685f75

No show: cbd114c486f60f8ad7532a9e0327c9a86a16217799d6e5bb4c9d002131f25d00

No show: 6226f5d745bfaa158a70ba7b3e0d8b41e71e40b3cafb9449a0e839a45d3981eb

sech1: paste.debian.net/hidden/5f75873c

does not seem useful

quite useful actually

or not

one thread is in hashAndFillAes1Rx4 but it's not the thread that crashed

it's probably the same cache flushing problem

code cache flushing

it definitely crashed in JIT generated code, in one of RandomX FP instructions

`libcryptonote_basic.a(cryptonote_format_utils.cpp.o): in function cryptonote::get_pruned_transaction_hash: undefined reference to cryptonote::get_transaction_prefix_hash(cryptonote::transaction_prefix const&, crypto::hash&)`

How did I get an undefined reference in a lib to something defined in the same lib?

I just need bp_prove and hash_to_curve and I've now linked ~11 libs I don't want just trying to get this to compile. I think this is my last error

it can happen if that function is inlined by compiler

... so how do I get it to *not* be inlined?

I just ran make. Didn't do any CMAKE config

Though I am also looking for a CMAKE to disable all optional depends. I have hidapi rn and I do not want it

hmm, actually there's no such function

"cryptonote::transaction_prefix const&, crypto::hash&" with this specific signature

there is one with "(const transaction_prefix& tx, crypto::hash& h, hw::device &hwdev)" signature

ah, it's in cryptonote_format_utils_basic.cpp

Yep. Which is still in cn_basic AFAICT

I also have undefined reference *from* device

it's in cryptonote_format_utils_basic library

... ah

see src/cryptonote_basic/CMakeLists.txt

Thanks :)

I can repro TrasherDK 's issue when my daemon is in the fluff epoch, and when stem isn't working. Trasher do you see this error in your daemon: `Unable to send transaction(s) via Dandelion++ stem`?

In fluff: my tx passes through my node with `tx_relay::local` so it doesn't get pushed out over zmq, then `on_transactions_relayed` in `dandelionpp_notify` will default upgrade it to "fluff" without pushing it out over zmq

When stem is working: my tx passes through my node with `tx_relay::stem`, then once my node sees it in the network from another node, the tx gets re-added to my node's tx pool and pushed over zmq because `already_have` in `handle_incoming_txs` is false

jberman: I have only 2 different log messages:

I background mining is enabled,

I Found block <6a9e00293490fa5d

every time you've ever tried submitting a tx through your own node, you don't see the tx in zmq? It's not a sporadic thing?

It's like 2-3 transfers is OK, 1 is not, then 2-3 OK and so on.

I'm at default log_level, maybe a higher would help?

can you try compiling with `const bool fluffing = false;` and seeing if they come through 100% of the time? github.com/monero-project/monero/bl…note_protocol/levin_notify.cpp#L701

`const bool fluffing = false;` the txs should always show up in zmq.. `const bool fluffing = true;` the txs should never show up

I'll give it a try.