09:16:16 <ofrnxmr[m]1> Using my own node over LAN... (full message at https://libera.ems.host/_matrix/media/r0/download/libera.chat/d59f68e8194b4c5d628da044055dea9d984d0b4a)
09:52:47 <ofrnxmr[m]1> It does appear to be a DNS issue (perhaps only people using custom DNS/vpn?).... (full message at https://libera.ems.host/_matrix/media/r0/download/libera.chat/7433872c18a5d6898247e40025f4e9c15fd02ddc)
09:56:28 <r4v3r23[m]> any update on the multisig audit?
09:56:45 <r4v3r23[m]> a third party going dark and holding up a release isnt a good look
10:29:31 <ofrnxmr[m]1> Onion
10:29:31 <ofrnxmr[m]1> Cake / 3m30s / 2m30s
10:29:31 <ofrnxmr[m]1> Monerujo / 3m39s / 3m30s
10:39:59 <selsta> r4v3r23[m]: currently waiting to hear back from trezor when they will release their firmware update
10:41:32 <selsta> https://github.com/monero-project/monero/pull/8299#issuecomment-1163448456
10:43:07 <selsta> ofrnxmr[m]1: https://status.torproject.org/
10:52:40 <r4v3r23[m]> <selsta> "r4v3r23: currently waiting to..." <- so v0.18 won't be dependent on multisig then? 
10:52:49 <selsta> no
10:53:29 <selsta> it would have been nice if the audit was finished before then but if it won't it also won't hold up the release for an indefinite amount of time
10:54:18 <selsta> but hardware wallets are important for end users, and that means we have to depend on the availability of external companies
10:56:30 <r4v3r23[m]> selsta: ok I see, that trezor PR has to be included in v0.18. its not just a trezor-side firmware update
10:57:00 <selsta> yes, but that trezor PR is ready and can be merged any time, the firmware update is out of our control
10:58:20 <selsta> jberman did the integration for Ledger, but they also have to do internal testing and I don't know how long that will take
10:59:09 <selsta> we notified Ledger almost 2 months ago, they did start looking at it until a couple days ago
10:59:15 <selsta> didn't*
11:01:18 <r4v3r23[m]> so realistically tag v0.18 when trezor PR gets merged and deal with MS in a .1 update or something
11:02:36 <selsta> we need to know when Trezor will be available to do the firmware update, we can't put out a release now and then put out a second release with a different fork height, that will split the network unnecessarily
11:06:49 <r4v3r23[m]> ah ok. got it. id really love to see DIY hwws take over and reduce reliance on these third parties
13:35:10 <ofrnxmr[m]1> <selsta> "ofrnxmr: https://status...." <- Fwiw, for me its been 2+mins on onions for at least 6+ months
13:35:10 <ofrnxmr[m]1> Get_info loads quicklt
13:57:30 <selsta> ofrnxmr[m]1: my test Tor tx yesterday was < 2 minutes, but I don't want to do further testing now if the Tor network itself might not be a normal speeds
16:27:25 <selsta> does this look like some broken system library? https://paste.debian.net/hidden/a57cf264/
16:28:09 <selsta> suddenly getting SIGSEGV (Address boundary error) when closing monerod and monero-wallet-cli, but it's not due to a change in monero codebase
16:34:03 <moneromooo> Looks like it could be dtor ordering.
16:35:17 <selsta> I didn't have this last week and it suddenly showed up after recompiling from scratch, no code changed
16:35:35 <moneromooo> Same compiler ?
16:36:03 <selsta> I updated my compiler 1-2 weeks ago so that's what I suspect currently
16:36:09 <moneromooo> I'd run with asan and see if anything coes up before that. Like use after free.
16:36:38 <moneromooo> You'd have to do that with openssl too though.
16:37:00 <selsta> openssl did have an update today
16:37:19 <ooo123ooo1234567> is it reproducible ?
16:37:30 <selsta> yes, happens every time on exit
16:38:20 <moneromooo> You could also try some small program like... make-mutlisig-something or monero-check-dns, see if that triggers it. If so, simplify till it stops.
16:38:47 <selsta> will try a couple things (downgrade openssl and compiler) first
16:38:48 <moneromooo> Or check with monerod --help
16:39:14 <ooo123ooo1234567> https://github.com/boostorg/asio/blob/develop/include/boost/asio/ssl/detail/impl/openssl_init.ipp#L90, it's an error within isolated class that init/free openssl for boost::asio
16:39:25 <selsta> moneromooo: also reproducible with --help
16:39:41 <moneromooo> That's good. It means you can simplify a whole lot already.
16:39:56 <ooo123ooo1234567> https://stackoverflow.com/questions/29845527/how-to-properly-uninitialize-openssl, in theory it could be something like bad order of init/free functions
16:40:19 <moneromooo> Try just "return 0;" as first line in main. If that also breaks, it's almost certainly a lgobal scope var somewhere.
16:40:31 <ooo123ooo1234567> `  static boost::asio::detail::shared_ptr<do_init> init(new do_init);` try to create isolated test.cpp that creates this shared_ptr
16:41:16 <ooo123ooo1234567> `boost::asio::asio::ssl::detail::openssl_init_base::do_init` this object
16:41:17 <ooo123ooo1234567> s/asio:://
16:42:32 <ooo123ooo1234567> is it x86_64 mac ?
16:42:47 <selsta> no, arm mac
16:43:15 <ooo123ooo1234567> https://github.com/boostorg/asio/blob/develop/include/boost/asio/ssl/detail/impl/openssl_init.ipp#L147, or just call this function within test.cpp
16:43:37 <selsta> will try
17:03:47 <selsta> so it does not show up with openssl1.1
17:04:14 <selsta> seems to be related to openssl 3.0.4 that I installed today
17:10:00 <ooo123ooo1234567> will you bisect it further to confirm and report openssl bug ?
17:10:38 <selsta> it can still be some wrong order in our code that only now started to make issues? but yes I'll try to bisect
17:11:19 <ooo123ooo1234567> selsta: still no isolated test.cpp ?
17:11:33 <selsta> not yet
17:49:34 <selsta> confirmed 3.0.3 -> no issues, 3.0.4 -> issues, trying to start bisecting now (had some weird signing issue otherwise i would be further)
17:59:24 <selsta> https://paste.debian.net/hidden/36db0e11/
18:01:34 <selsta> https://github.com/openssl/openssl/commit/f9f3096fd26ade6c188ae878a627920c0ddbd803
18:02:23 <ooo123ooo1234567> https://github.com/openssl/openssl/commit/f9f3096fd26ade6c188ae878a627920c0ddbd803, yes only this one is somehow related to init/free
18:04:45 <ooo123ooo1234567> https://github.com/openssl/openssl/blob/master/crypto/conf/conf_mod.c, this lock is unavailable there
18:04:55 <selsta> another thing i can test it compile openssl-3.0 branch to see if they fixed it already
18:05:14 <ooo123ooo1234567> * https://github.com/openssl/openssl/blob/master/crypto/conf/conf_mod.c#L485, this
18:07:17 <ooo123ooo1234567> what code do you use for testing ?
18:09:04 <selsta> monero code or openssl code?
18:09:13 <selsta> same issue with openssl master
18:11:42 <ooo123ooo1234567> do you have debug build of openssl ?
18:13:04 <ooo123ooo1234567> https://github.com/openssl/openssl/blob/master/crypto/conf/conf_mod.c#L89, it's interesting whether this was called before failure
18:13:43 <ooo123ooo1234567> hmm, was it sigsegv with nullptr ?
18:14:34 <selsta> ooo123ooo1234567: https://paste.debian.net/hidden/0360b6d9/
18:14:41 <selsta> this is with openssl debug build
18:18:49 <ooo123ooo1234567> so it's nullptr
18:18:50 <ooo123ooo1234567> check when that lock was freed
18:19:30 <ooo123ooo1234567> is it obvious how to do ?
18:20:30 <selsta> module_list_lock this pointer? will try to figure it out
18:21:10 <ooo123ooo1234567> module_lists_free, add breakpoint on this function
18:21:38 <ooo123ooo1234567> it must be called somewhere before CONF_modules_unload
18:25:01 <selsta> ooo123ooo1234567: https://gui.xmr.pm/files/step.txt say if you need more steps
18:25:39 <ooo123ooo1234567> https://github.com/openssl/openssl/blob/master/crypto/conf/conf_mod.c#L524, CONF_modules_unload -> conf_modules_finish_int -> CRYPTO_THREAD_write_lock(module_list_lock)
18:34:16 <ooo123ooo1234567> `bt`
18:35:46 <selsta> now, or after every step?
18:35:49 <ooo123ooo1234567> what is calling OPENSSL_cleanup
18:36:09 <selsta> https://paste.debian.net/hidden/1ac96635/
18:37:30 <ooo123ooo1234567> ok, set breakpoint on CONF_modules_unload and continue
18:37:36 <ooo123ooo1234567> and `bt`
18:38:32 <ooo123ooo1234567> it's expected that the order is  module_lists_free -> CONF_modules_unload, is it true ?
18:39:03 <selsta> https://paste.debian.net/hidden/d3d8772a/
18:47:03 <ooo123ooo1234567> https://github.com/chriskohlhoff/asio/issues/73, something like this
18:52:34 <ooo123ooo1234567> https://github.com/openssl/openssl/issues/18226, so there were trying to fix race condition during deinit with lock that can be already freed at that time
18:53:03 <ooo123ooo1234567> wow
18:54:10 <ooo123ooo1234567> https://github.com/openssl/openssl/commit/f9f3096fd26ade6c188ae878a627920c0ddbd803, this patch should be changed to be sure that lock is always available during deinit of openssl
18:57:14 <selsta> should I open an issue in openssl repo or is it easy to fix?
19:05:22 <ooo123ooo1234567> https://paste.debian.net/hidden/c4cc0c53/, does it fail ?
19:09:17 <ooo123ooo1234567> `g++ -g -std=c++11 /tmp/test.cpp -o test.elf -lcrypto && ./test.elf`
19:11:05 <selsta> trying to figure out the includes currently
19:11:45 <ooo123ooo1234567> it's just copy/paste from boost header
19:13:46 <selsta> https://paste.debian.net/hidden/efb4cb80/
19:15:01 <ooo123ooo1234567> add openssl libs on mac to clang args
19:15:24 <selsta> I did add -I ~/dev/openssl/include -L ~/dev/openssl
19:16:29 <selsta> ok
19:16:31 <selsta> done
19:17:29 <selsta> does not crash
19:19:12 <ooo123ooo1234567> https://paste.debian.net/hidden/eafa61d8/, `g++ -g /tmp/test.cpp -o test.elf -lssl -lcrypto && ./test.elf`
19:19:36 <ooo123ooo1234567> and this ?
19:20:48 <selsta> also no crash
19:22:43 <ooo123ooo1234567> ossl_config_modules_free, CONF_modules_unload
19:22:57 <ooo123ooo1234567> check whether these functions were called and what was the order
19:23:14 <selsta> first or second test you sent?
19:23:22 <ooo123ooo1234567> 2nd
19:24:13 <selsta> added breakpoint for both, and it just runs and exits
19:25:24 <ooo123ooo1234567> indeed
19:27:09 <ooo123ooo1234567> https://paste.debian.net/hidden/5d470218/
19:29:27 <selsta> https://paste.debian.net/hidden/b4260761/
19:30:57 <ooo123ooo1234567> different order ?
19:33:16 <selsta> what order should i change? sorry I'm confused
19:34:06 <selsta> that's the full https://paste.debian.net/hidden/58a5c5b2/
20:00:02 <ooo123ooo1234567> https://paste.debian.net/hidden/9bc64814/, does it fail ?
20:02:05 <selsta> also no fail
20:02:57 <ooo123ooo1234567> hmm
20:05:53 <ooo123ooo1234567> ossl_config_modules_free, CONF_modules_unload, both are called ?
20:08:59 <selsta> no, only CONF_modules_unload is called
20:11:03 <ooo123ooo1234567> OPENSSL_cleanup, check why this function didn't reach ossl_config_modules_free
20:11:59 <selsta>    353 	    /* If we've not been inited then no need to deinit */ related?
20:13:51 <selsta> it returns early here
20:15:02 <ooo123ooo1234567> `OPENSSL_init_ssl(0, NULL);` at the beginning in main
20:15:05 <ooo123ooo1234567> * `OPENSSL_init_ssl(0, NULL);` add it at the
20:17:13 <selsta> no, crash but both are called https://paste.debian.net/hidden/a54a6fb9/
20:23:52 <ooo123ooo1234567> https://github.com/openssl/openssl/blob/master/crypto/conf/conf_mod.c#L524, does it reach this line ?
20:28:25 <selsta> yes
20:28:51 <ooo123ooo1234567> is it 0 ?
20:29:17 <ooo123ooo1234567> `module_list_lock`
20:29:37 <selsta> it returns 1
20:30:33 <selsta> (CRYPTO_RWLOCK *) $0 = 0x0000600003904000
20:32:30 <ooo123ooo1234567> ossl_config_modules_free, module_lists_free, do_init_module_list_lock, conf_modules_finish_int; track how changed value of that lack between these calls
20:32:38 <ooo123ooo1234567> it should be 0 at least once
20:32:51 <ooo123ooo1234567> otherwise there is huge problem in understanding
20:33:13 <ooo123ooo1234567>  * ossl_config_modules_free, module_lists_free, do_init_module_list_lock, conf_modules_finish_int; track how changed value of that lock between these calls
20:35:43 <selsta> something like this? https://paste.debian.net/hidden/bc26f66d/
20:40:24 * selsta afk for a bit
21:01:03 <ooo123ooo1234567> https://paste.debian.net/hidden/fbd1d73d/, yes, but for some reason that lock isn't zero after module_lists_free 
21:05:56 <selsta> sorry i messed up
21:06:03 <selsta> fish: Job 1, './test3.elf' terminated by signal SIGSEGV (Address boundary error)
21:06:19 <ooo123ooo1234567> wow
21:06:34 <ooo123ooo1234567> what was the problem ?
21:06:40 <ooo123ooo1234567> different executable ?
21:06:55 <selsta> i added the init in the wrong place
21:07:16 <ooo123ooo1234567> can you share this ?
21:07:36 <selsta> I had it above the return
21:08:34 <ooo123ooo1234567> create issue in openssl with that POC
21:08:40 <selsta> ok will do
21:13:46 <ooo123ooo1234567> https://github.com/openssl/openssl/pull/18331, it's interesting what exactly all those reviewers checked in the patch
21:29:48 <ooo123ooo1234567> https://www.openssl.org/community/committers.html, " These are the people who can commit changes to the OpenSSL source tree, with appropriate code reviews. ", indeed, appropriate review
21:32:39 <ooo123ooo1234567> https://github.com/openssl/openssl/issues/18455, that wasn't the only problem with that commit
21:41:22 <selsta> ooo123ooo1234567: does it look good or what text should i add? https://usercontent.irccloud-cdn.com/file/I4z5Tagq/123.png
21:46:38 <ooo123ooo1234567> https://paste.debian.net/hidden/79ec99c3/, put it into test.c and add output from `gcc -fsanitize=address /tmp/test.c -o test.elf -lcrypto -lssl && ./test.elf`
21:47:01 <ooo123ooo1234567> but overall it will not help them to fix poor review process
21:51:54 <ooo123ooo1234567> hmm, why so small stack trace with clang ?
21:52:15 <selsta> that's what shows up here
21:52:28 <ooo123ooo1234567> is there gcc for mac ?
21:54:26 <ooo123ooo1234567> interesting how soon it will be fixed
21:54:48 <selsta> tried with gcc, it complains about missing -lasan
21:56:15 <selsta> whatever they can try it themselves
21:57:58 <ooo123ooo1234567> -fno-omit-frame-pointer
21:58:16 <ooo123ooo1234567> https://clang.llvm.org/docs/AddressSanitizer.html#usage, does it help ?
22:02:01 <selsta> looks the same
22:07:57 <ooo123ooo1234567> it's better to use gcc and linux then for more verbose output
22:24:53 <ooo123ooo1234567> https://github.com/openssl/openssl/issues/18049, https://github.com/openssl/openssl/issues/18578, similar reports with zero lock
22:24:54 <ooo123ooo1234567> funny