08:44:11 <ofrnxmr> pigeons looks like the homeserver is down
08:44:22 <DataHoarder> yeah, rip
09:17:52 <plowsof> strange https://matrix.monero.social/
09:19:29 <plowsof> DataHoarder a notice ^
09:19:45 <DataHoarder> notices get posted for your eyes only :)
09:19:53 <DataHoarder> "context deadline exceeded" is timeout
09:22:33 <DataHoarder> it fails on my own element too
09:25:35 <DataHoarder> stuff on https://matrix.monero.social:8448/_matrix/key/v2/server answers but that's not the endpoint that element tries either
09:50:18 <DataHoarder> weird https://matrix.monero.social/_matrix/client/versions loads on browser tab, but same page doesn't load under element.io on same browser
09:51:52 <DataHoarder> I think the DB is borked plowsof, sending any request with an Authorization header (for auth) makes it timeout
21:36:20 <pigeons> I didn't have this open, I see the report of matrix down. I just tested talking in a room that is on it and got replies, and tested various other things and they all seem to work so I wasn't aware its not working. I'll take a closer look now and figure it out and add an alert for whatever it ends up being and let you know
21:44:23 <plowsof> DataHoarder 
21:45:01 <DataHoarder> endpoints still timeout
21:47:12 <DataHoarder> time=2026-08-24T23:40:21.172+02:00 level=ERROR msg="Could not join matrix room" server=matrix room=#p2pool-log:monero.social err="request error: Post \"https://matrix.monero.social/_matrix/client/v3/join/%23p2pool-log:monero.social?via=monero.social\": context deadline exceeded"
21:47:39 <DataHoarder> any post to the matrix endpoints with a Bearer token for the accounts timeout
21:48:01 <DataHoarder> replicated via curl as well. stuff without the Authorization header return instantly fine
21:49:28 <DataHoarder> same issues testing from other network perspectives on monero.social user, the homeserver answers at some point but operations still timeout with 30s+ per response
21:51:45 <pigeons> Thanks, working on it
22:08:07 <pigeons> I think a bad disk, checking backups before pulling it
22:10:08 <DataHoarder> bad IO is consistent with sluggish DB and checking authenticated calls :)
22:10:22 <DataHoarder> or almost full disk, would be my other non-exotic guess
22:12:35 <pigeons> try it now. Opening a ticket for the disk replacement and adding more alerting so disk errors and authenticated user matrix errors would notify
22:12:48 <DataHoarder> it instantly worked on its own
22:13:03 <DataHoarder> seems a bit slow but returns in 1-2s instead of ms, which is doable
22:13:13 <DataHoarder> probably load due to clients catching up
22:13:22 <pigeons> ok i'll continue to check it out but at least we have something
22:14:12 <DataHoarder> > panic: M_UNKNOWN (HTTP 502): Failed to fetch alias
22:14:28 <DataHoarder> some backend errors now from your side, probably partial timeouts
22:24:11 <DataHoarder> seems bridge finally joined after getting happy with timeouts
22:24:52 <DataHoarder> transactions still slow, so messages still are being dropped
22:31:23 <DataHoarder> timeouts are back pigeons  https://irc.gammaspectra.live/cb315379b2b5698c/image.png 
22:47:38 <br-m> <ofrnxmr:xmr.mx> What happened to dh messages
22:48:00 <br-m> <ofrnxmr:xmr.mx> And my test
22:48:28 <br-m> <jpk68:matrix.org> If you're talking about DMs, I don't have a convo with your monero.social account
22:48:34 <br-m> <ofrnxmr:xmr.mx> Im talking about my convos, groupa etx
22:48:53 <br-m> <ofrnxmr:xmr.mx> Nothing loads, everything shows "yesterday"
23:19:54 <DataHoarder[m]> it's still catching up
23:20:21 <DataHoarder[m]> some rooms are now receiving messages from yesterday via federation
23:23:06 <DataHoarder[m]> also messages from today show before the messages from yesterday