surprise downtime post

i first realized that it had gone down while trying to post this image so it turns out i do know exactly what happened

hmm i should post this from an instance that's working

"hey what do you do when sidekiq backs up

how do you troubleshoot that"

we're like 10 minutes behind

sidekiq keeps getting net timeout errors

what the fuck is happening

Show thread federation is busted and i have no idea why

the good news is that my keyboard should come tomorrow

because i already did the database migration, just reverting to a known good state isn't an option anymore. i have to make this update work.

it's do or die

Show thread

nope, i can't sleep. i'm up.

let's get this fucking instance fixed

alright, god damn it, glaceon dot social is staying down for now

and i won't get to work on it for about 16 hours

sorry everyone

@lynnesbian every time i try to do a database restore i think it's hanging

but then i look again

and it's just trying to restore this post

oops has been down for hours now

pro tip, don't do what i did

what i did was...

*flips through notes*

uh, update the instance. hm

what the heck why does is down

i guess something finished and briefly freed up some memory because i was able to ssh in finally

jesus christ though

Show thread

hey what the fuck is going on is throwing 500 errors and all my columns are empty

tonight's bad news: is dead for probably at least a couple days until i figure out what the fuck i'm doing

tonight's good news: i've always been disappointed that [deadname] was taken but now i own

aaaand will be dead until someone gets back to me about that because i'm absolutely stumped

hey, trying to migrate to glitch-soc, when i do `bundle exec rails mastodon:feeds:build` i get:

PG::ConnectionBad: could not translate host name "db" to address: Temporary failure in name resolution

followed by a huge stack trace. same if i do `bundle exec rails db:migrate`

anyone know what's up?

Show older

Chitter is a social network fostering a friendly, inclusive, and incredibly soft community.