[01:01:16] <WikiForgeRelay>	 <tali64, replying to nionlogos18> I think it was intended to, but its administration decided to go with WikiForge instead
[01:02:13] <WikiForgeRelay>	 <lichmaster98> Either way the two (WikiForge and WikiTide) do share some overlaping core backend services from my understanding.
[01:11:12] <WikiForgeRelay>	 <eternitymediagroup, replying to tali64> Wikitide was the original plan, but a former miraheze Steward offered to fund our wikiforge subscription
[01:23:01] <WikiForgeRelay>	 <hb1290> WF has gone down. Getting 503s and “technical difficulties” errors on AVID. Meta is loading, but with no styling whatsoever
[01:27:09] <WikiForgeRelay>	 <hb1290, replying to hb1290> @Site Reliability Engineers (neglected to ping at first)
[01:27:43] <WikiForgeRelay>	 <agentisai> cc @cosmicalpha
[01:28:07] <WikiForgeRelay>	 <agentisai> mw* is up but not receiving any traffic
[01:28:16] <CosmicAlpha>	 Looking
[01:31:53] <CosmicAlpha>	 DB caused depooling @agentisai
[01:32:07] <WikiForgeRelay>	 <agentisai> oh
[01:32:40] <CosmicAlpha>	 I forced a repool, now will try to set back to auto health check again
[01:32:54] <WikiForgeRelay>	 <agentisai> Thanks
[01:33:13] <WikiForgeRelay>	 <agentisai> My imports seem to keep killing the database
[01:33:20] <WikiForgeRelay>	 <agentisai> wonder if there's a more effective way
[01:34:34] <CosmicAlpha>	 `sudo varnishadm backend.set_health 'mw1' sick` on cp* to depool, healthy to force pooled, and auto to use healthcheck by the way, can also use `mw*` instead of mw1 for example in the future to fix varnish pool issues.
[01:35:18] <CosmicAlpha>	 sudo varnishadm backend.list to list current status also.
[01:36:09] <CosmicAlpha>	 Thats the main varnish stuff just for future reference 
[01:36:58] <WikiForgeRelay>	 <agentisai> We should log this somewhere
[01:38:55] <WikiForgeRelay>	 <notaracham> iowiki welcomes this info
[01:41:03] <CosmicAlpha>	 This can be logged publicly somewhere in public onboarding documentation I think... I wonder if we should do a GitHub pages or GitHub wiki log of some tech documentation for better onboarding and layout than MW really provides for this type of docs...
[14:13:24] <WikiForgeRelay>	 <figoarto> @Site Reliability Engineers WF has gone down again.
[14:13:54] <WikiForgeRelay>	 <ap.png> wt too 😦
[14:15:07] <WikiForgeRelay>	 <agentisai> hmm
[14:15:11] <WikiForgeRelay>	 <agentisai> database is probably down again
[14:15:47] <WikiForgeRelay>	 <ap.png> wt too
[14:15:51] <WikiForgeRelay>	 <ap.png> btw cool font
[14:16:15] <WikiForgeRelay>	 <agentisai> thanks
[14:20:25] <WikiForgeRelay>	 <thedemon08> Greeeeeeeeeeeeeat we're down again
[14:21:15] <WikiForgeRelay>	 <agentisai> I'm guessing the big wiki import is the culprit again
[14:21:26] <WikiForgeRelay>	 <thedemon08> What wiki is importing?
[14:22:48] <WikiForgeRelay>	 <ap.png> rosetta code?
[14:23:00] <WikiForgeRelay>	 <agentisai> now back
[14:26:21] <WikiForgeRelay>	 <mostafatsaad, replying to agentisai> based club penguin profile pic
[14:26:40] <WikiForgeRelay>	 <mostafatsaad> very based in fact
[14:26:53] <WikiForgeRelay>	 <ap.png> umm its turned into a Sorry! This site is experiencing technical difficulties.
[14:28:22] <WikiForgeRelay>	 <ap.png, replying to agentisai> umm it turned into a `Sorry! This site is experiencing technical difficulties.`
[14:29:59] <WikiForgeRelay>	 <thedemon08> It's working for me
[14:32:04] <WikiForgeRelay>	 <ap.png> ohp fixed now
[15:51:22] <WikiForgeRelay>	 <epic eye, replying to nionlogos18> Yup
[17:48:14] <WikiForgeRelay>	 <epic eye> [1/4] Bruh 
[17:48:14] <WikiForgeRelay>	 <epic eye> [2/4] When reporting this, please include the information below:
[17:48:15] <WikiForgeRelay>	 <epic eye> [3/4] Error 503 Backend fetch failed, forwarded for 143.202.41.252, 127.0.0.1
[17:48:15] <WikiForgeRelay>	 <epic eye> [4/4] (Varnish XID 29172906) via cp4.wikiforge.net at Tue, 08 Aug 2023 17:46:58 GMT.
[18:21:58] <WikiForgeRelay>	 <nionlogos18, replying to epic eye> aged
[18:22:10] <WikiForgeRelay>	 <nionlogos18, replying to nionlogos18> thisi mean
[20:24:41] <WikiForgeRelay>	 <thisisanswer> Alright, member from AVID here (one of your biggest clients). I'm Thisisanswer, a former mod and writer for them.
[20:25:04] <WikiForgeRelay>	 <thisisanswer> The site's kinda been collapsing as of recently.
[20:36:38] <WikiForgeRelay>	 <epic eye> I'm finding it strange that WikiForge shows this error sometimes and sometimes not
[20:37:34] <WikiForgeRelay>	 <lichmaster98> Some error conditions can self-heal
[20:37:41] <WikiForgeRelay>	 <agentisai> It's mostly just me doing a big import
[20:37:46] <WikiForgeRelay>	 <agentisai> Database is bottlenecking
[20:38:14] <WikiForgeRelay>	 <lichmaster98> 🙂 Bottlenecks are wonderful things
[20:38:39] <WikiForgeRelay>	 <agentisai> We'll probably need to add a new database server soon
[20:38:56] <WikiForgeRelay>	 <agentisai> but for now, the database only lags when there's a lot of connections (which imports tend to generate a lot of)
[20:47:03] <WikiForgeRelay>	 <raidarr, replying to thisisanswer> avid has been collateral to growing pains for the wikiforge service
[20:47:30] <WikiForgeRelay>	 <raidarr> not much we can do about it aside from try to make things for wikiforge sre relatively painless so they are able to do what they need to do efficiently when available
[20:47:47] <WikiForgeRelay>	 <raidarr> but that's mostly between avid staff and wikiforge directly
[20:51:21] <WikiForgeRelay>	 <thisisanswer> I'm getting the feeling that AVID might be too big for anyone's good. We had plan to leave Miraheze when they wanted to shut down....and then they didn't.
[20:52:04] <WikiForgeRelay>	 <thisisanswer> So in a way, we're kinda like a big fish in a small pond.
[20:55:41] <WikiForgeRelay>	 <raidarr> avid has grown and survived by the courtesy of free hosting despite it now being at the point where typically there is a paying single owner who is either loaded to some degree or is making money off the site somehow
[20:56:24] <WikiForgeRelay>	 <raidarr> there is nobody and no arrangement that avid can make right now that's much better than this, which is a compromise between truly independent hosting and a managed solution that includes service by familiar technical experts from miraheze
[20:56:53] <WikiForgeRelay>	 <raidarr> without it avid would seriously struggle to make an arrangement or I would probably end up paying more and having to put a lot more time micromanaging technical affairs I don't really have time for
[20:57:43] <WikiForgeRelay>	 <raidarr> it's been admittedly disappointing in stability for a premium service but I can still chalk it up to the growing pains of the platform
[20:59:59] <WikiForgeRelay>	 <thisisanswer> And since most members, myself included, don't have much in the way of steady income or spendable cash to donate to your servers if you do ever go premium, I doubt that AVID will be able to survive if we live on a premium service.
[21:02:32] <WikiForgeRelay>	 <raidarr> it already is premium by my backing
[21:02:55] <WikiForgeRelay>	 <raidarr> admittedly the premium part is middle of the road but then, it is affordable and doesn't require too much from me on the backend
[21:03:06] <WikiForgeRelay>	 <raidarr> part of the overhead is handled directly by agent
[21:03:23] <WikiForgeRelay>	 <raidarr> so it's been rather fortunate in finances, although more always helps and more will be needed as time goes on