[00:01:11] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/25faa33e1c0d334000d598efe4aee17c4c0f6237
[00:01:11] <MacFanBot>	 02statichelp/03main 07WikiTideBot 0325faa33 Bot: Auto-update Tech namespace pages 2025-07-10 00:01:09
[00:05:01] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'CreatePageUw'} to test151
[00:05:02] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'CreatePageUw'} to test151 - SUCCESS in 1s
[00:05:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:05:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:06:43] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57366MiB (12% inode=99%);
[00:07:50] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57163MiB (12% inode=99%);
[00:07:55] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57819MiB (12% inode=99%);
[00:08:29] <MirahezeLSBot>	 !log [macfan@mwtask181] Began automatic backing up
[00:08:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:08:43] <MirahezeLSBot>	 !log [www-data@mwtask181] Began automatic backing up
[00:08:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:10:09] <MacFan4000>	 ^ just running on a single wiki, seeing if I fan figure out why there are failures
[00:10:14] <MacFan4000>	 can*
[00:12:59] <MirahezeRelay>	 <posix_memalign, replying to pskyechology> I could volunteer Strinova Wiki for it, though if something breaks I would be doing the community a disservice.
[00:13:40] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 38449MiB (17% inode=95%);
[00:21:56] <MirahezeRelay>	 <songngu.xyz, replying to pskyechology> [1/2] waking up a little brok today so, `lhmnwiki` + `100bangaiwiki` and `cgwiki` can handle them-
[00:21:56] <MirahezeRelay>	 <songngu.xyz, replying to pskyechology> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1392662082817753288/n.jpg?ex=68705922&is=686f07a2&hm=ef0630ef5c97a874940524e16377d962af4562735076ecb2be5d7678d47997e4&
[00:22:37] <MirahezeRelay>	 <songngu.xyz> RIP my water holder 2023-2025
[00:24:31] <MirahezeRelay>	 <songngu.xyz> if my wiki are up then lmk asap alr
[00:27:12] <MirahezeRelay>	 <pskyechology> probably a bit more testing before anyone gets sacrificed
[00:27:29] <MirahezeRelay>	 <posix_memalign> Maybe it's possible to set up a throwaway wiki on mirabeta, import everything from the old wiki, and see if everything works? Everything can just get deleted afterwards.
[00:28:14] <MirahezeRelay>	 <posix_memalign> Well, it's definitely possible to do the first few steps. I don't know if MH infra allows a wiki to be nuked without taking up any more space.
[00:34:10] <MirahezeRelay>	 <pskyechology> [1/2] yeah we can definitely clone/import wikis
[00:34:10] <MirahezeRelay>	 <pskyechology> [2/2] not exactly sure what you mean with the last message
[00:35:32] <MirahezeRelay>	 <posix_memalign> I'm just concerned about the wiki unnecessarily taking up space on mirabeta. It's probably not a big deal since most wikis are small.
[00:36:59] <MirahezeRelay>	 <pskyechology> [1/2] shouldn't be an issue, even with files
[00:36:59] <MirahezeRelay>	 <pskyechology> [2/2] and testing can definitely be done without them in a lot of cases
[03:09:43] <MirahezeRelay>	 <cosmicalpha, replying to pskyechology> We have one task blocking it then we can start to allow certain wikis to opt in to early upgrade/testing for 1.44.
[03:11:15] <MacFanBot>	 [02puppet] 07Universal-Omega un-readied pull request #4448: Add REL1_44 to prod (03main...03REL1_44-prod) 13https://github.com/miraheze/puppet/pull/4448
[04:03:01] <BlankEclair>	 !log curl -X DELETE https://opensearch-mw.wikitide.net/icarlywiki_{content,general}
[04:03:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:03:17] <MirahezeLSBot>	 !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=icarlywiki --startOver (END - exit=0)
[04:03:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:04:41] <MirahezeLSBot>	 !log [blankeclair@mwtask171] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=icarlywiki (END - exit=0)
[04:04:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:28:13] <MirahezeRelay>	 <songngu.xyz> {{Phorge:T13953}}
[08:28:15] <MirahezeRelay>	 <Wiki-Bot#2998> [1/2] Open: Re-index CirrusSearch for lhmnwiki and cgwiki
[08:28:16] <MirahezeRelay>	 <Wiki-Bot#2998> [2/2] <https://issue-tracker.miraheze.org/T13953>
[08:28:54] <MirahezeRelay>	 <songngu.xyz> Hope that someone are on the console and can get this task done...
[10:19:33] <MirahezeRelay>	 <rhinosf1, replying to songngu.xyz> Why do you need a reindex
[10:20:04] <MirahezeRelay>	 <songngu.xyz, replying to rhinosf1> Missing search results
[10:22:11] <MirahezeRelay>	 <rhinosf1, replying to songngu.xyz> Rename your task then, again https://xyproblem.info
[10:22:48] <MirahezeRelay>	 <rhinosf1> You want us to fix missing search results
[10:29:27] <MirahezeRelay>	 <songngu.xyz, replying to rhinosf1> renamed and added contexts
[10:30:15] <MirahezeRelay>	 <rhinosf1> Can you give an example of a page that should show but doesn't on both wikis @songngu.xyz
[10:33:17] <MirahezeRelay>	 <songngu.xyz> [1/2] I got that on `cgwiki` but currently no longer at `lhmnwiki`
[10:33:17] <MirahezeRelay>	 <songngu.xyz> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1392815938444722286/L.webm?ex=6870e86c&is=686f96ec&hm=5eb44ab21e58f8ae21e54cec99415ff150593ea86b3b53796440cd8ee743a3ea&
[10:33:25] <MirahezeRelay>	 <songngu.xyz> Will update the task soon
[10:39:50] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49825MiB (10% inode=99%);
[11:02:43] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49873MiB (10% inode=99%);
[11:27:55] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49929MiB (10% inode=99%);
[12:12:57] <MacFanBot>	 [02CreateWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/CreateWiki/commit/d82f70d2e07ff140f2e4f3312020173262a94379
[12:12:57] <MacFanBot>	 02CreateWiki/03main 07translatewiki.net 03d82f70d Localisation updates from https://translatewiki.net.
[12:12:58] <MacFanBot>	 [02ErrorPages] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ErrorPages/commit/b62f25f3809e113c2496eba0f974be7211cc542e
[12:12:58] <MacFanBot>	 02ErrorPages/03main 07translatewiki.net 03b62f25f Localisation updates from https://translatewiki.net.
[12:13:00] <MacFanBot>	 [02DiscordNotifications] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DiscordNotifications/commit/318e03cf30b971b56c759f305a49bb54ee374f05
[12:13:02] <MacFanBot>	 02DiscordNotifications/03main 07translatewiki.net 03318e03c Localisation updates from https://translatewiki.net.
[12:13:04] <MacFanBot>	 [02ImportDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ImportDump/commit/a1a330ffe646c30581d8bbcd0367af1be7c6d622
[12:13:06] <MacFanBot>	 [02GlobalNewFiles] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/GlobalNewFiles/commit/0095891b15fd7de1fcb6996fedd319cdd7aab3e6
[12:13:07] <MacFanBot>	 02GlobalNewFiles/03main 07translatewiki.net 030095891 Localisation updates from https://translatewiki.net.
[12:13:09] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/864789733c43b875dc21c8ab85e4fd7eefa6d363
[12:13:11] <MacFanBot>	 [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/cfe55fce56f65e6774ba3e9da2605af9b792a904
[12:13:13] <MacFanBot>	 02ImportDump/03main 07translatewiki.net 03a1a330f Localisation updates from https://translatewiki.net.
[12:13:15] <MacFanBot>	 [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MatomoAnalytics/commit/035aa4d8921602da395185c6511f41e55a953d0f
[12:13:17] <MacFanBot>	 [02ManageWiki] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/ManageWiki/commit/60fe93136d2b20a3a1308a010b794aefaafd9cf0
[12:13:19] <MacFanBot>	 [02RequestSSL] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RequestSSL/commit/069925b75d59280393857c52281e0b6a49a24c55
[12:13:21] <MacFanBot>	 [02IncidentReporting] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/IncidentReporting/commit/5a4d9950efefc511194a5d4c676a557580c9ca2f
[12:13:23] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/ca17ded494d90a995423d511604c05e54510730c
[12:13:25] <MacFanBot>	 [02RottenLinks] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RottenLinks/commit/0c29955d698267f005ca125c4838969548972a92
[12:13:27] <MacFanBot>	 [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/bf03940bf32c9bd7fec61a6ba02dd454a4651847
[12:13:29] <MacFanBot>	 02WikiDiscover/03main 07translatewiki.net 03bf03940 Localisation updates from https://translatewiki.net.
[12:13:30] <MacFanBot>	 02DataDump/03main 07translatewiki.net 03cfe55fc Localisation updates from https://translatewiki.net.
[12:13:32] <MacFanBot>	 02MatomoAnalytics/03main 07translatewiki.net 03035aa4d Localisation updates from https://translatewiki.net.
[12:13:33] <MacFanBot>	 02ManageWiki/03main 07translatewiki.net 0360fe931 Localisation updates from https://translatewiki.net.
[12:13:35] <MacFanBot>	 02RequestSSL/03main 07translatewiki.net 03069925b Localisation updates from https://translatewiki.net.
[12:13:36] <MacFanBot>	 02IncidentReporting/03main 07translatewiki.net 035a4d995 Localisation updates from https://translatewiki.net.
[12:13:38] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03ca17ded Localisation updates from https://translatewiki.net.
[12:13:40] <MacFanBot>	 02RottenLinks/03main 07translatewiki.net 030c29955 Localisation updates from https://translatewiki.net.
[12:13:41] <MacFanBot>	 02landing/03main 07translatewiki.net 038647897 Localisation updates from https://translatewiki.net.
[12:17:14] <miraheze-github>	 miraheze/ErrorPages - translatewiki the build passed.
[12:21:30] <miraheze-github>	 miraheze/IncidentReporting - translatewiki the build passed.
[12:21:39] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:22:03] <miraheze-github>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[12:22:20] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:22:35] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:22:50] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:22:53] <miraheze-github_>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:25:49] <miraheze-github>	 miraheze/RottenLinks - translatewiki the build passed.
[12:29:47] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:31:13] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:32:30] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:33:37] <miraheze-github>	 miraheze/DiscordNotifications - translatewiki the build passed.
[12:38:15] <miraheze-github>	 miraheze/RequestSSL - translatewiki the build passed.
[16:27:22] <icinga-miraheze>	 PROBLEM - graylog161 graylog tls port 12210 ssl cert check on graylog161 is WARNING: WARNING - Certificate 'wikitide.net' expires in 15 day(s) (Sat Jul 26 16:24:14 2025 +0000).
[20:50:01] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 59%
[20:54:01] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 20%
[21:22:37] <MirahezeRelay>	 <pskyechology> and no one is complaining about 503s? great day