[00:11:30] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/606fa1f005ce...8a8f7483b0ee [00:11:32] [02dns] 07Universal-Omega 038a8f748 - add ownercheck TXT record for mattermost1.wikitide.net [00:27:30] PROBLEM - cp26 Puppet on cp26 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [00:48:16] PROBLEM - mattermost1 APT on mattermost1 is CRITICAL: APT CRITICAL: 28 packages available for upgrade (6 critical updates). [00:48:20] PROBLEM - mattermost1 NTP time on mattermost1 is WARNING: NTP WARNING: Offset 0.193982482 secs [00:50:05] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:51:14] PROBLEM - mattermost1 NTP time on mattermost1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:52:06] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:53:08] RECOVERY - mattermost1 NTP time on mattermost1 is OK: NTP OK: Offset 0.00767916441 secs [00:54:53] !log added mattermost1 to puppet and added ipv6 config on mattermost1 to netplan [00:55:32] What happened to logbot? [00:57:46] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/264c67ffaca8...aee69271427c [00:57:49] [02puppet] 07Universal-Omega 03aee6927 - Add mattermost1 [01:02:10] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[pg-initdb] [01:04:00] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8a8f7483b0ee...cfb6613de57b [01:04:01] [02dns] 07Universal-Omega 03cfb6613 - Use mattermost1 [01:04:10] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [01:12:13] PROBLEM - mattermost1 Puppet on mattermost1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [01:18:05] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aee69271427c...58f91918adeb [01:18:06] [02puppet] 07Universal-Omega 0358f9191 - Fix [01:20:16] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:26:36] !log [void@bots171] restart relaybot + logbot [01:26:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:59] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:14] !log added mattermost1 to puppet and added ipv6 config on mattermost1 to netplan [02:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:04] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/58f91918adeb...55ec0df28539 [02:54:06] [02puppet] 07Universal-Omega 0355ec0df - mattermost: set TeammateNameDisplay to use nickname if available [03:02:14] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209712 seconds old and 93 bytes [03:42:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:47:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:53:52] PROBLEM - index.pdcommunity.ir - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [07:04:07] PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [07:22:56] RECOVERY - index.pdcommunity.ir - Cloudflare on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Thu 28 Nov 2024 01:31:26 PM GMT +0000. [08:41:31] PROBLEM - worldless.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldless.wiki' expires in 15 day(s) (Tue 24 Sep 2024 08:16:23 AM GMT +0000). [08:41:42] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f0c1cdd25371...660acda18595 [08:41:43] [02ssl] 07WikiTideSSLBot 03660acda - Bot: Update SSL cert for worldless.wiki [08:59:07] RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000. [09:10:33] RECOVERY - worldless.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldless.wiki' will expire on Sat 07 Dec 2024 07:43:06 AM GMT +0000. [09:38:07] became sentient [09:53:37] [02puppet] 07BlankEclair opened pull request 03#3917: T12559: Workaround resources not loading for non-Cloudflare domains - 13https://github.com/miraheze/puppet/pull/3917 [09:53:45] [02puppet] 07coderabbitai[bot] commented on pull request 03#3917: T12559: Workaround resources not loading for non-Cloudflare domains - 13https://github.com/miraheze/puppet/pull/3917#issuecomment-2336621814 [10:07:01] [02puppet] 07BlankEclair opened pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918 [10:07:14] [02puppet] 07coderabbitai[bot] commented on pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918#issuecomment-2336625870 [10:08:09] [02puppet] 07RhinosF1 commented on pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918#issuecomment-2336626129 [10:08:35] [02puppet] 07coderabbitai[bot] commented on pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918#issuecomment-2336626277 [10:08:37] miraheze/puppet - BlankEclair the build passed. [10:08:42] [02puppet] 07coderabbitai[bot] commented on pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918#issuecomment-2336626322 [10:09:44] [02puppet] 07coderabbitai[bot] edited pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918 [10:10:11] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3918: Use better Firejail wrappers - 13https://github.com/miraheze/puppet/pull/3918#issuecomment-2336626322 [10:13:00] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [10:13:01] !log [alex@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [10:13:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:56] huh, guess what else is broken by https://issue-tracker.miraheze.org/T12554 [10:14:11] maybe I should create a self-signed cert mwdeploy can use [10:14:19] Yes you should [10:14:23] That would the best idea [10:14:48] We could use that for icinga too [10:39:35] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/660acda18595...0e9073de25d9 [10:39:36] [02ssl] 07redbluegreenhat 030e9073d - Add internal client certificate [11:02:47] [02puppet] 07redbluegreenhat created branch 03T12554-internal-client-certificate - 13https://github.com/miraheze/puppet [11:02:50] [02puppet] 07redbluegreenhat pushed 031 commit to 03T12554-internal-client-certificate [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c82e82b0e1fe [11:02:52] [02puppet] 07redbluegreenhat 03c82e82b - T12554: Trust the internal client certificate on mirabeta.org [11:03:17] [02puppet] 07redbluegreenhat opened pull request 03#3919: T12554: Trust the internal client certificate on mirabeta.org - 13https://github.com/miraheze/puppet/pull/3919 [11:03:23] [02puppet] 07coderabbitai[bot] commented on pull request 03#3919: T12554: Trust the internal client certificate on mirabeta.org - 13https://github.com/miraheze/puppet/pull/3919#issuecomment-2336642362 [11:03:57] [02puppet] 07redbluegreenhat closed pull request 03#3919: T12554: Trust the internal client certificate on mirabeta.org - 13https://github.com/miraheze/puppet/pull/3919 [11:03:59] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55ec0df28539...f711da8b2df6 [11:04:01] [02puppet] 07redbluegreenhat 03f711da8 - T12554: Trust the internal client certificate on mirabeta.org (#3919) [11:04:02] [02puppet] 07redbluegreenhat deleted branch 03T12554-internal-client-certificate [11:04:04] [02puppet] 07redbluegreenhat deleted branch 03T12554-internal-client-certificate - 13https://github.com/miraheze/puppet [11:18:57] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f711da8b2df6...db6c4348caf3 [11:19:00] [02puppet] 07redbluegreenhat 03db6c434 - Fix syntax [11:22:56] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/db6c4348caf3...488ca3256f38 [11:22:58] [02puppet] 07redbluegreenhat 03488ca32 - Fix file name of the internal client certificate [11:27:09] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:27:21] will fix itself in a moment [11:27:33] this is because it has the wrong filename for the internal client certificate [11:32:31] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 [11:32:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:31] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 [11:36:32] !log [alex@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [11:36:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:41:19] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/488ca3256f38...a5b3e1e51143 [11:41:20] [02puppet] 07redbluegreenhat 03a5b3e1e - Trust the internal client certificate [11:43:51] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 [11:43:52] !log [alex@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [11:43:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:10] I'm testing with a modified mwdeploy that sends the client certificate btw [11:46:38] I think it wants me to use a certificate signed by that certificate [11:47:16] a very fun thing to do in an environment where we have had no need to run an internal CA before [11:48:28] so here's what I'm going to do, I'm going to set ssl_verify_client to optional and come back to this next week [11:48:45] this fixes mwdeploy until then [11:51:07] truly, the most marvelous of bug fixes [11:55:08] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:07:57] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5b3e1e51143...07dd28b93bfe [12:07:59] [02puppet] 07redbluegreenhat 0307dd28b - Set ssl_verify_client to optional; comment out the internal client cert [12:11:39] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 [12:11:40] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 - SUCCESS in 0s [12:11:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:12:14] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.631 second response time [12:15:28] [02puppet] 07redbluegreenhat created branch 03log-client-cert - 13https://github.com/miraheze/puppet [12:15:30] [02puppet] 07redbluegreenhat pushed 031 commit to 03log-client-cert [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fe4d9d4b2b9d [12:15:32] [02puppet] 07redbluegreenhat 03fe4d9d4 - Log the Distinguished Name of the client cert who made the request [12:15:41] [02puppet] 07redbluegreenhat opened pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920 [12:15:50] [02puppet] 07coderabbitai[bot] commented on pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920#issuecomment-2336663891 [12:16:37] [02puppet] 07redbluegreenhat commented on pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920#issuecomment-2336664146 [12:16:40] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.420 second response time [12:16:41] [02puppet] 07coderabbitai[bot] commented on pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920#issuecomment-2336664183 [12:16:45] [02puppet] 07coderabbitai[bot] commented on pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920#issuecomment-2336664204 [12:17:31] [02puppet] 07coderabbitai[bot] edited pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920 [12:18:04] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920#issuecomment-2336664204 [12:18:13] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.302 second response time [12:19:34] [02puppet] 07redbluegreenhat edited pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920 [12:21:41] [02puppet] 07redbluegreenhat closed pull request 03#3920: Log the Distinguished Name of the client cert who made the request - 13https://github.com/miraheze/puppet/pull/3920 [12:21:44] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07dd28b93bfe...b1a04f9aa7f8 [12:21:47] [02puppet] 07redbluegreenhat 03b1a04f9 - Log the Distinguished Name of the client cert who made the request (#3920) [12:21:50] [02puppet] 07redbluegreenhat deleted branch 03log-client-cert [12:21:51] [02puppet] 07redbluegreenhat deleted branch 03log-client-cert - 13https://github.com/miraheze/puppet [12:23:20] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1a04f9aa7f8...0e08484ade6e [12:23:22] [02puppet] 07redbluegreenhat 030e08484 - fix [12:23:45] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.543 second response time [12:26:34] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.156 second response time [12:29:13] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.552 second response time [12:32:58] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.469 second response time [12:36:22] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.546 second response time [12:38:04] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.349 second response time [12:39:13] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.545 second response time [12:39:25] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.344 second response time [12:51:40] [02mw-config] 07BlankEclair opened pull request 03#5653: T12556: Set some image thumbnail settings for dominionstrategywiki - 13https://github.com/miraheze/mw-config/pull/5653 [12:52:46] miraheze/mw-config - BlankEclair the build passed. [13:02:36] can anyone !log anything or [13:02:44] is it admins only [13:02:55] I feel like it should be admins only [13:02:56] good question [13:02:59] someone could abuse that [13:02:59] !log claire was here :3 [13:03:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:21] oh it is still enabled for everyone [13:03:22] it blocks !log messages on discord but not on irc [13:03:24] eh, not like it's really matters [13:03:51] https://cdn.discordapp.com/attachments/808001911868489748/1282325529629818973/IMG_4927.png?ex=66def236&is=66dda0b6&hm=0e6b6f915f74c162a5eb95db4fd98dfbe946e9fe5a2b02a7cb228dc5e4ad2101& [13:03:57] Lmaaaao [13:04:01] If it becomes an issue [13:04:04] We’ll deal with it [13:04:15] if needed, we can make this channel moderated [13:04:40] We could easily set up an abuse filter [13:04:48] Or a check on the bot [13:04:52] I think there already is [13:04:57] +m (i think that's the mode) should do the trick [13:05:02] It’s been so long since I added the discord side log thoug [13:05:06] makes it so that only +v (voice) people can send messages in the channel [13:05:07] I don’t remember the code much [13:05:16] Nah [13:05:25] Its a discussion channel [13:05:36] hmm, true... [13:05:42] we can probably make the bot check if someone is +v then [13:06:33] The !log command is unrestricted on purpose [13:06:48] I believe someone tried to restrict it a while back but John said there was no need so why even do it [13:07:05] /shrug and it's never been abused to [13:07:16] Wow back in the dark I mean John ages [13:07:19] agreed--why bother if it hasn't been abused yet [13:07:40] ye olde days [13:08:49] Well it’s semi restricted here [13:10:09] !log test [13:10:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:10:23] 🤔 [13:11:12] I’d assume infra can do it [13:11:29] what 'bout chu? [13:11:37] !log vive la France [13:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:11:48] 1984 [13:11:50] Oui oui [13:11:54] 🇫🇷 [13:11:55] i'm having a [[WP:BEANS]] moment [13:12:21] Get those beans out of your nose you silly little girl!!1!1! [13:12:43] but i like snorting them :( [13:15:35] Pft [13:21:55] for a moment I thought Claire became the techteam [13:22:06] i wish i was in it [13:22:09] though i don't wish i was it [13:22:33] LMAO [13:22:46] Welcome new director of tech [13:23:43] the first thing i'd do is recommend everyone to :3 in official correspondence [13:32:05] there should be a "miraheze manages a wiki" wiki where * has managewiki perms [13:37:20] Test wiki? [13:52:43] !log [alex@test151] changed config for mirabeta.org and restarted nginx [13:52:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:53:26] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to test151 [13:53:27] !log [alex@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [13:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:26] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Testing nginx config changes, last run 30 minutes ago with 0 failures [13:55:40] I said I would do this tomorrow, but I just had to figure out right now [13:56:27] I tested changing the mirabeta.org server to only set `ssl_client_certificate /etc/ssl/localcerts/internal-client-certificate.crt;`, even that didn't work [13:57:27] so it does seem to expect me to use a cert signed by the cert in that config [13:59:42] I've turned puppet back on and I'm running it manually [14:00:01] now I'm really leaving this for next week [14:00:26] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:05:51] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.34, 5.44, 3.76 [16:07:51] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.63, 5.36, 3.93 [16:09:33] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cca33fc01abc...2bd200f6a181 [16:09:34] [02mw-config] 07waki285 032bd200f - T12553: Add edittemplateprotected to cricketnepalwiki (#5652) [16:09:35] [02mw-config] 07OAuthority closed pull request 03#5652: T12553: Add edittemplateprotected to cricketnepalwiki - 13https://github.com/miraheze/mw-config/pull/5652 [16:10:27] miraheze/mw-config - OAuthority the build passed. [16:23:22] !log [@test151] starting deploy of {'config': True} to test151 [16:23:23] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [16:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:32:32] !log [@mwtask181] starting deploy of {'config': True} to all [16:32:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:32:46] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 14s [16:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:02] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:14] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 12s [16:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:24] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.53, 6.90, 6.08 [16:42:21] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.59, 6.55, 6.06 [17:01:39] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.23, 6.96, 6.58 [17:03:35] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 5.84, 6.48, 6.45 [18:08:59] !log [salt-user@mwtask181] Renamed lamiascoilswiki to mindlessforminervawiki using renamewiki.py [18:09:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:54] !log run delbackups (bash ./delbackups.sh /home/reception/delwikis.txt /srv/mediawiki/w/maintenance/dumpBackup.php) [18:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:46:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:11:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:18:15] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e08484ade6e...565d27bbad51 [20:18:17] [02puppet] 07Universal-Omega 03565d27b - puppet: use bookworm [20:18:53] PROBLEM - graphite151 Puppet on graphite151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 23 minutes ago with 0 failures [20:18:57] PROBLEM - eventgate181 Puppet on eventgate181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 19 minutes ago with 0 failures [20:18:58] PROBLEM - jobchron171 Puppet on jobchron171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 11 minutes ago with 0 failures [20:19:01] PROBLEM - cp26 Puppet on cp26 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 26 minutes ago with 0 failures [20:19:03] PROBLEM - cloud16 Puppet on cloud16 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 19 minutes ago with 0 failures [20:19:06] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 14 minutes ago with 0 failures [20:19:06] PROBLEM - cloud17 Puppet on cloud17 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 4 minutes ago with 0 failures [20:19:07] PROBLEM - ldap171 Puppet on ldap171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 13 minutes ago with 0 failures [20:19:08] PROBLEM - db182 Puppet on db182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 18 minutes ago with 0 failures [20:19:09] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 25 minutes ago with 0 failures [20:19:10] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 23 minutes ago with 0 failures [20:19:15] PROBLEM - cloud18 Puppet on cloud18 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 8 minutes ago with 0 failures [20:19:17] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 2 minutes ago with 0 failures [20:19:28] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 24 minutes ago with 0 failures [20:19:34] PROBLEM - cp51 Puppet on cp51 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 19 minutes ago with 0 failures [20:19:35] PROBLEM - swiftobject151 Puppet on swiftobject151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 2 minutes ago with 0 failures [20:19:37] PROBLEM - db161 Puppet on db161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 22 minutes ago with 0 failures [20:19:38] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 16 minutes ago with 0 failures [20:19:38] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 20 minutes ago with 0 failures [20:19:39] PROBLEM - bast161 Puppet on bast161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 20 minutes ago with 0 failures [20:19:45] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 12 minutes ago with 0 failures [20:19:47] PROBLEM - swiftobject171 Puppet on swiftobject171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 7 minutes ago with 0 failures [20:19:47] PROBLEM - swiftobject181 Puppet on swiftobject181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 22 minutes ago with 0 failures [20:19:48] PROBLEM - cp37 Puppet on cp37 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 20 minutes ago with 0 failures [20:19:48] PROBLEM - mem161 Puppet on mem161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 21 minutes ago with 0 failures [20:19:53] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 27 minutes ago with 0 failures [20:19:59] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 17 minutes ago with 0 failures [20:20:04] PROBLEM - cp27 Puppet on cp27 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 13 minutes ago with 0 failures [20:20:11] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 12 minutes ago with 0 failures [20:20:12] PROBLEM - rdb151 Puppet on rdb151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 22 minutes ago with 0 failures [20:20:13] PROBLEM - mw161 Puppet on mw161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 11 minutes ago with 0 failures [20:20:15] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 5 minutes ago with 0 failures [20:20:16] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 2 minutes ago with 0 failures [20:20:16] PROBLEM - mem151 Puppet on mem151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 17 minutes ago with 0 failures [20:20:17] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 19 minutes ago with 0 failures [20:20:19] PROBLEM - kafka181 Puppet on kafka181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 9 minutes ago with 0 failures [20:20:20] PROBLEM - os161 Puppet on os161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 22 minutes ago with 0 failures [20:20:21] PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 26 minutes ago with 0 failures [20:20:22] PROBLEM - bast181 Puppet on bast181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 24 minutes ago with 0 failures [20:20:23] PROBLEM - swiftobject161 Puppet on swiftobject161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 25 minutes ago with 0 failures [20:20:23] PROBLEM - db181 Puppet on db181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 20 minutes ago with 0 failures [20:20:26] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 26 minutes ago with 0 failures [20:20:33] PROBLEM - graylog161 Puppet on graylog161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 21 minutes ago with 0 failures [20:20:33] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 12 minutes ago with 0 failures [20:20:47] PROBLEM - os162 Puppet on os162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 10 minutes ago with 0 failures [20:20:47] PROBLEM - db151 Puppet on db151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 5 minutes ago with 0 failures [20:20:48] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 8 minutes ago with 0 failures [20:20:51] PROBLEM - db171 Puppet on db171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 23 minutes ago with 0 failures [20:20:51] PROBLEM - cp36 Puppet on cp36 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 32 minutes ago with 0 failures [20:20:51] PROBLEM - reports171 Puppet on reports171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 24 minutes ago with 0 failures [20:20:53] PROBLEM - os151 Puppet on os151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 18 minutes ago with 0 failures [20:20:53] PROBLEM - cloud15 Puppet on cloud15 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 16 minutes ago with 0 failures [20:20:53] PROBLEM - mattermost1 Puppet on mattermost1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 9 minutes ago with 0 failures [20:20:56] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 23 minutes ago with 0 failures [20:20:58] PROBLEM - prometheus151 Puppet on prometheus151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 6 minutes ago with 0 failures [20:20:59] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:21:04] PROBLEM - swiftac171 Puppet on swiftac171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -upgrading puppet, last run 26 minutes ago with 0 failures [20:27:38] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:28:51] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:51] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:53] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:54] RECOVERY - graphite151 Puppet on graphite151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:54] RECOVERY - mattermost1 Puppet on mattermost1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:57] RECOVERY - eventgate181 Puppet on eventgate181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:57] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:58] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:28:59] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:03] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:05] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:06] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:08] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:09] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:09] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:09] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:10] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:15] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:17] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:27] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:33] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:35] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:38] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:39] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:29:41] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:45] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:46] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:47] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:47] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:52] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:53] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:29:59] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:04] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:11] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:30:12] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:13] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:15] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:15] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:30:16] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:16] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:16] RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:30:21] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:30:22] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:23] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:30:23] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:23] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:30:26] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:33] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:36] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:30:41] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:30:47] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [20:30:51] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:30:54] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:31:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:34:20] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [20:34:45]