[02:33:25] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [02:33:28] [02puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/75aeca2a7a93 [02:33:30] [02puppet] 07The-Voidwalker 0375aeca2 - wikitide-backup update find_backups [02:33:32] [02puppet] 07The-Voidwalker opened pull request 03#3869: wikitide-backup update find_backups - 13https://github.com/miraheze/puppet/pull/3869 [02:37:50] [02puppet] 07coderabbitai[bot] commented on pull request 03#3869: wikitide-backup update find_backups - 13https://github.com/miraheze/puppet/pull/3869#issuecomment-2177429693 [03:00:26] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 25 seconds old and 0 bytes [03:00:43] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 42 seconds old and 0 bytes [04:17:17] [02ssl] 07BlankEclair opened pull request 03#784: T12226: Redirect pizzatower.miraheze.org to pizzatower.wiki - 13https://github.com/miraheze/ssl/pull/784 [04:17:23] [02ssl] 07coderabbitai[bot] commented on pull request 03#784: T12226: Redirect pizzatower.miraheze.org to pizzatower.wiki - 13https://github.com/miraheze/ssl/pull/784#issuecomment-2177627566 [04:19:57] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#784: T12226: Redirect pizzatower.miraheze.org to pizzatower.wiki - 13https://github.com/miraheze/ssl/pull/784#issuecomment-2177627566 [04:29:07] [02puppet] 07The-Voidwalker closed pull request 03#3869: wikitide-backup update find_backups - 13https://github.com/miraheze/puppet/pull/3869 [04:29:08] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [04:29:09] [02puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0dbf3c460ab2...6cee0ed0997f [04:29:12] [02puppet] 07The-Voidwalker 036cee0ed - wikitide-backup update find_backups (#3869) [04:29:13] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [04:36:23] [02ssl] 07The-Voidwalker closed pull request 03#784: T12226: Redirect pizzatower.miraheze.org to pizzatower.wiki - 13https://github.com/miraheze/ssl/pull/784 [04:36:25] [02ssl] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c8be7f33d9fc...637f715314c7 [04:36:26] [02ssl] 07BlankEclair 03637f715 - T12226: Redirect pizzatower.miraheze.org to pizzatower.wiki (#784) [04:37:49] PROBLEM - wiki.grab-tools.live - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.grab-tools.live All nameservers failed to answer the query. [04:50:51] !log [void@puppet181] Upgraded packages composer on matomo151 [04:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:19] !log [void@puppet181] Upgraded packages composer on mwtask171 [04:52:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:38] !log [void@puppet181] Upgraded packages composer on mwtask181 [04:52:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:50] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [04:52:58] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [04:53:09] !log [void@puppet181] Upgraded packages composer on reports171 [04:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:38] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [04:54:04] !log [void@puppet181] Upgraded packages composer on test151 [04:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:54:14] RECOVERY - reports171 APT on reports171 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [04:54:48] RECOVERY - test151 APT on test151 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [05:04:59] !log [void@test151] created account for self (Void) [05:05:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:30] RECOVERY - wiki.grab-tools.live - reverse DNS on sslhost is OK: SSL OK - wiki.grab-tools.live reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:15:11] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:17:27] !log [void@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all [05:17:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:17:38] !log [void@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 11s [05:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:45] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:17:04] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:44:45] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:40:08] I was going through the ssl repo and I noticed the bot last renewed https://github.com/miraheze/ssl/blob/master/certificates/fanon.polandballwiki.com.crt 4 months ago [09:40:15] which basically means that cert has expired [09:40:32] https://fanon.polandballwiki.com/wiki/Main [09:40:43] "The origin web server does not have a valid SSL certificate." [09:43:23] turns out Cloudflare cares about that [09:46:46] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:49:08] RhinosF1: Can you somehow make CloudFlare serve a redirect on requests to .well-known pointing towards wikitide.net (https://github.com/miraheze/puppet/commit/0dbf3c460ab28821def53998bb1b18517d2b9747) [09:49:29] this is needed to unbreak the renew script so I can at least run it manually [10:02:50] https://issue-tracker.miraheze.org/T12227 [10:08:41] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.85, 10.91, 4.86 [10:10:41] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 2.77, 7.66, 4.40 [10:14:45] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:01:21] Orange_Star: when does it expire [11:01:23] https://issue-tracker.miraheze.org/T12227#244700 [11:03:38] localhost:~/compare$ openssl x509 -in fanon.polandballwiki.com.crt -out - -text | grep "Not After :" [11:03:47] Not After : May 15 15:29:06 2024 GMT [11:09:26] Orange_Star: I need to work out how it works for saas [11:09:40] Cause the origin server certs don't include custom hostnames [12:04:05] PROBLEM - steem.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'steem.wiki' expires in 15 day(s) (Fri 05 Jul 2024 11:40:26 AM GMT +0000). [12:04:18] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/637f715314c7...e04cd535f243 [12:04:21] [02ssl] 07WikiTideSSLBot 03e04cd53 - Bot: Update SSL cert for steem.wiki [12:09:45] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:16:56] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:32:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:33:10] RECOVERY - steem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steem.wiki' will expire on Tue 17 Sep 2024 11:04:11 AM GMT +0000. [12:39:10] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:52:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:35:51] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [14:36:03] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.016 second response time [14:37:01] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [14:39:01] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.032 second response time [14:39:51] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [14:40:03] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19091 bytes in 0.067 second response time [15:26:47] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'} [15:29:04] Orange_Star: I have asked the question about origin certs and SaaS [15:29:15] To see if we only need one for Miraheze.org [15:29:28] I need to see how origin pull works [15:29:43] nice [15:31:36] For SaaS [15:32:03] If it only needs Miraheze.org and *.miraheze.org then it's fine [15:35:59] https://issue-tracker.miraheze.org/project/view/55/ huh [15:36:09] no-one has used this since 2022 [15:36:18] time to click archive [16:44:10] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o