[00:14:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/457eeab61fa9...1896754bf9e5 [00:14:53] [02ssl] 07WikiTideSSLBot 031896754 - Bot: Update SSL cert for lebork.info.pl [00:39:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1896754bf9e5...f012af78b0a2 [00:39:21] [02ssl] 07WikiTideSSLBot 03f012af7 - Update certificate for lebork.info.pl [00:51:36] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Tue 16 Jul 2024 12:27:30 AM GMT +0000). [00:58:07] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Tue 16 Jul 2024 12:27:30 AM GMT +0000). [00:58:22] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Tue 16 Jul 2024 12:27:30 AM GMT +0000). [00:58:28] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f012af78b0a2...4c92433c7089 [00:58:31] [02ssl] 07WikiTideSSLBot 034c92433 - Bot: Update SSL cert for www.thesimswiki.com [01:20:11] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000. [01:27:13] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000. [01:27:45] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 27 Sep 2024 11:58:22 PM GMT +0000. [01:29:15] PROBLEM - rockrevolution.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rockrevolution.wiki' expires in 15 day(s) (Tue 16 Jul 2024 12:57:51 AM GMT +0000). [01:29:27] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c92433c7089...88c7474cc302 [01:29:30] [02ssl] 07WikiTideSSLBot 0388c7474 - Bot: Update SSL cert for rockrevolution.wiki [01:59:07] RECOVERY - rockrevolution.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Sat 28 Sep 2024 12:29:21 AM GMT +0000. [04:08:21] [02mw-config] 07BlankEclair opened pull request 03#5596: T12271: Add $wgUseLegacyMediaStyles - 13https://github.com/miraheze/mw-config/pull/5596 [04:09:28] miraheze/mw-config - BlankEclair the build passed. [05:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:24:01] PROBLEM - holocron.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holocron.net' expires in 15 day(s) (Tue 16 Jul 2024 05:01:14 AM GMT +0000). [05:24:12] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/88c7474cc302...488d8577791c [05:24:15] [02ssl] 07WikiTideSSLBot 03488d857 - Bot: Update SSL cert for holocron.net [05:53:52] RECOVERY - holocron.net - LetsEncrypt on sslhost is OK: OK - Certificate 'holocron.net' will expire on Sat 28 Sep 2024 04:24:05 AM GMT +0000. [06:11:55] PROBLEM - swiftac171 PowerDNS Recursor on swiftac171 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:12:15] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:12:35] PROBLEM - swiftac171 Current Load on swiftac171 is CRITICAL: LOAD CRITICAL - total load average: 15.81, 9.64, 5.03 [06:13:49] RECOVERY - swiftac171 PowerDNS Recursor on swiftac171 is OK: DNS OK: 0.204 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [06:14:14] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 44 packages available for upgrade (0 critical updates). [06:14:34] PROBLEM - swiftac171 Current Load on swiftac171 is WARNING: LOAD WARNING - total load average: 10.42, 11.01, 6.18 [06:16:34] RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 3.78, 8.28, 5.76 [07:15:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:18:15] [02mw-config] 07RhinosF1 commented on pull request 03#5596: T12271: Add $wgUseLegacyMediaStyles - 13https://github.com/miraheze/mw-config/pull/5596#issuecomment-2198462416 [07:24:51] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.89, 17.68, 12.95 [07:30:51] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.61, 21.22, 16.43 [07:32:51] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.74, 24.07, 18.06 [07:36:51] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.13, 22.73, 19.08 [07:38:51] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.44, 24.80, 20.28 [07:42:51] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.87, 23.09, 20.69 [07:44:11] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:44:51] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.61, 19.38, 19.66 [08:34:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.46, 20.24, 17.69 [08:38:18] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.12, 20.82, 18.38 [08:40:14] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.40, 21.11, 18.74 [08:49:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 22.39, 20.27 [08:56:44] [02MirahezeMagic] 07waki285 opened pull request 03#493: Add global-patroller xlations - 13https://github.com/miraheze/MirahezeMagic/pull/493 [08:56:51] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#493: Add global-patroller xlations - 13https://github.com/miraheze/MirahezeMagic/pull/493#issuecomment-2198487444 [08:59:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 12.84, 20.13, 20.88 [09:00:32] miraheze/MirahezeMagic - waki285 the build has errored. [09:01:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.21, 18.90, 20.31 [09:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:45:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:12:15] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.01, 21.19, 18.46 [10:13:59] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [10:14:11] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.08, 20.38, 18.48 [10:18:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.71, 21.81, 19.45 [10:21:54] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.20, 19.27, 18.99 [10:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:21:22] [02MirahezeMagic] 07waki285 opened pull request 03#494: Add wiki-mechanic xlations - 13https://github.com/miraheze/MirahezeMagic/pull/494 [11:22:09] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#494: Add wiki-mechanic xlations - 13https://github.com/miraheze/MirahezeMagic/pull/494#issuecomment-2198527964 [11:25:06] miraheze/MirahezeMagic - waki285 the build has errored. [12:17:47] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:14:45] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:34:55] [02MirahezeMagic] 07waki285 synchronize pull request 03#493: Add global-patroller xlations - 13https://github.com/miraheze/MirahezeMagic/pull/493 [13:38:31] miraheze/MirahezeMagic - waki285 the build has errored. [14:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:29:18] [1/2] https://github.com/miraheze/MirahezeMagic/pull/493 [14:29:19] [2/2] Due to rename and expand, please merge [14:29:29] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [14:31:29] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 4.959106445e-05 secs [15:10:23] PROBLEM - www.thegreatwar.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'thegreatwar.uk' expires in 7 day(s) (Mon 08 Jul 2024 03:02:33 PM GMT +0000). [15:13:07] [02MirahezeMagic] 07waki285 synchronize pull request 03#493: Add global-patroller xlations - 13https://github.com/miraheze/MirahezeMagic/pull/493 [15:14:08] [02MirahezeMagic] 07waki285 synchronize pull request 03#494: Add wiki-mechanic xlations - 13https://github.com/miraheze/MirahezeMagic/pull/494 [15:17:20] miraheze/MirahezeMagic - waki285 the build has errored. [15:17:37] miraheze/MirahezeMagic - waki285 the build has errored. [15:25:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.05, 21.57, 18.19 [15:27:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.39, 21.78, 18.66 [15:29:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.05, 19.72, 18.33 [15:31:37] PROBLEM - thegreatwar.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'thegreatwar.uk' expires in 7 day(s) (Mon 08 Jul 2024 03:02:33 PM GMT +0000). [15:52:13] [02mw-config] 07waki285 opened pull request 03#5597: global-sysop -> global-admin - 13https://github.com/miraheze/mw-config/pull/5597 [15:59:39] [02MirahezeMagic] 07waki285 opened pull request 03#495: Add global-admin xlations, change grouppage-global-sysop - 13https://github.com/miraheze/MirahezeMagic/pull/495 [16:00:22] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#495: Add global-admin xlations, change grouppage-global-sysop - 13https://github.com/miraheze/MirahezeMagic/pull/495#issuecomment-2198606425 [16:04:04] miraheze/MirahezeMagic - waki285 the build has errored. [17:16:20] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:44:31] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:15:07] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [18:15:37] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:41:17] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:15:09] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:18] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:44:39] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.68, 21.17, 17.96 [21:23:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.52, 19.78, 18.59 [22:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:36:25] [02mw-config] 07BlankEclair opened pull request 03#5598: T12214: Avoid redirects if there are query parameters - 13https://github.com/miraheze/mw-config/pull/5598 [23:37:25] miraheze/mw-config - BlankEclair the build passed. [23:53:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.37, 20.07, 18.27 [23:55:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.61, 22.12, 19.22 [23:59:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.60, 19.20, 15.44