[00:24:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.75, 22.88, 23.83 [00:31:50] !log [oa@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all [00:31:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:02] !log [oa@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 12s [00:32:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 23.22, 23.44 [00:38:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.16, 22.92, 23.31 [00:40:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.07, 23.64, 23.50 [00:44:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.63, 23.71, 23.57 [00:48:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.84, 25.09, 24.07 [01:02:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.39, 23.53, 23.87 [01:04:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.24, 24.43, 24.17 [01:18:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.80, 22.20, 23.65 [01:28:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.80, 23.36, 23.09 [01:32:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.42, 23.10, 23.11 [01:36:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.76, 23.26, 23.04 [01:40:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.69, 22.76, 22.95 [01:58:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 23.68, 22.60 [02:00:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.09, 22.13, 22.19 [02:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:22:02] [02mediawiki-repos] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/101d0b110d33...13d742a89479 [02:22:05] [02mediawiki-repos] 07The-Voidwalker 0313d742a - CustomSearchProfiles update branch to main [02:22:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.69, 21.49 [02:24:47] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:25:02] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:26:22] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [02:40:05] [02ssl] 07The-Voidwalker closed pull request 03#785: T12223: Redirect www.lebork.info.pl to lebork.info.pl - 13https://github.com/miraheze/ssl/pull/785 [02:40:07] [02ssl] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8f96a785026...43b594941d5e [02:40:09] [02ssl] 07BlankEclair 0343b5949 - T12223: Redirect www.lebork.info.pl to lebork.info.pl (#785) [02:48:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.81, 20.72, 23.23 [02:58:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.65, 24.45, 23.53 [03:02:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.23, 22.87, 23.26 [03:08:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 23.25, 23.14 [03:10:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.88, 22.35, 22.82 [03:34:10] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.89, 19.19, 20.14 [03:41:00] [02ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/65eec4d2ec7f [03:41:00] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.04, 21.61, 20.96 [03:41:02] [02ssl] 07MacFan4000 0365eec4d - Delete certificates/fallofsanctuary.com.crt [03:41:05] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [03:41:08] [02ssl] 07MacFan4000 opened pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786 [03:41:13] [02ssl] 07coderabbitai[bot] commented on pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786#issuecomment-2197880828 [03:41:47] [02ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/65eec4d2ec7f...ec8d89bb8940 [03:41:48] [02ssl] 07MacFan4000 03ec8d89b - Delete certificates/terraforming.li.crt [03:41:50] [02ssl] 07MacFan4000 synchronize pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786 [03:41:54] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786#issuecomment-2197880828 [03:42:41] [02ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ec8d89bb8940...162930d9d733 [03:42:43] [02ssl] 07MacFan4000 03162930d - Update certs.yaml [03:42:47] [02ssl] 07MacFan4000 synchronize pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786 [03:42:50] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786#issuecomment-2197880828 [03:44:56] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.80, 18.43, 19.85 [03:45:33] [02ssl] 07MacFan4000 closed pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786 [03:45:35] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-2/±1] 13https://github.com/miraheze/ssl/compare/43b594941d5e...3043c974ca5f [03:45:36] [02ssl] 07MacFan4000 033043c97 - Delete certificates/fallofsanctuary.com.crt (#786) [03:45:37] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [03:45:40] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [03:47:01] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#786: Delete certificates/fallofsanctuary.com.crt - 13https://github.com/miraheze/ssl/pull/786#issuecomment-2197880828 [03:49:19] [02dns] 07MacFan4000 opened pull request 03#532: remove fallofsanctuary.com and terraforming.li zones - 13https://github.com/miraheze/dns/pull/532 [03:49:34] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.06, 21.78, 17.01 [03:51:30] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.35, 21.11, 17.33 [03:54:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.82, 19.91, 19.66 [03:57:16] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.44, 18.98, 17.93 [03:58:40] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 22.34, 20.63 [04:00:37] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.72, 20.33, 20.12 [04:03:04] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.63, 20.97, 18.93 [04:06:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.00, 23.96, 20.60 [04:10:45] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.46, 23.13, 21.15 [04:14:36] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.22, 19.46, 20.23 [04:27:54] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.84, 21.73, 20.04 [04:29:49] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.65, 19.48, 19.42 [04:35:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.32, 20.10, 19.62 [04:37:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.80, 19.28, 19.37 [04:43:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.37, 20.78, 20.01 [04:53:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.87, 19.95, 20.26 [04:59:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.03, 21.00, 20.64 [05:03:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.10, 19.43, 20.12 [05:11:30] PROBLEM - wiki.sadboyzpod.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sadboyzpod.com' expires in 15 day(s) (Mon 15 Jul 2024 05:06:46 AM GMT +0000). [05:11:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3043c974ca5f...6c9f156523f7 [05:11:44] [02ssl] 07WikiTideSSLBot 036c9f156 - Bot: Update SSL cert for wiki.sadboyzpod.com [05:26:14] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 21.22, 19.70 [05:30:04] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.79, 21.55, 20.27 [05:33:56] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.80, 20.20, 19.97 [05:39:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.30, 20.47, 20.07 [05:41:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 22.62, 20.89 [05:43:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.82, 21.10, 20.62 [05:45:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.68, 19.48, 20.12 [06:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:09:55] RECOVERY - wiki.sadboyzpod.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sadboyzpod.com' will expire on Fri 27 Sep 2024 04:11:35 AM GMT +0000. [06:45:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.08, 18.94, 15.15 [06:49:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.15, 19.91, 16.65 [08:11:56] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.45, 19.23, 18.12 [08:31:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.75, 20.31, 20.13 [09:01:49] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 21.58, 19.90 [09:03:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.42, 21.13, 19.91 [09:05:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.73, 19.29, 19.40 [09:58:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.30, 20.96, 19.21 [10:00:14] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.28, 19.54, 18.90 [10:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:22:58] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864160 seconds old and 133572 bytes [10:46:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:55:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.97, 19.84, 17.99 [10:59:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.77, 20.03, 18.53 [11:10:36] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.74, 20.69, 19.43 [11:12:32] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.73, 19.95, 19.33 [11:14:31] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:28:57] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.86, 22.27, 20.43 [11:30:53] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.14, 20.94, 20.19 [11:44:21] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 23.67, 21.72 [11:46:17] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.36, 22.56, 21.53 [11:55:54] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 23.07, 21.93 [12:03:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.71, 22.22, 22.48 [12:11:03] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 15.53, 10.19, 4.23 [12:13:02] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 2.35, 6.92, 3.75 [12:14:55] PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864169 seconds old and 133515 bytes [12:15:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.47, 22.72, 22.22 [12:17:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.04, 22.50, 22.19 [12:19:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.00, 23.69, 22.66 [12:31:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.72, 23.79, 23.64 [12:35:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.54, 23.07, 23.28 [12:37:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.66, 22.47, 23.03 [12:47:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 21.78, 22.04 [12:49:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.00, 21.28, 21.82 [13:03:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.73, 17.41, 19.73 [13:15:34] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [14:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:59:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 22.11, 18.75 [15:01:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.01, 21.29, 18.88 [15:05:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.17, 19.42, 18.70 [15:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:29:08] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.19, 20.75, 19.74 [15:31:05] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 22.18, 20.39 [15:32:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:32:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.18, 22.34, 20.65 [15:44:32] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.53, 19.56, 20.30 [16:18:52] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o