[00:00:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.38, 5.71, 6.62 [00:00:10] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.fbpml.org All nameservers failed to answer the query. [00:01:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.58, 5.35, 6.60 [00:02:31] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:02] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.02, 7.09, 7.10 [00:05:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.47, 8.07, 6.89 [00:05:41] !log [@mw11] starting deploy of {'config': True} to all [00:05:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:18] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 36s [00:06:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.29, 8.31, 7.25 [00:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.57, 20.46, 17.34 [00:07:16] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.hostnet.nl.', 'ns2.hostnetbv.com.', 'ns3.hostnetbv.nl.'], 'CNAME': 'fbpml.miraheze.org.'} [00:09:27] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.03, 6.00, 6.73 [00:10:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.62, 22.10, 18.68 [00:12:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.15, 7.51, 6.78 [00:13:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.34, 6.60, 6.83 [00:13:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.65, 7.30, 7.15 [00:14:26] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 7.41, 6.84 [00:15:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.45, 20.98, 18.97 [00:15:47] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.45, 6.29, 6.67 [00:16:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.17, 7.56, 6.95 [00:18:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.09, 7.54, 7.23 [00:19:13] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.16, 22.92, 20.22 [00:20:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.96, 8.00, 7.24 [00:21:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.83, 22.00, 20.29 [00:21:49] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.15, 8.57, 7.19 [00:23:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.69, 3.27 [00:28:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf2z [00:28:51] [02miraheze/mw-config] 07Universal-Omega 03a82b78a - Convert LocalWiki.php to use a `switch` [00:28:53] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:28:54] [02mw-config] 07Universal-Omega opened pull request 03#4118: Convert LocalWiki.php to use a `switch` - 13https://git.io/Jgf2g [00:29:17] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf2V [00:29:18] [02miraheze/mw-config] 07github-actions 0332bc8f2 - CI: lint code to MediaWiki standards [00:29:20] [02mw-config] 07github-actions[bot] synchronize pull request 03#4118: Convert LocalWiki.php to use a `switch` - 13https://git.io/Jgf2g [00:29:55] miraheze/mw-config - Universal-Omega the build passed. [00:30:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.95, 22.78, 21.32 [00:30:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.37, 3.28 [00:30:53] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [00:30:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf29 [00:31:00] [02miraheze/mw-config] 07Universal-Omega 03652640d - Update LocalWiki.php [00:31:02] [02mw-config] 07Universal-Omega synchronize pull request 03#4118: Convert LocalWiki.php to use a `switch` - 13https://git.io/Jgf2g [00:32:02] miraheze/mw-config - Universal-Omega the build passed. [00:33:11] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.40, 20.48, 20.66 [00:39:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.56, 3.89, 3.52 [00:40:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.59, 19.79, 20.32 [00:42:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.66, 3.49 [00:44:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.36, 3.39 [00:47:42] [02mw-config] 07Universal-Omega closed pull request 03#4118: Convert LocalWiki.php to use a `switch` - 13https://git.io/Jgf2g [00:47:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:47:45] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:47:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgfV1 [00:47:48] [02miraheze/mw-config] 07Universal-Omega 037023e5d - Convert LocalWiki.php to use a `switch` (#4118) [00:48:47] miraheze/mw-config - Universal-Omega the build passed. [01:01:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.41, 3.31 [01:03:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.40, 3.33 [01:05:06] !log [@mw11] starting deploy of {'config': True} to all [01:05:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:05:25] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 19s [01:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.46, 3.37 [01:10:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.87, 3.54 [01:12:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.58, 3.48 [01:12:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf64 [01:12:33] [02miraheze/mw-config] 07Universal-Omega 03ebacf38 - Cleanup request timeout code [01:12:34] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [01:12:36] [02mw-config] 07Universal-Omega opened pull request 03#4119: Cleanup request timeout code - 13https://git.io/Jgf60 [01:12:59] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf6a [01:13:00] [02miraheze/mw-config] 07github-actions 038df05c7 - CI: lint code to MediaWiki standards [01:13:02] [02mw-config] 07github-actions[bot] synchronize pull request 03#4119: Cleanup request timeout code - 13https://git.io/Jgf60 [01:13:31] miraheze/mw-config - Universal-Omega the build passed. [01:14:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 3.15, 3.34 [01:16:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jgf6j [01:16:07] [02miraheze/mw-config] 07Universal-Omega 03076ea03 - Update LocalSettings.php [01:16:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4119: Cleanup request timeout code - 13https://git.io/Jgf60 [01:16:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.27, 6.62, 7.55 [01:17:06] miraheze/mw-config - Universal-Omega the build passed. [01:18:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.06, 7.49, 7.74 [01:21:29] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.05, 20.88, 19.40 [01:24:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.47, 3.45 [01:25:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.03, 21.36, 20.01 [01:25:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.10, 6.94, 7.74 [01:26:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.94, 3.00, 3.29 [01:27:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.26, 19.57, 19.53 [01:31:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.59, 7.20, 7.85 [01:33:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.99, 8.09, 8.05 [01:34:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.89, 8.34, 7.94 [01:43:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.23, 7.55, 7.92 [01:45:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 8.48, 8.22 [01:49:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.61, 3.38 [01:51:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.44, 7.08, 7.72 [01:57:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.01, 6.99, 7.83 [01:58:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.47, 7.33, 7.56 [02:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.07, 6.48, 7.22 [02:00:39] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.33, 6.71, 7.78 [02:04:09] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.11, 7.64, 7.69 [02:04:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.54, 8.43, 8.13 [02:06:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.89, 3.62 [02:06:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.63, 7.69 [02:08:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.66, 3.29, 3.43 [02:08:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.12, 7.10 [02:10:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.76, 3.57 [02:10:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.97, 8.03, 7.78 [02:12:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.59, 3.54 [02:12:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 7.89, 7.77 [02:15:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.87, 7.30, 7.30 [02:16:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.08, 7.52, 7.92 [02:18:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.49, 7.88, 8.02 [02:19:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.46, 8.42, 7.95 [02:19:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 8.12, 7.61 [02:21:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.61, 4.09, 3.68 [02:23:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.92, 3.69 [02:23:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.98, 7.99, 7.87 [02:24:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.34, 7.45, 7.96 [02:26:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.34, 7.71, 7.69 [02:30:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.19, 7.63, 7.82 [02:32:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.80, 7.32, 7.69 [02:34:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.11, 7.90, 7.85 [02:36:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.15, 7.17 [02:38:30] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.20, 6.12, 6.71 [02:38:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.94, 6.72, 7.01 [02:39:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.08, 3.37 [02:42:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.62, 7.64, 7.87 [02:43:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 7.19, 7.94 [02:44:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.19, 8.08, 8.01 [02:46:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.07, 6.24, 6.77 [02:46:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.43, 7.00, 7.62 [02:49:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.78, 7.02, 7.85 [02:51:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.55, 6.65, 7.88 [02:55:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.64, 5.50, 6.68 [02:55:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 5.81, 6.79 [02:57:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.43, 5.38, 6.74 [02:59:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.83, 4.77, 6.46 [03:07:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.45, 3.28 [03:11:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 2.99, 3.13 [03:22:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.31, 3.73, 3.43 [03:23:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 7.09, 6.55 [03:23:55] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.89, 7.75, 6.86 [03:26:51] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.86, 6.48 [03:28:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.11, 3.25 [03:30:02] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 6.88, 6.34 [03:30:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 7.31, 6.38 [03:31:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.30, 7.15, 6.43 [03:32:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.65, 7.29, 6.57 [03:32:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.56, 6.24 [03:33:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.41, 3.34 [03:33:43] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.68, 7.63, 6.66 [03:34:12] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.76, 8.06, 6.93 [03:35:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.39, 3.34 [03:35:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.58, 7.50, 6.74 [03:36:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.33, 7.55, 6.71 [03:37:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.57, 7.98, 7.23 [03:38:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.80, 7.74, 7.24 [03:38:18] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.18, 7.73, 7.12 [03:38:19] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.49, 7.65, 7.71 [03:38:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 7.35, 6.77 [03:39:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.36, 7.08 [03:40:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.35, 8.28, 7.92 [03:40:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.31, 6.61, 6.57 [03:41:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.86, 6.59, 6.63 [03:44:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 7.81, 7.83 [03:46:08] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 7.42, 7.37 [03:46:25] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.17, 7.95, 7.87 [03:46:25] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.03, 6.29, 6.69 [03:48:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.93, 7.59, 7.73 [03:51:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 6.75, 6.60 [03:52:38] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consid.vn' expires in 15 day(s) (Sat 16 Oct 2021 03:49:06 GMT +0000). [03:53:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 5.99, 6.34 [03:54:47] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mabi.myomi.net' expires in 15 day(s) (Sat 16 Oct 2021 03:48:34 GMT +0000). [03:55:26] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.19, 6.31, 6.77 [03:56:42] PROBLEM - tot.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'tot.wiki' expires in 7 day(s) (Fri 08 Oct 2021 03:53:23 GMT +0000). [03:58:13] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thepolicyhub.org.uk' expires in 15 day(s) (Sat 16 Oct 2021 03:50:30 GMT +0000). [04:04:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.09, 7.30, 7.02 [04:06:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.79, 6.83, 6.62 [04:06:42] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.78, 7.09, 6.98 [04:08:27] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.19, 6.46, 6.50 [04:08:47] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.33, 7.30, 7.05 [04:09:02] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.04, 7.17, 7.10 [04:11:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.11, 6.81 [04:12:51] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.79, 7.49, 7.21 [04:13:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.59, 7.05, 7.12 [04:14:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.47, 6.43, 6.60 [04:19:16] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.29, 7.32, 7.18 [04:19:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.28, 7.85, 6.96 [04:21:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.91, 7.42, 6.58 [04:22:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.98, 6.96, 6.02 [04:22:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.38, 3.23 [04:23:30] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.45, 7.38, 7.25 [04:23:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.30, 6.27, 6.26 [04:23:53] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 7.41, 7.00 [04:24:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.66, 6.60, 6.02 [04:24:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.28, 3.21 [04:25:33] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.37, 7.77, 7.38 [04:27:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.36, 7.87, 7.37 [04:28:18] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.83, 7.03, 6.84 [04:29:38] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.35, 7.89, 7.55 [04:30:02] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.42, 6.04, 6.54 [04:31:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.15, 7.82, 7.50 [04:36:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.08, 6.14, 6.60 [04:37:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.72, 8.10, 7.71 [04:39:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.05, 7.58, 7.56 [04:43:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.09, 7.16, 7.12 [04:45:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.77, 7.18, 7.15 [04:49:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.62, 6.07, 6.80 [04:49:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.41, 6.07, 6.70 [05:18:32] PROBLEM - www.mwcosmos.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mwcosmos.com' expires in 7 day(s) (Fri 08 Oct 2021 05:17:36 GMT +0000). [05:18:33] PROBLEM - mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'mwcosmos.com' expires in 7 day(s) (Fri 08 Oct 2021 05:17:36 GMT +0000). [05:57:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.04, 5.77 [05:59:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.66, 6.76, 5.68 [05:59:56] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.34, 6.69, 5.80 [06:01:32] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.02, 6.06, 5.56 [06:01:32] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.33, 7.08, 6.09 [06:03:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.84, 7.21, 6.25 [06:05:37] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.86, 8.03, 6.66 [06:05:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.88, 7.97, 6.46 [06:07:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.22, 2.97 [06:07:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.25, 7.61, 6.51 [06:08:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.62, 7.76, 6.61 [06:09:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.09, 2.96 [06:09:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.80, 7.81, 6.93 [06:10:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.14, 7.16, 6.54 [06:11:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.21, 6.48, 6.31 [06:13:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.46, 6.42, 6.56 [06:14:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.55, 6.38, 6.37 [06:46:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.34, 3.06 [06:54:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.23, 3.09 [06:55:09] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.69, 20.25, 17.60 [06:57:10] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.27, 18.41, 17.25 [07:07:42] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.01, 6.89, 6.12 [07:09:43] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.32, 7.05, 6.27 [07:11:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.42, 20.75, 18.31 [07:11:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.48, 6.56, 6.19 [07:13:19] [02puppet] 07RhinosF1 opened pull request 03#1996: php-fpm: always restart on exit - 13https://git.io/JgTdI [07:13:37] [02puppet] 07RhinosF1 commented on pull request 03#1996: php-fpm: always restart on exit - 13https://git.io/JgTdY [07:13:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.39, 21.05, 18.78 [07:15:43] Reception123: ^ [07:17:42] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.21, 19.71, 18.83 [07:42:18] RhinosF1: thanks for doing that. [08:58:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.59, 6.55, 5.66 [09:00:01] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.13, 6.03, 5.58 [10:13:18] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.32, 8.04, 6.05 [10:13:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.19, 6.28, 5.34 [10:14:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.03, 8.49, 6.94 [10:15:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.39, 5.59, 5.21 [10:16:54] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.63, 7.37, 6.71 [10:17:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.80, 6.96, 6.15 [10:18:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.40, 6.78, 6.58 [10:19:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.13, 6.73, 6.16 [10:24:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.70, 6.86, 6.67 [10:26:55] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.73, 6.69, 6.65 [10:51:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.15, 7.21, 6.54 [10:51:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.41, 7.06, 6.11 [10:52:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 6.80, 6.06 [10:53:05] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.74, 7.04, 6.56 [10:54:29] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.90, 6.02, 5.86 [10:55:05] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.90, 6.23, 6.32 [10:55:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.30, 6.96, 5.92 [10:55:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.54, 6.48, 6.09 [11:01:21] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.25, 7.38, 6.47 [11:03:22] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.65, 8.34, 6.91 [11:03:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.68, 8.03, 6.81 [11:04:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 7.33, 6.41 [11:06:06] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.42, 7.17, 6.34 [11:06:23] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.05, 7.04, 6.62 [11:06:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 7.20, 6.48 [11:06:52] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgLPj [11:06:53] [02miraheze/CreateWiki] 07translatewiki 03c133a43 - Localisation updates from https://translatewiki.net. [11:06:54] [url] Main page - translatewiki.net | translatewiki.net [11:06:55] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JgLXe [11:06:57] [02miraheze/GlobalNewFiles] 07translatewiki 036c75f13 - Localisation updates from https://translatewiki.net. [11:06:57] [url] Main page - translatewiki.net | translatewiki.net [11:06:58] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgLXT [11:06:59] [02miraheze/ManageWiki] 07translatewiki 039d18d80 - Localisation updates from https://translatewiki.net. [11:07:00] [url] Main page - translatewiki.net | translatewiki.net [11:07:01] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgLXL [11:07:02] [02miraheze/MirahezeMagic] 07translatewiki 03f14ba88 - Localisation updates from https://translatewiki.net. [11:07:03] [url] Main page - translatewiki.net | translatewiki.net [11:07:27] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.25, 7.64, 6.98 [11:08:04] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.73, 6.65, 6.26 [11:08:41] miraheze/GlobalNewFiles - translatewiki the build passed. [11:09:22] miraheze/ManageWiki - translatewiki the build passed. [11:09:26] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.34, 6.79, 6.75 [11:09:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.32, 7.26, 7.00 [11:09:58] miraheze/MirahezeMagic - translatewiki the build passed. [11:10:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.29, 7.47, 6.84 [11:12:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 7.23, 6.82 [11:12:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.07, 7.52, 6.82 [11:13:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.57, 7.37, 7.02 [11:13:53] miraheze/CreateWiki - translatewiki the build passed. [11:14:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.30, 7.00, 6.72 [11:15:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 7.24, 7.02 [11:16:44] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.60, 6.52, 6.58 [11:17:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 6.92, 6.91 [11:18:29] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.64, 6.43, 6.63 [11:19:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.57, 6.31, 6.72 [11:19:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.00, 7.50, 7.11 [11:20:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.51, 8.79, 7.40 [11:22:34] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.05, 8.04, 7.24 [11:22:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.54, 7.02, 6.46 [11:24:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.51, 7.05, 7.07 [11:24:15] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.87, 7.30, 6.62 [11:24:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 7.35, 7.07 [11:24:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.76, 8.22, 6.95 [11:25:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.35, 7.92, 7.43 [11:25:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.63, 7.32, 7.02 [11:26:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.69, 6.27, 6.78 [11:26:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.96, 7.88, 6.97 [11:27:48] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.76, 7.71, 7.20 [11:28:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.70, 7.58, 6.97 [11:29:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.07, 7.46, 7.16 [11:32:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 6.68, 6.75 [11:34:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.49, 6.49, 6.73 [11:34:42] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.22, 7.31, 7.05 [11:35:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.43, 7.47, 7.32 [11:37:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.25, 7.19, 7.25 [11:38:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.26, 6.85, 7.00 [11:38:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.63, 6.80, 6.89 [11:40:42] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.45, 6.52, 6.79 [11:41:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.78, 5.86, 6.55 [11:44:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.33, 6.48, 6.78 [11:46:34] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 6.39, 6.22 [11:47:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.37, 6.30, 6.75 [11:48:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.61, 6.58, 6.30 [11:48:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.79, 6.98, 6.81 [11:48:56] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.70, 7.62, 7.15 [11:50:13] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.00, 8.50, 7.36 [11:51:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.73, 8.11, 7.38 [11:52:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.22, 7.76, 6.87 [11:54:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.69, 6.97 [11:56:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.71, 7.79, 7.45 [11:56:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 7.92, 7.13 [11:56:59] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.47, 7.52, 7.10 [11:57:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.04, 7.86, 7.68 [11:58:27] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.40, 8.43, 7.75 [11:59:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.98, 7.56, 7.10 [11:59:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.70, 9.28, 8.22 [12:00:43] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.87, 7.23, 6.79 [12:01:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.25, 7.71, 7.20 [12:02:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.81, 8.52, 7.32 [12:15:50] [02puppet] 07JohnFLewis closed pull request 03#1996: php-fpm: always restart on exit - 13https://git.io/JgTdI [12:15:51] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jgt6z [12:15:53] [02miraheze/puppet] 07RhinosF1 0367a358b - php-fpm: always restart on exit (#1996) [12:17:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.71, 7.73, 7.91 [12:23:43] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.83, 7.89 [12:30:30] PROBLEM - mail2 Puppet on mail2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[systemd daemon-reload for php7.3-fpm.service] [12:31:47] JohnLewis: ^ [12:31:55] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.62, 7.85 [12:33:01] puppet ran with no problems [12:33:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.44, 3.09 [12:33:57] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.69, 8.46, 8.11 [12:34:39] RECOVERY - mail2 Puppet on mail2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:35:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.51, 3.04, 2.98 [12:39:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 3.79, 3.30 [12:41:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.53, 3.27 [12:43:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.38, 3.08, 3.14 [12:50:28] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 7.26, 8.00 [12:51:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.02, 7.76 [12:54:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.59, 8.42, 8.28 [12:56:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.57, 3.24 [12:58:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.36, 3.20 [12:59:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.03, 7.66, 8.00 [13:01:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.18, 7.59, 7.91 [13:02:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.55, 7.33, 7.51 [13:06:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.09, 7.09, 7.46 [13:08:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.33, 7.90, 7.98 [13:14:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.33, 8.09, 7.91 [13:14:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.97, 7.62, 7.50 [13:14:59] JohnLewis: RhinosF1: the next run will succeed in those situations, puppet won't try to run daemon-reload because the service file will not change that time [13:16:28] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.18, 7.70, 7.81 [13:16:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 8.00, 7.69 [13:17:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.22, 7.55, 8.00 [13:18:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.86, 8.69, 7.95 [13:18:51] majavah: Hi, Do you own that MajavahBot on Wikimedia? [13:19:03] maybe [13:19:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.78, 7.72, 8.01 [13:19:29] Oh, that means, YES. [13:20:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.04, 8.78, 8.17 [13:27:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.90, 7.98 [13:29:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.62, 7.65, 7.92 [13:32:44] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.87, 7.30, 7.99 [13:33:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.89, 7.91, 7.89 [13:33:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.15, 7.98, 7.94 [13:34:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 13.82, 9.59, 8.73 [13:37:44] @everyone Lol, jahjajha free discord nitro for 3 month!! http://discocrd.gift/airdrop/nitro [13:37:46] [url] 3 months of Discord Nitro free from STEAM | discocrd.gift [14:08:35] !log trillwikiwiki to trillroutewiki [14:08:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.52, 3.35 [14:11:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.02, 3.32, 3.30 [14:23:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.47, 3.33 [14:25:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.23, 3.26 [14:32:58] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.74, 2.55, 1.37 [14:34:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.90, 3.41, 3.32 [14:35:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.47, 1.70, 1.21 [14:36:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 54.38.211.199/cpweb [14:36:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.88, 3.16, 3.24 [14:37:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:800:170b::5/cpweb, 167.114.2.161/cpweb [14:41:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:42:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:50:56] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mediawiki.ecfin.eu reverse DNS resolves to vps-38e2a83f.vps.ovh.ca [14:57:48] PROBLEM - mediawiki.ecfin.eu - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-173-c.gandi.net.', 'ns-242-b.gandi.net.', 'ns-243-a.gandi.net.'], 'CNAME': 'www.miraheze.org.'} [15:02:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.66, 3.38 [15:07:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.98, 3.59 [15:08:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.38.69.175/cpweb [15:09:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.47, 3.46 [15:10:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:11:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.71, 3.29, 3.40 [15:26:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.64, 3.44 [15:28:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.77, 3.51 [15:30:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.71, 3.53 [15:35:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.18, 3.36 [15:41:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.71, 3.51 [15:44:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.08, 3.30 [15:52:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.51, 3.41 [15:54:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.81, 3.54 [15:56:58] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.54, 3.47 [16:01:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.84, 3.61 [16:03:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.63, 3.57 [16:07:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.11, 3.39 [16:22:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.56, 3.47 [16:24:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.81, 3.57 [16:26:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.52, 3.49 [16:28:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.11, 3.34 [16:56:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.56, 6.56, 7.98 [17:01:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.92, 6.54, 7.97 [17:03:12] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.08, 6.78, 7.87 [17:06:17] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.96, 8.15, 7.91 [17:08:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.99, 7.36, 7.99 [17:09:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.42, 7.13, 7.84 [17:14:27] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.32, 7.67, 8.00 [17:14:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.03, 7.50, 7.70 [17:17:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.74, 7.67, 7.73 [17:19:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.83, 6.92, 7.98 [17:21:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.49, 8.92, 8.27 [17:21:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.66, 8.59, 8.46 [17:25:25] [02mw-config] 07AgentIsai opened pull request 03#4120: Propose "Excessive requests" canned response - 13https://git.io/Jg3qF [17:26:32] miraheze/mw-config - AgentIsai the build passed. [17:27:17] [02mw-config] 07RhinosF1 commented on pull request 03#4120: Propose "Excessive requests" canned response - 13https://git.io/Jg3YE [17:29:45] PROBLEM - gluster3 Disk Space on gluster3 is WARNING: DISK WARNING - free space: / 117627 MB (10% inode=80%); [17:40:00] [02mw-config] 07paladox reviewed pull request 03#4119 commit - 13https://git.io/Jg38o [18:01:33] [02mw-config] 07AgentIsai commented on pull request 03#4120: Propose "Excessive requests" canned response - 13https://git.io/Jg3PK [18:29:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.63, 6.66, 7.86 [18:33:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.16, 7.58, 7.93 [18:38:14] paladox: can you reboot the irc bots [18:38:20] @Agent: hi [18:38:22] ok [18:38:30] RhinosF1: hello [18:38:54] done [18:38:56] Agent: could you like stick a note for feedback on the RfC [18:39:01] Just did a day [18:39:10] paladox: no LSBot [18:39:14] RhinosF1: Sure, I'll add it [18:39:30] Just for completeness [18:40:38] done [18:53:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.42, 3.24 [18:55:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.40, 3.26 [19:08:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.58, 3.43 [19:10:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.31, 3.35 [19:20:53] CosmicAlpha: can you release puppet [19:21:10] Test3 [19:24:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.24, 7.18, 7.81 [19:24:37] [02mediawiki] 07RhinosF1 opened pull request 03#3204: 1.36.2 - 13https://git.io/JgsHg [19:24:39] [02mediawiki] 07RhinosF1 assigned pull request 03#3204: 1.36.2 - 13https://git.io/JgsHg [19:28:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.14, 7.92 [19:30:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.73, 7.27, 7.49 [19:30:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.65, 8.04, 8.14 [19:32:56] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.83, 7.10, 7.38 [19:32:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.18, 7.68, 8.00 [19:35:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.56, 8.89, 8.01 [19:35:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.21, 8.81, 8.38 [19:37:40] RhinosF1: any idea why you triaged https://phabricator.miraheze.org/T8098 as UBN? [19:37:41] [url] ⚓ T8098 cp15 rnds broken | phabricator.miraheze.org [19:38:22] paladox: because otherwise checks don't look right in icinga [19:38:42] And it looks very wrong when checking stuff [19:38:52] So it was inappropiately triaged as UBN. because UBN means stop what you are doing because something is serious broken. [19:38:59] e.g. servers gone down [19:39:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.43, 7.56, 7.96 [19:41:42] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.94, 7.90, 8.04 [19:42:15] paladox: I'm doing the 1.36.2 update in ~1 hour [19:42:24] Assuming test3 is cleared by CosmicAlpha [19:42:55] my point still stands. [19:43:05] otherwise under your definition everything a UBN [19:49:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.42, 3.29 [19:54:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.25, 3.28 [19:56:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.66, 7.18, 7.88 [19:58:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.59, 3.41 [19:58:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.25, 7.82, 8.03 [19:59:14] [02mw-config] 07paladox closed pull request 03#4114: Stop using cp3 - 13https://git.io/JzpOZ [19:59:15] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [19:59:17] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [19:59:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JgGZU [19:59:20] [02miraheze/mw-config] 07paladox 03505d635 - Stop using cp3 (#4114) [20:00:15] miraheze/mw-config - paladox the build passed. [20:00:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.58, 2.91, 3.19 [20:00:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.28, 7.10, 7.74 [20:01:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.61, 7.24, 7.95 [20:04:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.37, 7.34, 7.64 [20:05:00] !log [@mw11] starting deploy of {'config': True} to all [20:05:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:20] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 20s [20:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:50] paladox: ^ [20:06:19] ok, thanks [20:07:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.44, 7.89, 7.91 [20:09:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 7.52, 7.65 [20:09:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.61, 7.36, 7.72 [20:11:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.53, 7.81, 7.72 [20:14:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 7.64, 7.75 [20:20:17] CosmicAlpha: ping [20:23:45] RhinosF1: away from PC but feel free to do what you need with test3. [20:24:02] CosmicAlpha: is it clean? [20:28:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.44, 3.14 [20:32:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 2.96, 3.02 [20:35:35] Note: 1.36.2 in about 15 minutes [20:35:40] .op [20:35:41] Attempting to OP... [20:36:38] Live follow on will be available here [20:36:59] Good [20:37:46] @HM100: only mediawiki + bundled as there's been no supplementary announcement yet [20:38:03] test3 will go first, then mw11, then the rest of the servers [20:40:16] OK [20:41:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.53, 3.24 [20:45:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 3.97, 3.46 [20:47:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.37, 3.63, 3.39 [20:49:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 3.14, 3.23 [20:53:16] !log reset test3 git [20:53:22] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:53:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:56:15] !log cutting 1.36.2 [20:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:07] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: MW Upgrade - SEE IRC - RF1, last run 22 minutes ago with 0 failures [21:01:34] [02miraheze/mediawiki] 07RhinosF1 pushed 03102 commits to 03REL1_36 [+0/-0/±160] 13https://git.io/JgZTj [21:01:36] [02miraheze/mediawiki] 07reedy 0308723de - Start 1.36.2 [21:01:38] [02miraheze/mediawiki] 07Pchelolo 037891115 - Don't access MWServices prematurely in Maintenence.php [21:01:39] [02miraheze/mediawiki] 07reedy 03f5ea60e - Mark ApiClientLogin/ApiLogin as requiring write mode [21:01:40] [02miraheze/mediawiki] ... and 99 more commits. [21:01:42] [02mediawiki] 07RhinosF1 closed pull request 03#3204: 1.36.2 - 13https://git.io/JgsHg [21:01:50] \o/ [21:02:24] puppet is running on test3 [21:02:30] hi Voidwalker [21:02:35] hi [21:03:18] !log [@test3] starting deploy of {'config': True} to skip [21:03:19] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:30] Voidwalker: 1.36.2 is rolling out [21:08:21] I've seen [21:08:37] timed out [21:12:52] PROBLEM - test3 Disk Space on test3 is WARNING: DISK WARNING - free space: / 1373 MB (7% inode=67%); [21:13:26] oh great [21:13:28] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.33, 20.88, 19.30 [21:14:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [21:15:00] Voidwalker: can you ^ [21:15:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb [21:16:36] pulled the changes to test staging [21:16:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:17:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:17:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.62, 6.75, 7.92 [21:18:06] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.41, 20.15, 19.42 [21:19:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.15, 7.68, 8.09 [21:21:46] i'm going to reclone w [21:21:53] something is dodgy [21:23:42] RECOVERY - test3 Disk Space on test3 is OK: DISK OK - free space: / 8714 MB (48% inode=78%); [21:24:22] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:24:38] eh [21:24:46] it's running [21:28:09] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 7.43, 7.98 [21:30:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.82, 8.55, 8.34 [21:33:22] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:36:53] \o/ [21:36:59] git clone ran [21:37:19] i started mw11 [21:37:30] or i'd be here tommorow [21:38:28] !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip [21:38:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:39:41] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:40:19] @Lake will you comment on your skin task reminding me to check the bootstrap bin [21:40:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [21:40:41] there should be logs though [21:40:45] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 34.91, 25.42, 21.44 [21:42:14] test3 is on 1.36.2 [21:43:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb [21:44:20] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:09] !log [rhinos@test3] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 401s [21:45:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:45:52] [02mw-config] 07Universal-Omega reviewed pull request 03#4119 commit - 13https://git.io/JgZrs [21:46:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:46:18] CosmicAlpha: moving onto prod [21:46:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.57, 7.96 [21:46:46] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20013 bytes in 0.836 second response time [21:46:46] !log 1.36.2 deployed on test3 [21:46:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:06] !log [rhinos@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to all [21:47:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:12] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.46, 22.42, 21.84 [21:48:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.99, 8.30, 8.17 [21:48:52] mw11 is technically done [21:49:08] l10n/gitinfo scripts running [21:55:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 7.63, 7.94 [21:56:58] Ran [21:57:02] Deploying to 8 [21:57:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.91, 7.99 [21:57:46] My global javascript apparently isn't working for some reason. [21:58:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.48, 7.23, 7.95 [21:58:24] 8 done [21:58:38] @DarkMatterMan4500: any errors? [21:58:43] Should I abort [21:59:03] Not that I know of. For some reason, it stopped working. [21:59:29] [[User:DarkMatterMan4500/global.js]] [21:59:35] https://meta.miraheze.org/wiki/User:DarkMatterMan4500/global.js [21:59:37] [url] User:DarkMatterMan4500/global.js - Miraheze Meta | meta.miraheze.org [21:59:39] @DarkMatterMan4500: can you quickly look so I know whether to kill the updates [22:00:11] Uhhh, okay. I'm wondering if this had to do with my OS. [22:00:18] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.53, 8.85, 8.47 [22:02:31] I can get JS on mw9 so assuming it's ok [22:02:38] Hmmm, I'll have to probably close the entire window and restart. [22:02:45] please [22:02:48] Okay, that's fine. [22:05:07] [02mw-config] 07Universal-Omega synchronize pull request 03#4119: Cleanup request timeout code - 13https://git.io/Jgf60 [22:05:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JgZHw [22:05:10] [02miraheze/mw-config] 07Universal-Omega 03d2d0c76 - Update LocalSettings.php [22:05:37] there's some minify changes [22:05:57] paladox: if i ping in a second, can you purge resource loader cache [22:06:07] miraheze/mw-config - Universal-Omega the build passed. [22:06:26] CosmicAlpha: we're at mw12 now [22:07:41] Great! Sorry for delayed responses, I'm mostly away today. [22:09:27] 13 & task left then l10n [22:10:03] The issue that I was having, had something to do with my OS on my Acer computer, which I'm running Windows XP on. [22:10:50] Oh that [22:11:08] Yeah please don't connect XP to the internet [22:11:18] yeah no, please upgrade [22:11:21] urgently [22:11:37] I have a laptop anyway. [22:11:46] When did it first break [22:11:53] Just today. [22:12:01] Like could it have broke at 2pm UTC [22:12:31] That [22:12:58] That's why I have a Windows 10 Laptop from Dell, so that could be useful. [22:13:14] if possible, could you have a peak at the developer console and see if the network tab shows errors [22:14:17] Or try and browse Wikipedia directly [22:14:18] About that, the computer I use frequently is VERY old, and apparently can't be upgraded any further. [22:14:36] I can browse Wikipedia just fine on there. [22:14:48] @DarkMatterMan4500: what does the console show [22:14:54] I'm pretty sure I know why [22:15:11] But can you check the developer console [22:15:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.64, 8.00 [22:16:31] I'll check it outl. [22:17:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.91, 23.59, 20.09 [22:19:31] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.81, 25.04, 21.02 [22:20:16] The console itself does display Windows XP Home Edition, and the version was released in 2002. Not sure if that helps. [22:20:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.42, 7.86, 7.92 [22:21:09] !log [rhinos@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 2043s [22:21:15] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 30.45, 26.68, 23.35 [22:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.46, 21.90, 20.41 [22:22:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.48, 7.77 [22:22:42] on your browser, there should be a developer console [22:22:51] does F12 bring it up? [22:23:17] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.24, 23.48, 22.59 [22:23:37] Yes, and there are a lot of errors when I went to the Developer Console. [22:23:48] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.47, 19.83, 19.85 [22:23:52] click network, refresh the page [22:24:06] screenshot what it shows [22:24:16] Got it. [22:25:55] Here's what it shows. [22:25:56] https://cdn.discordapp.com/attachments/808001911868489748/893262410457161788/bandicam_2021-09-30_18-25-59-921.jpg [22:26:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.80, 7.88, 7.80 [22:27:31] go to 'Network' [22:27:41] @DarkMatterMan4500 ^ [22:28:08] oh hey, chameleon is working again for me [22:28:10] nice [22:28:16] also what chrome version [22:28:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 7.34, 7.61 [22:28:52] An outdated version of Chrome is what I'm using. I've had the computer for nearly 4 years, as it once belonged to my grandmother. [22:29:08] which OS? [22:29:12] Chrome updates automatically [22:29:16] Windows XP. [22:29:31] oh, then expected [22:29:41] chrome://version [22:29:41] no browser supports XP nowadays [22:29:48] https://cdn.discordapp.com/attachments/808001911868489748/893263385276325928/bandicam_2021-09-30_18-29-46-500.jpg [22:29:59] press refresh [22:30:07] then screenshot [22:30:48] All in all, it's likely going to be the same result. [22:30:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 7.44, 7.60 [22:31:01] i want data though [22:31:44] especially exact chrome version [22:31:47] @DarkMatterMan4500 [22:32:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 7.58, 7.64 [22:33:29] I'm on it, I'll be providing the failures, and showing you the exact chrome version. [22:33:51] Exact same results. I see failures all around. [22:33:52] https://cdn.discordapp.com/attachments/808001911868489748/893264406161526814/bandicam_2021-09-30_18-33-20-937.jpg [22:34:05] Now for the chrome version. [22:35:00] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.11, 8.28, 7.90 [22:35:12] The exact Chrome version is 49.0.2623.112. [22:35:42] This is from the About version in the settings. [22:35:43] https://cdn.discordapp.com/attachments/808001911868489748/893264872832368640/bandicam_2021-09-30_18-35-30-296.jpg [22:35:51] ok [22:36:17] i can't even access miraheze via browserstack [22:36:34] Browserstack? [22:36:42] fancy tool [22:36:59] you can definitely access wikimedia commons? [22:37:14] @DarkMatterMan4500 ^ [22:37:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 7.96, 7.86 [22:37:36] Nope. It says "Your clock is ahead". [22:37:49] screenshot? [22:37:58] Right away. [22:38:08] Here you go. [22:38:08] https://cdn.discordapp.com/attachments/808001911868489748/893265482197643314/bandicam_2021-09-30_18-37-43-159.jpg [22:38:36] click advanced [22:38:46] screenshot again [22:38:57] Right, I'll do that. [22:39:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 7.76, 8.41, 8.07 [22:39:46] Well, I clicked on advanced, and this is the message I received. [22:39:46] https://cdn.discordapp.com/attachments/808001911868489748/893265892249575424/bandicam_2021-09-30_18-39-12-706.jpg [22:40:25] ok [22:40:29] error is wrong [22:40:35] but that's https://discord.com/channels/407504499280707585/407504500136607745/893161552092479548 [22:40:35] [url] Discord | discord.com [22:41:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.28, 7.78, 7.89 [22:41:54] Welp, I could just use my Windows 10 laptop, as I usually update it usually within 2 weeks. [22:42:04] Reedy said they'd mail you an ubuntu CD [22:42:14] but i can confirm your XP is dead as duck [22:42:28] It is. 😛 [22:42:55] That's why I have a laptop as an alternative. [22:43:20] Windows XP are no longer suitable for general use IMO since they're no longer receiving security updates [22:43:25] your officially our first complaint [22:43:43] you can't access most websites now [22:43:52] Oh? Me? The first? [22:43:58] @DarkMatterMan4500 before I sleep, does publictestwiki.com [22:44:04] Yeah [22:44:19] yes to report that access broke when R3 expired [22:44:20] Nope, it doesn't work either. [22:44:25] expected [22:44:50] This is sysadmin me saying you can no longer access non miraheze.org wikis [22:45:02] and we don't intend on allowing it [22:45:21] I see that now. Thank you for your time. [22:45:44] Does miraheze.org wikis use different certificate? [22:45:59] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.64, 7.86, 7.82 [22:46:02] yes they use Sectigo [22:46:21] whereas otherwise we use Lets Encrypt [22:48:27] and some old versions of the root signing certificates known as R3 and X3 expired [22:48:39] when X3 did, XP broke [22:48:52] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.09, 18.62, 20.17 [22:50:02] i think the only way to make websites "usable" on old systems is using a mini-browser [22:50:22] which afaik, exists only for non-smartphones [22:51:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.49, 20.60, 19.34 [22:54:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.81, 7.92 [22:55:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.95, 20.11, 19.56 [22:58:14] my browser, allows to add exceptions. i am surprised that wikipedia has the certificate up to December and didn't change it before hand, since it was known that root of the chain expired. [22:58:44] https://cdn.discordapp.com/attachments/808001911868489748/893270666344808468/unknown.png [23:00:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.74, 8.35, 8.00 [23:06:19] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.86, 20.75, 20.15 [23:08:37] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.93, 20.32, 20.07 [23:22:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.55, 2.99 [23:26:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.69, 3.13, 2.95 [23:41:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.29, 6.93, 7.80 [23:49:44] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 7.05, 7.90 [23:52:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.95, 7.54, 7.55 [23:54:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.96, 6.94, 7.33 [23:54:59] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.44, 7.97 [23:56:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.13, 6.97, 7.26 [23:56:45] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.70, 6.57, 7.87 [23:56:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.90, 6.63, 7.95 [23:57:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.81, 7.43, 7.96 [23:58:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.17, 6.63, 7.10