[00:00:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.25, 1.58, 1.67 [00:01:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.52, 3.21 [00:02:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.97, 10.98, 10.69 [00:03:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.33, 3.38, 3.19 [00:04:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.03, 10.71, 10.61 [00:07:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.92, 3.54 [00:09:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.75, 3.52 [00:10:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.87, 1.74 [00:11:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 4.04, 3.65 [00:12:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.03, 1.91, 1.76 [00:13:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.71, 3.58 [00:13:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.38, 10.70, 10.38 [00:13:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.09, 4.02, 3.54 [00:14:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.09, 1.61, 1.67 [00:15:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.61, 3.92, 3.66 [00:17:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.85, 3.68 [00:18:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.87, 10.70, 9.69 [00:19:27] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.85, 3.28, 2.37 [00:20:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:20:29] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.97, 9.55 [00:21:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 3.90, 3.69 [00:21:27] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.32, 3.22, 2.46 [00:22:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.45, 2.01, 1.80 [00:22:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.00, 9.69, 10.16 [00:23:18] [02mw-config] 07Universal-Omega closed pull request 03#4814: Move `$wgJsonConfigs` out of LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4814 [00:23:19] [url] Page not found · GitHub · GitHub | github.com [00:23:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [00:23:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/c6888a3f5c6a...6de6c33b9256 [00:23:22] [url] Comparing c6888a3f5c6a...6de6c33b9256 · miraheze/mw-config · GitHub | github.com [00:23:23] [02miraheze/mw-config] 07Universal-Omega 036de6c33 - Move `$wgJsonConfigs` out of LocalSettings.php (#4814) [00:23:24] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [00:23:25] ... [00:23:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.85, 3.70 [00:24:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 10.66, 9.91 [00:24:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.77, 1.73 [00:24:34] miraheze/mw-config - Universal-Omega the build passed. [00:26:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.47, 10.59, 10.41 [00:28:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 11.28, 10.31 [00:28:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.89, 2.02, 1.86 [00:28:30] [02mw-config] 07Universal-Omega closed pull request 03#4813: Make vector default skin versions honour `$wgDefaultSkin` - 13https://github.com/miraheze/mw-config/pull/4813 [00:28:30] [url] Page not found · GitHub · GitHub | github.com [00:28:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:28:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/6de6c33b9256...18dccf8f2e87 [00:28:34] [url] Comparing 6de6c33b9256...18dccf8f2e87 · miraheze/mw-config · GitHub | github.com [00:28:34] [02miraheze/mw-config] 07Universal-Omega 0318dccf8 - Make vector default skin versions honour `$wgDefaultSkin` (#4813) [00:28:36] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:28:36] [url] Page not found · GitHub · GitHub | github.com [00:29:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 2.98, 3.37 [00:29:44] miraheze/mw-config - Universal-Omega the build passed. [00:30:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.38, 10.97, 10.32 [00:30:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 1.69, 1.76 [00:32:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.41, 1.65 [00:35:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.61, 3.97 [00:36:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.06, 12.28, 10.97 [00:38:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.98, 11.50, 10.86 [00:41:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.68, 11.65, 11.03 [00:42:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.82, 11.13 [00:43:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.85, 11.19, 10.95 [00:44:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 11.05, 10.92 [00:45:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:47:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.62, 2.59, 3.28 [00:50:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.03, 11.83, 11.14 [00:51:24] !log [@test101] starting deploy of {'config': True} to all [00:51:25] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:51:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 11.20, 10.91 [00:52:29] !log [@mwtask111] starting deploy of {'config': True} to all [00:52:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.40, 11.71, 11.19 [00:52:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:39] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [00:52:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 10.67, 10.75 [00:57:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.48, 10.80, 10.58 [00:58:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 11.72, 11.30 [00:59:12] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.95, 3.52 [01:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 3.22, 3.30 [01:01:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.68, 10.56, 10.57 [01:01:48] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 9.70, 4.22, 2.39 [01:03:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.75, 1.99 [01:03:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.33, 11.48, 10.90 [01:05:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.90, 11.18, 10.86 [01:06:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.92, 10.96, 10.63 [01:06:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.79, 11.95, 11.67 [01:06:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.63, 3.43 [01:07:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.54, 3.33 [01:08:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.81, 11.16, 10.77 [01:09:29] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.76, 1.92, 2.00 [01:10:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.66, 3.47 [01:11:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.46, 3.35 [01:11:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.56, 2.23, 2.09 [01:11:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 11.30, 10.95 [01:12:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.08, 3.27 [01:13:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.27, 3.37, 3.33 [01:13:17] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.29, 1.64 [01:13:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.44, 1.85, 1.96 [01:13:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.14, 11.26, 10.98 [01:14:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.55, 9.10, 10.05 [01:15:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.20, 11.61, 11.15 [01:16:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.74, 3.51 [01:17:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 1.66, 1.69 [01:17:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 11.55, 11.18 [01:18:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 3.19, 3.34 [01:21:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.27, 1.76, 1.82 [01:22:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.76, 3.57 [01:22:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.38, 2.69, 1.88 [01:23:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.38, 1.61, 1.77 [01:24:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.74, 4.04, 3.71 [01:25:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.43, 1.68 [01:26:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.62, 3.60 [01:27:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.79, 1.97, 1.95 [01:28:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.97, 1.81 [01:30:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.74, 3.64 [01:30:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.98, 10.96, 10.86 [01:30:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.96, 1.63, 1.70 [01:32:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.35, 3.50 [01:32:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.82, 11.20, 10.96 [01:39:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.89, 1.88 [01:39:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 10.85, 10.68 [01:39:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.38, 3.18, 3.37 [01:41:32] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404266595840454 seconds [01:41:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.66, 11.05, 10.77 [01:42:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.12, 9.26, 10.20 [01:44:23] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [01:51:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.89, 1.66 [01:51:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 3.66, 3.38 [01:53:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 2.12, 1.78 [01:53:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.33, 3.29 [01:53:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.26, 9.68, 10.16 [01:58:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.10, 1.79, 1.45 [01:58:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.30, 9.51, 9.62 [02:00:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.63, 1.71, 1.46 [02:02:02] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.48, 1.54, 1.42 [02:02:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 9.27, 9.52 [02:07:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.54, 4.85, 3.97 [02:07:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 4.75, 3.79 [02:10:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.77, 1.52, 1.90 [02:11:23] RECOVERY - wiki.zergmafia.top - reverse DNS on sslhost is OK: SSL OK - wiki.zergmafia.top reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:11:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.93, 3.68 [02:12:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.62, 11.00, 10.32 [02:14:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.60, 10.73, 10.31 [02:15:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.33, 3.72, 3.73 [02:16:27] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:16:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.28, 10.56 [02:16:53] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.54, 1.32, 1.68 [02:18:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 10.29, 9.47 [02:18:34] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.05, 11.23, 10.63 [02:19:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.02, 4.18, 3.77 [02:19:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.50, 4.20, 3.90 [02:20:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.96, 11.84, 10.92 [02:21:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:22:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.16, 10.18, 9.62 [02:22:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.56, 10.26, 9.79 [02:22:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.61, 11.84, 11.05 [02:23:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.71, 3.67 [02:23:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.72, 3.79 [02:24:57] SRE: issue on rothwellwiki when editing [02:25:00] [61a2eb81467e4916d647f3f4] 2022-07-14 02:24:04: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [02:25:29] CosmicAlpha, paladox: ^ [02:26:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 10.76, 9.98 [02:26:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.56, 9.72, 9.71 [02:27:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.52, 9.87, 8.92 [02:27:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.79, 3.86, 3.77 [02:28:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.21, 10.22 [02:28:34] please ping me if one of you responds [02:29:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.31, 9.81, 9.03 [02:29:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 3.85, 3.66 [02:29:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.59, 3.68 [02:30:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 10.79, 10.19 [02:31:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.74, 3.65 [02:31:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [02:32:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.29, 11.09, 10.24 [02:32:36] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.84, 1.65 [02:33:10] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 6.73, 6.84, 3.75 [02:34:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.44, 11.89, 10.75 [02:34:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.76, 10.43, 10.10 [02:34:34] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.62, 1.59 [02:37:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.07, 3.38 [02:37:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.38, 3.32, 3.01 [02:38:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.97, 11.81, 11.02 [02:38:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.56, 11.59, 11.02 [02:39:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.78, 2.00 [02:40:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.58, 10.82, 10.82 [02:41:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.44, 3.58, 3.50 [02:43:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 3.84, 3.65 [02:44:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.83, 9.02, 10.07 [02:45:03] BrandonWM: I am away. [02:45:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.49, 3.67, 3.57 [02:45:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.48, 3.55 [02:47:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.52, 4.51, 3.87 [02:47:14] I am away pretty much entirly this whole week with reduced activity for another week also, but semi-around next week. [02:47:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.78, 1.88 [02:47:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.87, 1.67, 1.43 [02:47:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.87, 4.03, 3.73 [02:48:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 10.12, 10.32 [02:49:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.36, 3.89, 3.71 [02:49:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.35, 3.51 [02:50:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.08, 11.91, 10.96 [02:51:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.83, 4.14, 3.79 [02:51:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.66, 1.52 [02:51:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.21, 3.81 [02:52:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.91, 9.83, 10.19 [02:52:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.29, 11.86, 11.20 [02:52:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 11.25, 10.84 [02:53:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.54, 3.61 [02:53:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.32, 3.31, 3.52 [02:54:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 11.42, 11.12 [02:55:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.33, 1.92, 2.00 [02:56:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.71, 1.57 [02:57:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.38, 3.88, 3.69 [02:58:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.38, 1.46 [02:59:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.60, 3.61 [03:00:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.39, 11.45, 11.00 [03:01:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.04, 3.40 [03:01:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.75, 3.24 [03:02:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.25, 11.19, 10.97 [03:05:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 10.35, 10.13 [03:05:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 10.44, 10.56 [03:07:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.59, 9.48, 9.82 [03:07:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.17, 10.22, 10.49 [03:09:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.08, 10.83, 10.68 [03:11:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.17, 3.16 [03:11:54] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.96, 11.57, 10.99 [03:13:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.47, 3.26 [03:15:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 1.89, 1.79 [03:15:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.36, 3.24 [03:18:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.50, 1.40 [03:19:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.34, 3.33, 3.25 [03:23:26] [02mw-config] 07Universal-Omega edited a comment on pull request 03#4736: Set new RemovePII DPA validation variables - 13https://github.com/miraheze/mw-config/pull/4736#issuecomment-1160912631 [03:23:26] [url] Set new RemovePII DPA validation variables by Universal-Omega · Pull Request #4736 · miraheze/mw-config · GitHub | github.com [03:23:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.93, 3.50 [03:23:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.82, 11.69, 11.20 [03:23:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.60, 3.31 [03:24:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.29, 1.90, 1.60 [03:25:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.83, 3.51 [03:25:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.63, 11.21, 11.08 [03:27:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.20, 5.15, 4.04 [03:27:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [03:30:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 10.88, 10.54 [03:32:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.75, 10.69, 10.51 [03:34:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.51, 11.16, 10.69 [03:36:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.11, 10.38, 10.47 [03:37:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [03:38:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.51, 9.40, 10.08 [03:41:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.27, 11.62, 10.92 [03:43:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.05, 10.95, 10.77 [03:51:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 11.36, 11.04 [03:53:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.66, 11.18, 11.03 [04:00:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.67, 10.03, 9.82 [04:00:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.76, 1.28, 1.88 [04:06:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.44, 9.94, 9.92 [04:06:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.17, 1.67 [04:06:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.34, 9.94, 9.36 [04:08:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.14, 9.18, 9.15 [04:09:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [04:10:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.81, 10.51, 10.22 [04:11:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.29, 10.73, 10.49 [04:12:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.89, 9.98, 10.06 [04:13:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.64, 10.27, 10.35 [04:15:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.05, 10.85, 10.55 [04:16:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.24, 11.47, 10.68 [04:17:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.88, 10.56, 10.49 [04:19:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.14, 3.21, 3.94 [04:20:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.86, 11.57, 10.96 [04:21:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.88, 2.95, 3.87 [04:21:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.13, 11.03, 10.66 [04:23:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.01, 10.51, 10.51 [04:24:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [04:25:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.02, 3.65 [04:25:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.04, 3.74, 3.89 [04:29:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.63, 3.77 [04:29:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.78, 9.53, 10.11 [04:33:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.62, 1.93 [04:36:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.70, 8.62, 9.90 [04:37:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.60, 3.95 [04:39:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.54, 3.57 [04:43:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.28, 3.43, 3.57 [04:43:17] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.43, 1.67 [04:47:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.67, 1.88, 1.78 [04:51:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.26, 3.03, 3.34 [04:55:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 2.76, 3.29 [04:59:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.86, 1.93 [05:05:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.82, 3.88, 3.43 [05:07:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.36, 3.78, 3.39 [05:09:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.49, 3.32 [05:09:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.67, 3.42 [05:11:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.80, 1.82 [05:11:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.80, 4.31, 3.62 [05:11:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 3.89, 3.53 [05:13:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.90, 3.58 [05:15:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.72, 4.50, 3.81 [05:16:13] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kachow.org' expires in 15 day(s) (Sat 30 Jul 2022 04:59:24 GMT +0000). [05:17:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a1b4318e85c3...005d413981c6 [05:17:15] [url] Comparing a1b4318e85c3...005d413981c6 · miraheze/ssl · GitHub | github.com [05:17:16] [02miraheze/ssl] 07MirahezeSSLBot 03005d413 - Bot: Update SSL cert for wiki.kachow.org [05:21:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.92, 3.80 [05:22:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.12, 1.96, 1.39 [05:23:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.88, 3.80 [05:23:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 4.30, 3.96 [05:24:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.72, 1.36 [05:26:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.10, 1.55, 1.35 [05:27:22] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [05:27:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.60, 3.77 [05:29:22] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:32:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.17, 2.26, 1.79 [05:33:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.47, 3.95, 3.83 [05:34:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.73, 1.71, 1.64 [05:35:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.62, 3.72 [05:36:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.84, 1.43, 1.54 [05:39:03] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 2.99, 3.36 [05:40:15] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [05:42:13] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:43:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.04, 3.37 [05:45:28] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 9.14, 8.55 [05:46:33] PROBLEM - wiki.jkrstudios.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.82, 4.01, 3.62 [05:47:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.74, 3.55 [05:48:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.79, 8.87, 8.52 [05:49:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.62, 3.52 [05:49:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.36, 3.43 [05:49:47] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw121 [05:49:51] PROBLEM - wiki.dunmoore.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:11] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 2 backends are down. mw111 mw122 [05:51:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.15, 10.94, 9.94 [05:53:03] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.54, 4.45, 3.82 [05:53:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.13, 3.66, 3.52 [05:53:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.69, 10.90, 10.05 [05:55:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.40, 3.44 [05:55:48] PROBLEM - evil.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:32] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:57:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [05:57:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.21, 3.36 [05:59:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.70, 3.68 [06:01:03] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.02, 10.15, 9.10 [06:02:34] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:05:13] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 2 backends are down. mw111 mw112 [06:05:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 10.42, 9.52 [06:05:26] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.06, 2.98, 3.33 [06:05:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [06:06:57] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw101 [06:07:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.67, 9.43, 9.27 [06:09:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.72, 9.01, 9.94 [06:12:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.04, 3.04 [06:13:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.21, 3.49 [06:14:12] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=islamwikiwiki (START) [06:14:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:33] RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Wed 12 Oct 2022 04:17:08 GMT +0000. [06:14:46] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:15:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.32, 3.68, 3.48 [06:15:39] RECOVERY - wiki.jkrstudios.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jkrstudios.org' will expire on Mon 26 Sep 2022 05:51:00 GMT +0000. [06:17:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.73, 3.45 [06:18:43] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw102 [06:19:14] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Sun 02 Oct 2022 10:05:42 GMT +0000. [06:19:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.74, 4.62, 3.85 [06:19:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 4.37, 3.73 [06:20:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.43, 1.15 [06:21:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.98, 3.71 [06:22:10] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:18] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.36, 1.16 [06:22:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.74, 9.81, 8.80 [06:22:47] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 9.61, 4.95, 2.54 [06:23:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 4.46, 3.92 [06:23:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:24:02] PROBLEM - sharqedia.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.75, 9.50, 8.81 [06:24:34] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:37] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=islamwikiwiki (END - exit=0) [06:25:08] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Wed 10 Aug 2022 22:13:25 GMT +0000. [06:28:47] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.31, 3.22, 2.66 [06:29:41] RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Wed 31 Aug 2022 18:13:33 GMT +0000. [06:31:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.84, 3.91 [06:33:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 4.01, 3.94 [06:34:09] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:35:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.85, 3.90 [06:36:05] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:37:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.97, 4.05, 3.98 [06:38:39] PROBLEM - rarewiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:19] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:23] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:41:04] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:15] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3402 bytes in 0.019 second response time [06:41:35] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:45] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [06:43:45] PROBLEM - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:45] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:10] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:17] PROBLEM - cp21 SSH on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:18] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [06:46:07] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 99061 bytes in 0.251 second response time [06:46:13] RECOVERY - cp21 SSH on cp21 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:50:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [06:50:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 9.56, 8.20 [06:50:51] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [06:51:08] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Mon 10 Oct 2022 05:23:51 GMT +0000. [06:52:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 9.11, 8.22 [06:53:11] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Mon 03 Oct 2022 13:40:45 GMT +0000. [06:54:24] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 13 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [07:07:25] RECOVERY - rarewiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewiki.org' will expire on Wed 28 Sep 2022 05:06:19 GMT +0000. [07:08:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.78, 1.29 [07:09:26] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.58, 10.22, 9.08 [07:10:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.58, 1.28 [07:10:43] RECOVERY - mr.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mr.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:58:00 GMT +0000. [07:11:34] RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Tue 23 Aug 2022 00:31:22 GMT +0000. [07:13:02] RECOVERY - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'mcwiki.kirbygang.tk' will expire on Wed 05 Oct 2022 22:00:10 GMT +0000. [07:13:02] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Fri 16 Sep 2022 12:56:22 GMT +0000. [07:13:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 10.68, 9.55 [07:19:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.01, 10.05, 9.71 [07:39:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 10.05, 9.69 [07:41:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.44, 11.26, 10.18 [07:41:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [07:42:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.58, 10.90, 10.21 [07:46:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [07:46:55] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.68, 9.38, 9.76 [07:50:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.71, 9.22, 8.68 [07:54:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.22, 9.74, 9.04 [08:00:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.38, 10.26, 9.56 [08:04:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.77, 9.84, 9.64 [08:26:25] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.84, 1.89, 1.36 [08:28:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.88, 1.57, 1.31 [08:42:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.15, 11.67, 10.00 [08:44:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.77, 10.89, 9.72 [08:44:11] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.72, 10.99, 9.96 [08:46:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.93, 12.03, 10.28 [08:52:06] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.40, 11.68, 10.61 [08:54:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.06, 10.02, 8.90 [08:58:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.88, 10.99, 9.49 [09:00:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.83, 10.10, 9.34 [09:02:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [09:05:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.69, 11.88, 11.81 [09:06:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.86, 11.13, 10.04 [09:07:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.45, 12.09, 11.89 [09:10:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.81, 9.99, 9.80 [09:13:49] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 11.62, 11.80 [09:14:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.76, 10.85, 10.19 [09:14:58] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [09:15:48] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.93, 12.15, 11.97 [09:17:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [09:17:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.80, 11.39, 11.72 [09:20:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.71, 10.11, 10.10 [09:23:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.54, 11.21, 11.82 [09:31:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.35, 10.42, 10.52 [09:33:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.83, 9.45, 10.16 [09:34:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.48, 3.90 [09:37:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 10.43, 10.40 [09:37:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.62, 11.16, 10.94 [09:38:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 10.34, 9.93 [09:39:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.23, 11.17, 10.68 [09:40:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 3.88 [09:41:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.59, 10.76, 10.56 [09:43:11] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:46:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.07, 9.86, 9.95 [09:47:32] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.20, 12.12, 11.17 [09:51:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 11.79, 11.31 [09:53:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 11.24, 11.62 [09:54:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.73, 3.98 [09:57:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.50, 3.91 [09:57:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.21, 1.66, 1.94 [09:58:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.92, 3.99 [09:59:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.73, 4.11, 4.09 [10:00:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.85, 3.96 [10:01:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.99, 10.72, 10.84 [10:02:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 4.23, 4.08 [10:03:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.71, 11.26, 11.04 [10:06:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.20, 11.47, 10.10 [10:07:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.81, 3.99 [10:07:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.77, 12.49, 11.56 [10:08:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 10.71, 9.97 [10:09:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.62, 4.76, 4.31 [10:11:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.57, 12.03, 11.25 [10:14:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.15, 10.16, 9.96 [10:16:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.95, 10.05, 9.31 [10:16:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.60, 3.94 [10:17:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.78, 1.78 [10:18:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.88, 9.87, 9.33 [10:20:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 4.21, 4.09 [10:21:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.59, 3.97 [10:21:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 11.97, 11.69 [10:22:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.87, 3.98 [10:24:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.84, 3.94 [10:25:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.83, 3.98 [10:25:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.69, 11.74, 11.94 [10:25:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.48, 12.48, 11.89 [10:29:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 11.84, 11.79 [10:31:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.18, 12.25, 11.99 [10:33:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.10, 10.80, 10.01 [10:35:16] Reception123: renew wildcards [10:35:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.50, 10.94, 10.02 [10:35:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.27, 11.49, 11.56 [10:37:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.62, 10.23, 9.89 [10:38:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.69, 3.98 [10:39:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.14, 11.01, 10.41 [10:39:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.62, 3.96 [10:41:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.39, 4.32, 4.18 [10:44:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.77, 4.47, 4.12 [10:45:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 9.50, 9.98 [10:45:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.90, 9.93, 9.95 [10:47:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.62, 11.56, 11.92 [10:48:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.79, 3.91 [10:49:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 10.50, 10.17 [10:49:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 11.16, 11.88 [10:51:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.57, 3.92 [10:51:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.09, 11.81, 11.93 [10:53:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.94, 11.57, 11.81 [10:54:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [10:54:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.26, 4.05 [10:55:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.44, 4.01, 4.01 [10:59:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [11:00:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.52, 9.98 [11:00:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.67, 11.11, 11.15 [11:02:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.25, 11.07, 11.13 [11:06:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 11.12, 10.45 [11:06:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.77, 11.87, 11.41 [11:07:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.44, 11.60, 11.40 [11:08:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.19, 11.11, 10.53 [11:08:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.69, 11.68, 11.39 [11:10:33] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/311a0068fb69...dc0123480194 [11:10:34] [url] Comparing 311a0068fb69...dc0123480194 · miraheze/MirahezeMagic · GitHub | github.com [11:10:35] [02miraheze/MirahezeMagic] 07translatewiki 03dc01234 - Localisation updates from https://translatewiki.net. [11:10:35] [url] Main page - translatewiki.net | translatewiki.net [11:10:53] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.57, 11.88, 11.50 [11:12:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.79, 11.12, 11.25 [11:13:25] miraheze/MirahezeMagic - translatewiki the build has errored. [11:14:52] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.14, 12.32, 11.68 [11:16:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 11.29, 10.79 [11:18:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.92, 10.72 [11:26:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.57, 11.83, 11.92 [11:30:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.85, 10.72, 10.65 [11:30:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.36, 11.87, 11.85 [11:31:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.78, 2.00 [11:32:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.03, 10.29, 10.50 [11:35:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.81, 10.25, 9.66 [11:36:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.77, 11.57, 11.75 [11:37:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.75, 9.50, 9.47 [11:40:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.34, 9.96 [11:40:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.01, 12.12, 11.91 [11:41:17] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.47, 1.43, 1.69 [11:42:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.26, 11.63, 11.75 [11:43:22] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:25] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:48] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:07] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.64, 1.72 [11:45:31] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.685 second response time [11:45:33] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.461 second response time [11:45:53] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.631 second response time [11:46:06] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.335 second response time [11:46:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.62, 11.08, 10.40 [11:47:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.82, 1.78 [11:48:36] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.56, 12.15, 11.86 [11:50:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.79, 11.23, 11.54 [11:52:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.65, 11.09, 10.80 [11:52:34] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.86, 11.96, 11.77 [11:54:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.97, 11.98, 11.80 [11:54:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.61, 3.99 [11:56:20] Reception123: renew wildcards [11:57:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.92, 1.97 [11:57:22] Ah yes, thanks! [11:57:27] Will do in ~20 [11:57:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [11:57:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.51, 10.78, 11.93 [11:59:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.25, 11.61, 12.08 [12:00:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.19, 8.85, 9.90 [12:00:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.38, 12.10, 11.79 [12:01:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.99, 1.98 [12:02:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.04, 11.76, 11.71 [12:02:41] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.73, 1.51, 1.15 [12:03:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.92, 1.95 [12:03:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 11.51, 11.95 [12:04:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.65, 1.48, 1.18 [12:05:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.99, 1.98 [12:05:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.23, 11.68, 11.95 [12:06:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 10.91, 10.37 [12:06:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.84, 11.86, 11.72 [12:07:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.90, 1.94 [12:07:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.45, 11.43, 11.84 [12:08:41] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 4.04, 3.97 [12:09:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 2.14, 2.03 [12:09:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.69, 12.50, 12.17 [12:10:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 10.92, 10.51 [12:11:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.32, 9.81, 8.55 [12:13:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.36, 9.65, 8.65 [12:20:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 11.28, 10.74 [12:33:12] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:29] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:45] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:34:52] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:18] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.53, 11.21, 10.14 [12:38:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.29, 10.30 [12:40:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.31, 11.99, 11.99 [12:40:42] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.692 second response time [12:40:59] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.591 second response time [12:41:12] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.488 second response time [12:41:23] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.663 second response time [12:42:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 12.32, 12.11 [12:42:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.06, 10.18, 10.08 [12:42:23] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:28] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [12:42:28] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:24] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:52] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:09] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:30] [discord] https://meta.miraheze.org/wiki/Special:RequestWikiQueue/25669#mw-section-request Just wanted to warn you guys that when I accepted someone's wiki request, an error has occurred. [12:47:30] [url] Wiki requests queue - Miraheze Meta | meta.miraheze.org [12:47:59] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.519 second response time [12:48:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 11.07, 11.66 [12:48:15] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.289 second response time [12:48:54] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.711 second response time [12:49:51] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.934 second response time [12:50:57] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.880 second response time [12:51:23] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:31] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.69, 11.78 [12:52:50] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:54] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:17] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:26] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.13, 1.58, 1.05 [12:54:52] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.496 second response time [12:55:00] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.052 second response time [12:55:19] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.417 second response time [12:55:25] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.669 second response time [12:55:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.27, 0.99 [12:55:38] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f6d694ec8efc...2d10492c0c96 [12:55:38] [url] Comparing f6d694ec8efc...2d10492c0c96 · miraheze/dns · GitHub | github.com [12:55:39] [02miraheze/dns] 07Reception123 032d10492 - acme challenge for miraheze.com [12:55:41] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.284 second response time [12:56:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.55, 11.50, 11.72 [12:57:19] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2d10492c0c96...e559f872dff9 [12:57:19] [url] Comparing 2d10492c0c96...e559f872dff9 · miraheze/dns · GitHub | github.com [12:57:20] [02miraheze/dns] 07Reception123 03e559f87 - acme challenge for miraheze.com [12:58:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.64, 12.03, 11.89 [12:59:58] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/005d413981c6...8c58866af12b [12:59:58] [url] Comparing 005d413981c6...8c58866af12b · miraheze/ssl · GitHub | github.com [12:59:59] [02miraheze/ssl] 07Reception123 038c58866 - renew *.miraheze.com [13:00:31] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e559f872dff9...be1304ecb86b [13:00:32] [url] Comparing e559f872dff9...be1304ecb86b · miraheze/dns · GitHub | github.com [13:00:33] [02miraheze/dns] 07Reception123 03be1304e - acme challenge for miraheze.wiki [13:02:23] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/be1304ecb86b...fcd7409fcc0f [13:02:24] [url] Comparing be1304ecb86b...fcd7409fcc0f · miraheze/dns · GitHub | github.com [13:02:25] [02miraheze/dns] 07Reception123 03fcd7409 - acme challenge for miraheze.wiki [13:03:10] !log sudo apt-get upgrade on mw*/mwtask111 [13:03:11] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:21] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:23] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:25] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [13:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:57] PROBLEM - mw112 APT on mw112 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [13:04:34] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:05:06] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:05:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.08, 10.40, 9.48 [13:05:10] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 4.826 second response time [13:05:11] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:05:21] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.413 second response time [13:05:22] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.095 second response time [13:05:39] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fcd7409fcc0f...012a4dee4bfa [13:05:40] [url] Comparing fcd7409fcc0f...012a4dee4bfa · miraheze/dns · GitHub | github.com [13:05:41] [02miraheze/dns] 07Reception123 03012a4de - acme challenge (again) for miraheze.wiki [13:05:58] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.17, 11.93, 10.75 [13:05:58] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:07:11] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.260 second response time [13:07:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c58866af12b...4c8d883f9466 [13:07:30] [url] Comparing 8c58866af12b...4c8d883f9466 · miraheze/ssl · GitHub | github.com [13:07:31] [02miraheze/ssl] 07Reception123 034c8d883 - renew miraheze.wiki [13:07:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.35, 11.13, 10.58 [13:09:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 10.71, 9.77 [13:09:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.64, 1.95 [13:13:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.64, 12.57, 11.30 [13:15:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.92, 11.09, 10.33 [13:15:13] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:15:29] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:15:54] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:17:06] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:17:27] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:17:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.25, 10.78, 11.95 [13:19:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.72, 2.02, 1.93 [13:21:28] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:23:09] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:23:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.01, 11.55, 11.53 [13:24:27] https://www.irccloud.com/pastebin/U0IR73A4/ [13:24:30] ^ MacFan4000 [13:25:01] hope this won't happen every time we do it [13:25:26] Strange [13:25:51] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:26:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.25, 10.90, 11.88 [13:27:09] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:27:17] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.80, 1.92 [13:27:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 11.39, 11.44 [13:29:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.99, 9.27, 9.94 [13:29:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.85, 11.57, 11.50 [13:32:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.65, 12.16, 12.00 [13:33:38] MacFan4000: yeah, anyway I pulled and it's fine now but if it happens again there's probably something wrong [13:34:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 10.09, 11.67 [13:35:03] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 11.88, 11.64 [13:35:40] RECOVERY - ns2 APT on ns2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:36:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.11, 11.86 [13:36:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.36, 11.90, 11.94 [13:37:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 10.95, 11.35 [13:37:09] !log sudo apt-get upgrade on ns*/cp*/puppet111 [13:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:27] db* and gluster* I'm leaving to infra as last time didn't go well at all [13:37:28] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:37:47] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.29, 8.81, 10.18 [13:37:57] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: connect to address poserdazfreebies.orain.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:38:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.27, 9.21, 11.10 [13:38:26] RECOVERY - cp30 APT on cp30 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:38:43] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:38:43] PROBLEM - cp30 HTTPS on cp30 is CRITICAL: connect to address 2607:5300:201:3100::929a and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:38:45] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: connect to address allthetropes.orain.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:38:52] RECOVERY - cp20 APT on cp20 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:39:23] RECOVERY - cp31 APT on cp31 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:39:27] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:39:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:39:38] paladox: around? not sure what could've gone wrong with an apt-upgrade again but ns seems down :( [13:39:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:39:46] PROBLEM - wiki.aoemods.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.aoemods.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:46] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.whentheycry.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:46] https://www.irccloud.com/pastebin/4b3G4gW6/ [13:39:49] cc MacFan4000 RhinosF1 [13:39:49] PROBLEM - eng.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address eng.archiopedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:51] PROBLEM - microworldwiki.ga - LetsEncrypt on sslhost is CRITICAL: connect to address microworldwiki.ga and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:51] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address permanentfuturelab.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:51] PROBLEM - hellointernet.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address hellointernet.miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:51] PROBLEM - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is CRITICAL: connect to address marinebiodiversitymatrix.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:54] PROBLEM - vrcdev.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address vrcdev.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:55] PROBLEM - www.sdiy.info - LetsEncrypt on sslhost is CRITICAL: connect to address www.sdiy.info and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:39:57] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address crocwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:03] PROBLEM - gametheorywiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address gametheorywiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 1.45, 6.46, 9.86 [13:40:16] PROBLEM - pastport.org - LetsEncrypt on sslhost is CRITICAL: connect to address pastport.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:20] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: connect to address 2001:41d0:801:2000::1b80 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:40:27] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.globasa.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:27] PROBLEM - housing.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address housing.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 1.36, 6.57, 9.80 [13:40:34] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address it.famepedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:34] PROBLEM - it.famepedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address it.famepedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:40] PROBLEM - mr.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address mr.gyaanipedia.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:44] PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.openhatch.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:45] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.mxlinuxusers.de and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:47] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.elgeis.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:48] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address data.nonbinary.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:49] PROBLEM - wiki.pacebi.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.pacebi.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:40:51] Reception123: what got upgraded? [13:40:52] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: connect to address 2001:41d0:801:2000::4c25 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:41:02] base-files bash dirmngr distro-info-data dpkg dpkg-dev gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv libdpkg-perl [13:41:03] libfreetype6 libfribidi0 libgnutls30 libsmbclient libssl1.1 libwbclient0 linux-libc-dev logrotate nano openssl postfix python3-samba samba-common samba-common-bin [13:41:03] samba-dsdb-modules samba-libs smbclient tcpdump [13:41:03] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.mineland.eu and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 1.16, 5.69, 9.11 [13:41:03] PROBLEM - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.umbranet.mandelbrotgames.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:05] Yeah full outage [13:41:06] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: connect to address m.miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:07] PROBLEM - mariopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address mariopedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:10] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: connect to address antiguabarbudacalypso.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:10] PROBLEM - dkwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address dkwiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:10] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.worldsofweary.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:12] PROBLEM - ethene.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ethene.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:12] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.anglish.info and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:12] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.leapit.no and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:13] PROBLEM - www.electrowiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.electrowiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:13] PROBLEM - correypedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address correypedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:21] Reception123: try a reboot [13:41:26] !log reboot ns2 [13:41:26] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 0.87, 2.78, 3.98 [13:41:33] yep, just thought of that [13:41:33] PROBLEM - cp31 HTTPS on cp31 is CRITICAL: connect to address 2607:5300:201:3100::5ebc and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [13:41:34] Yeah we're down [13:41:36] PROBLEM - arrieretoile.fr - LetsEncrypt on sslhost is CRITICAL: connect to address arrieretoile.fr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:37] PROBLEM - wc.miraheze.org on sslhost is CRITICAL: connect to address miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:37] PROBLEM - kunwok.org - LetsEncrypt on sslhost is CRITICAL: connect to address kunwok.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:40] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is CRITICAL: connect to address wikiru.wildterra2.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:42] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address equestripedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:45] PROBLEM - finalfantasy.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address finalfantasy.miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:45] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is CRITICAL: connect to address guia.esporo.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:54] waiting for ns2 to respond before trying reboot on ns1 [13:41:54] PROBLEM - www.opendatascot.org - LetsEncrypt on sslhost is CRITICAL: connect to address www.opendatascot.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:54] PROBLEM - wiki.parappa.dev - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.parappa.dev and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:55] PROBLEM - rct.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address rct.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:41:59] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.climatechange.ai and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:04] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.meregos.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:04] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.lgbtqia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:05] PROBLEM - thepolicyhub.org.uk - LetsEncrypt on sslhost is CRITICAL: connect to address thepolicyhub.org.uk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:06] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is CRITICAL: connect to address wiki-asterix.cf and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:22] PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address grc.repository.archiopedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:32] PROBLEM - lgbta.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address lgbta.miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:32] PROBLEM - rare.miraheze.org - Sectigo on sslhost is CRITICAL: connect to address rare.miraheze.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:32] PROBLEM - lg.gyaanipedia.com - LetsEncrypt on sslhost is CRITICAL: connect to address lg.gyaanipedia.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:42:41] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fbpml.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:41] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nevillepedia.eu and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:42] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.mcpk.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:52] .op [13:42:52] Attempting to OP... [13:42:56] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.permanentfuturelab.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:57] PROBLEM - wiki.kourouklides.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.kourouklides.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:57] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fossbots.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:42:57] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address grk.repository.archiopedia.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:00] PROBLEM - wiki.phgalaxy.eu.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.phgalaxy.eu.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:00] PROBLEM - www.turtletown.ca - LetsEncrypt on sslhost is CRITICAL: connect to address www.turtletown.ca and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:00] PROBLEM - trollpasta.com - LetsEncrypt on sslhost is CRITICAL: connect to address trollpasta.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:02] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is CRITICAL: connect to address beaconspace.unrestrictedlorefare.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 0.76, 2.24, 3.84 [13:43:04] PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address puzzles.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:07] PROBLEM - gs.sidem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address gs.sidem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:08] PROBLEM - docse.tk - LetsEncrypt on sslhost is CRITICAL: connect to address docse.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:12] RhinosF1: doesn't seem to work [13:43:13] Reception123: did you reboot it? [13:43:16] [discord] Miraheze is down? [13:43:16] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.53, 1.70 [13:43:17] PROBLEM - wiki.erag.eu.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.erag.eu.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:17] PROBLEM - mcwiki.kirbygang.tk - LetsEncrypt on sslhost is CRITICAL: connect to address mcwiki.kirbygang.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:17] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.rosestulipsandliberty.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:21] paladox: yes, just rebooted ns2 [13:43:25] PROBLEM - thenintendowiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address thenintendowiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:25] PROBLEM - wiki.wilderyogi.eu - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.wilderyogi.eu and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:26] PROBLEM - bn.wikiyri.org - LetsEncrypt on sslhost is CRITICAL: connect to address bn.wikiyri.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:27] PROBLEM - zenith.nsmbu.net - LetsEncrypt on sslhost is CRITICAL: connect to address zenith.nsmbu.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:27] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.danvs.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:27] PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address en.pornwiki.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:27] PROBLEM - datawiki.online - LetsEncrypt on sslhost is CRITICAL: connect to address datawiki.online and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:28] Yes we're down [13:43:29] [discord] Ah ok [13:43:31] PROBLEM - fanon.polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address fanon.polandballwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:43:31] I meant ns1 [13:43:32] doesn't seem to have helped [13:43:36] no, will do ns1 too now I gues [13:43:38] *guess [13:43:42] No don’t [13:43:47] Reception123: no [13:43:51] Oh, ok [13:43:55] Is ns2 still an issue after reboot [13:44:00] One ns server must be up at any time [13:44:05] That's what I initially thought, to wait for ns2 to see if the result is the same [13:44:18] and ns2 seems to still be having issues [13:44:22] paladox: we are completely down [13:44:22] keep seeing [13:44:24] channel 3: open failed: connect failed: Connection timed out [13:44:24] channel 5: open failed: connect failed: Connection timed out [13:44:24] channel 4: open failed: connect failed: Connection timed out [13:44:29] Both ns servers are having issues [13:44:48] That's the connection between bast and ns I think [13:45:12] I can't really see anything that was upgraded that would cause issues [13:45:29] ns1 works for me [13:45:43] i'm ssh'd in and gdnsd is running [13:45:46] well yes I can ssh to them but otherwise it keeps saying connection timed out when you're inside [13:45:53] (at least for me that's what ns2 says) [13:46:07] https://www.irccloud.com/pastebin/p3zAUC7x/ [13:46:07] Reception123: do we know that ns is definitely causing the outage [13:46:20] RhinosF1: I mean it can't be a coincidence for sure [13:46:29] it said [13:46:29] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:46:31] https://www.irccloud.com/pastebin/lWTzKXs9/ [13:46:40] Reception123: I mean the errors we're seeing don't look like dns related at all [13:46:45] but I still don't see what the upgrades could've done as there weren't any critical things upgraded [13:46:55] I did also upgrade cp* [13:47:03] nginx is failing [13:47:15] https://www.irccloud.com/pastebin/9KqL79nE/ [13:47:24] oh because puppet didn't run after I fixed that [13:47:35] could it be that because nginx was failing running upgrades on cp* caused them to go down? [13:47:49] Reception123: yes [13:47:54] cp* crashed [13:48:06] nothing to do with ns at all [13:48:09] though I did manually run puppet on one cp in order to make sure that it was working so shouldn't that still be working then? [13:48:23] but yes, it clearly seems to be cp [13:48:25] Reception123: nginx isn't reloaded after 1st attempt again [13:48:32] and never has been [13:48:42] You need to manually run reload steps if they fail [13:48:56] runnig puppet [13:49:16] Reception123: did you definitely get the private keys right [13:49:39] https://www.irccloud.com/pastebin/GQBHG22H/ [13:49:40] at [13:49:42] *wat [13:49:51] going to have to reinstall nginx [13:49:56] What [13:50:06] RhinosF1: they should've been yes, the issue was that after MacFan did that change /home/* said that it was behind [13:50:11] so I just did git pull then git push [13:50:16] !log reinstall nginx on cp31 [13:50:28] Reception123: and are the keys in the final destination actually right [13:50:29] then I ran puppet manually on cp30 and it was fine [13:50:34] RhinosF1: I'll check that now [13:50:37] puppet will be fine [13:50:44] Reload steps don't happen more than once [13:51:43] yeah that was a bad test [13:52:02] but private keys shouldn't be different when upgrading wildcards [13:52:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4fd74dc35d68...b0c61365d0ef [13:52:34] [url] Comparing 4fd74dc35d68...b0c61365d0ef · miraheze/puppet · GitHub | github.com [13:52:35] [02miraheze/puppet] 07paladox 03b0c6136 - nginx: add missing fastcgi_param to fastcgi_params [13:52:43] that would indicate the private key has changed Reception123 [13:52:52] it can happen sometimes [13:53:20] judging from the end part it's the same [13:53:53] For both the public and private key Reception123 [13:54:22] I remember there was a command to check if they matched [13:54:26] but not sure what it is [13:55:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9b921498f9e2 [13:55:44] [02miraheze/puppet] 07paladox 039b92149 - nginx: set types_hash_max_size to 2048 [13:55:45] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [13:55:45] [url] Page not found · GitHub · GitHub | github.com [13:55:47] [02puppet] 07paladox opened pull request 03#2696: nginx: set types_hash_max_size to 2048 - 13https://github.com/miraheze/puppet/pull/2696 [13:55:47] [url] Page not found · GitHub · GitHub | github.com [13:55:55] [02puppet] 07paladox closed pull request 03#2696: nginx: set types_hash_max_size to 2048 - 13https://github.com/miraheze/puppet/pull/2696 [13:55:55] [url] Page not found · GitHub · GitHub | github.com [13:55:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b0c61365d0ef...59ff0c0774a8 [13:55:57] [url] Comparing b0c61365d0ef...59ff0c0774a8 · miraheze/puppet · GitHub | github.com [13:55:58] [02miraheze/puppet] 07paladox 0359ff0c0 - nginx: set types_hash_max_size to 2048 (#2696) [13:55:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [13:56:01] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [13:56:01] [url] Page not found · GitHub · GitHub | github.com [13:56:44] seems to not match yeah [13:56:47] but that doesn't make sense [13:57:56] If somebody is trying to make icingabot work it is because I +q’d it [13:58:22] !log reinstall nginx on cp20/21 [13:58:27] Reception123: did you not say they didn't push [13:58:37] ffs [13:58:40] it's still broken [13:58:43] RhinosF1: he then fixed it [13:58:43] the key [13:58:48] maybe I'm using a wrong command but even the old public key doesn't seem to match [13:58:58] but yes, I did pull changes and then pushed and there was no error [13:59:09] paladox: the key will still be broken [13:59:19] yes [13:59:26] and why is it reloading if it's broken [13:59:27] I don't know what's going on with that key really, but it doesn't make sense that it doesn't seem to match the old cert either [13:59:43] paladox: well if you've just reinstalled it then it has to restart [14:00:04] paladox, Reception123: can we just create the key from fresh again [14:00:10] And make sure it actually pushes fine [14:00:19] if you want paladox I can regenerate it [14:00:22] no [14:00:29] puppet ran successfully first time [14:01:12] If the key isn't matching, nginx will never come back up [14:01:24] you'll need to regenerate the key then [14:01:49] paladox: should I redo it then? [14:02:19] That's what I just said [14:02:28] .com is the issue right? [14:02:29] yeh [14:02:32] no [14:02:33] .wiki [14:02:33] paladox: What do you want me to look up? [14:02:34] ah no, .wiki [14:02:48] the keys look fine to me (like the pub and priv key) [14:03:18] paladox: hmm it generated without challenges [14:03:22] Reception123: did you do /root/ssl-certificate -d miraheze.wiki -s *miraheze.wiki -r -w -o? [14:03:35] I did /root/ssl-certificate -d *.miraheze.wiki -s miraheze.wiki -g -p -o -w --no-use-key [14:03:49] the output is weird though [14:03:51] why is bots involved? [14:03:53] https://www.irccloud.com/pastebin/G21uUup4/ [14:03:53] err typo [14:04:15] huh [14:04:23] yeh [14:04:28] don't do -d *.miraheze.wiki [14:04:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c8d883f9466...52646548d1a5 [14:04:31] [url] Comparing 4c8d883f9466...52646548d1a5 · miraheze/ssl · GitHub | github.com [14:04:32] [02miraheze/ssl] 07Reception123 035264654 - Update miraheze.wiki.crt [14:04:48] paladox: oh, you mean it's the opposite? but when I tried that it said something about changing the domains [14:05:07] hold on let me do it [14:05:12] i'll replace the key as well [14:05:16] *private [14:05:41] ok. I thought the key would be though with --no-use-key [14:06:33] We used to have a seperate cert for *.bots.mh.wiki [14:06:41] It was removed in https://github.com/miraheze/ssl/commit/571dc6a14af10f76b8f0cbc075d8f13a7ce8c420 [14:06:42] [url] remove legacy bots redirects (#435) · miraheze/ssl@571dc6a · GitHub | github.com [14:06:55] Which means the private key never got removed [14:06:59] oh damn [14:07:01] There were too many requests of a given type :: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: *.miraheze.wiki,miraheze.wiki: see https://letsencrypt.org/docs/duplicate-certificate-limit/ [14:07:01] [url] Duplicate Certificate Limit - Let's Encrypt | letsencrypt.org [14:07:06] and i deleted the cert wat [14:07:26] urgh [14:07:29] It is uasually a good idea to remove private keys as well and not just public when removing certs [14:08:08] paladox: can we disable Miraheze.wiki [14:08:13] https://www.irccloud.com/pastebin/N5tTK8wq/ [14:08:13] paladox: worst case I'd say remove miraheze.wiki completely until we figure it out [14:08:17] I know it's insane because that breaks phab [14:08:21] we can remove it yeh (it'll break phabs ui tho [14:08:24] But otherwise we are down for a week [14:08:25] yeah, it's better than breaking wikis [14:08:32] i can try and see if the workaround works [14:08:34] so we should sacrifice Phab for the time being [14:08:43] I'd say phab being broken for a week is better than everything [14:08:54] Maybe we can move phab temp to another domain [14:09:11] Technically you could temp fix phab by disabling the alternate file domain [14:09:15] yeah I wouldn't worry about that, that can be done [14:09:16] It isn’t required [14:09:20] right now we just need things back up [14:09:20] paladox: you can try adding another domain [14:09:24] As it says exact set [14:09:26] ok it's doing [14:09:31] gotta add the acme challenge [14:09:41] So if you issue a cert for blah.Miraheze.wiki, *.mh.wiki etc [14:09:45] It should work [14:10:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/012a4dee4bfa...23c7bbbaad5d [14:10:46] [url] Comparing 012a4dee4bfa...23c7bbbaad5d · miraheze/dns · GitHub | github.com [14:10:47] [02miraheze/dns] 07paladox 0323c7bbb - miraheze.wiki add acme challenge [14:12:42] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/52646548d1a5...3790fd2b076e [14:12:43] [url] Comparing 52646548d1a5...3790fd2b076e · miraheze/ssl · GitHub | github.com [14:12:44] [02miraheze/ssl] 07paladox 033790fd2 - Update miraheze.wiki.crt [14:13:20] lets try now... [14:15:38] hmm [14:15:41] it still fails?! [14:16:01] didn't replace the private key? [14:16:14] That's been the issue all along [14:16:34] nginx: [emerg] SSL_CTX_use_PrivateKey("/etc/ssl/private/miraheze.com.key") failed (SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch) [14:16:45] Yes [14:16:57] That's .com though [14:17:00] Not .wiki [14:17:07] so I guess both didn't get pushed [14:17:11] Both needed renewing [14:17:34] [discord] Am around if extra help is needed [14:17:43] You'll have to issue a new .com too paladox [14:17:45] good thing I didn't do any more [14:17:51] but yeah I guess something bad happened with .com too [14:18:06] [discord] SSL certificates are just broken [14:18:09] hold on [14:18:14] [discord] And nginx got restarted [14:19:05] I just did git reset —hard in /etc/puppetlabs/puppet/ssl-keys. Should be good now? [14:19:19] thanks! [14:19:38] was wondering why it didn't pick up the private key! [14:20:33] somehow there were uncommitted changes as well as an untracked .swp file [14:21:07] [discord] .swp is unsaved text editor changes, no? [14:21:41] [discord] *.swp should be on .gitignore [14:21:45] [discord] Yep [14:22:54] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3790fd2b076e...68cefed7b0c1 [14:22:54] [url] Comparing 3790fd2b076e...68cefed7b0c1 · miraheze/ssl · GitHub | github.com [14:22:55] [02miraheze/ssl] 07paladox 0368cefed - Update miraheze.com.crt [14:24:05] Just ran again [14:24:24] ok so there seems to be a problem [14:24:28] Somehow when private keys are updated we get uncommitted changes [14:24:39] when pushing to /etc/puppetlabs/puppet/ssl-keys you have to git reset that dir after [14:24:52] It seems so [14:24:56] [discord] We didnt use to? What changed? [14:25:21] it means it's broken [14:25:56] well the only thing that was changed is https://github.com/miraheze/puppet/commit/031f3157c8770501ef9135cdafa4664dc8365408 [14:25:57] [url] Manage /home/ssl-admins recursively + ensure existence and permission… · miraheze/puppet@031f315 · GitHub | github.com [14:26:05] (and I ended up reverting the recurse part due to errors) [14:26:27] [discord] Maybe we need to revert the whole commit? [14:26:29] That was slightly changed [14:26:58] [discord] Are we back? [14:27:25] Seems like it [14:28:05] it work snow [14:28:08] *now [14:28:37] I wonder if it's because Reception123 dropped the recurse on /home [14:28:45] So the permissions are now not in sync [14:28:58] git reset will change it to match /home [14:28:59] [discord] I think we def need a incident report on this [14:29:07] [discord] We were down for 30 mins [14:29:14] And puppet will revert back [14:30:38] Between both locations the perms are different [14:30:50] They shouldn't be [14:30:54] They need to be in sync [14:31:15] In /home it’s root:ssl-admins [14:31:31] It's more the mask [14:31:38] https://github.com/miraheze/ssl/search?q=wiki.rtapp.tk [14:31:38] [url] Search · wiki.rtapp.tk · GitHub | github.com [14:31:42] Isn't one 0660 and one 0770 [14:31:43] we shouldn't have duplicates [14:31:51] one defined in certs.yaml and one in redirects [14:31:58] should we remove the one from certs? [14:32:03] In /etc it’s macfan:ssl-admins (no idea how that happened) [14:32:26] [discord] Macfan owns /etc? [14:32:30] paladox: is it causing any issues right now [14:32:46] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/2bc7af465b31 [14:32:47] [02miraheze/ssl] 07paladox 032bc7af4 - Remove wiki.rtapp.tk from certs.yaml [14:32:49] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [14:32:49] [url] Page not found · GitHub · GitHub | github.com [14:32:50] [02ssl] 07paladox opened pull request 03#565: Remove wiki.rtapp.tk from certs.yaml - 13https://github.com/miraheze/ssl/pull/565 [14:32:51] [url] Page not found · GitHub · GitHub | github.com [14:32:54] No /etc/puppetlabs/puppet/ssl-keys [14:32:56] RhinosF1: no the domain is being ignored [14:33:08] https://www.irccloud.com/pastebin/QdDjRs0h/ [14:33:21] paladox: can we focus on fixing the impossible push then [14:33:29] and the ssl script adds everything as root:root [14:33:35] [discord] Macfan try changing to match the other [14:33:39] RhinosF1: it's fixed.. [14:33:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/68cefed7b0c1...75121e0065b4 [14:33:46] [url] Comparing 68cefed7b0c1...75121e0065b4 · miraheze/ssl · GitHub | github.com [14:33:47] [02miraheze/ssl] 07MirahezeSSLBot 0375121e0 - Bot: Update SSL cert for pwiki.drydraytonvillagehall.org.uk [14:33:50] paladox: you never merged [14:33:56] merged what [14:34:00] i fixed the certificatesw [14:34:02] oh [14:34:06] paladox: removing the cert [14:34:09] you talking about pushing the private certs [14:34:13] Yes [14:34:18] That's what likely caused the outage [14:34:29] We still should focusing on the root cause [14:34:34] Otherwise we risk a repeat [14:34:39] i doubt that tbh [14:34:49] i checked both the cert in /home/ssl* and also on the machine [14:34:51] both the same [14:34:57] it was Reception123 did the cert wrong [14:35:15] Given that nginx said the private key and public key didn't match, they can't have been pushed right [14:35:34] well it both looked the same to me [14:35:54] They can't have been [14:36:01] thing is that mismatched keys have happend in the past due to one being pulled before the other and it hasn’t caused full outages [14:36:02] hence why i said to do a new certificate... [14:36:13] MacFan4000: normally nginx guards it [14:36:24] Nginx got restarted and then failed to start [14:36:34] it shouldn't have restarted [14:36:43] we have a check that if nginx -t fails it doesn't restart [14:36:50] The puppet reload mechanism is supposed to protect against it [14:37:49] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/75121e0065b4...234d9dcedc84 [14:37:49] [url] Comparing 75121e0065b4...234d9dcedc84 · miraheze/ssl · GitHub | github.com [14:37:50] [02miraheze/ssl] 07paladox 03234d9dc - Remove wiki.rtapp.tk from certs.yaml (#565) [14:37:52] [02ssl] 07paladox closed pull request 03#565: Remove wiki.rtapp.tk from certs.yaml - 13https://github.com/miraheze/ssl/pull/565 [14:37:52] [url] Page not found · GitHub · GitHub | github.com [14:37:53] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [14:37:53] Reception123: what did you do upgrading cp* [14:37:55] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [14:37:55] [url] Page not found · GitHub · GitHub | github.com [14:38:01] I have to think that it could have something to do with the apt upgrades that reception did [14:38:10] Yes it 100% did [14:38:34] yeah, definitely but I don't see which one as they didn't seem very related [14:38:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/e49aec38093d [14:38:35] [02miraheze/ssl] 07paladox 03e49aec3 - Remove wiki.recaptime.tk from certs.yaml [14:38:37] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [14:38:37] [url] Page not found · GitHub · GitHub | github.com [14:38:38] [discord] So we need to investigate reception since they are the root cause 😂 [14:38:38] [02ssl] 07paladox opened pull request 03#566: Remove wiki.recaptime.tk from certs.yaml - 13https://github.com/miraheze/ssl/pull/566 [14:38:39] ... [14:38:55] Reception123: did nginx get restarted though [14:39:04] oh I see, nginx got upgraded [14:39:06] so that would be why then [14:39:09] Yes [14:39:19] I don't know what happened with the certs and the keys though [14:39:22] Don't upgrade nginx while doing ssl renewals [14:39:34] Reception123: your ssl renewal can't have been pushed properly [14:39:46] [02ssl] 07paladox closed pull request 03#566: Remove wiki.recaptime.tk from certs.yaml - 13https://github.com/miraheze/ssl/pull/566 [14:39:47] ... [14:39:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/234d9dcedc84...b5207dd99dce [14:39:48] [url] Comparing 234d9dcedc84...b5207dd99dce · miraheze/ssl · GitHub | github.com [14:39:49] [02miraheze/ssl] 07paladox 03b5207dd - Remove wiki.recaptime.tk from certs.yaml (#566) [14:39:51] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [14:39:52] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [14:39:52] Because you upgraded nginx while it was in a bad state it got restarted [14:39:53] [url] Page not found · GitHub · GitHub | github.com [14:39:56] .op [14:39:56] Attempting to OP... [14:40:22] Reception123: the cause of numerous of our outages has been doing too much at once [14:40:35] We need to stop trying to multi-task [14:41:37] oh found it [14:41:54] Found what [14:43:11] nvm [14:43:44] Icinga will auto unquiet in ~1 min [14:43:54] MacFan4000: what's the best way to make things oushable [14:44:10] not sure why it’s happening [14:44:29] MacFan4000: likely that the permissions (mainly the mask) don't match up [14:44:56] RECOVERY - dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Mon 22 Aug 2022 22:44:29 GMT +0000. [14:44:56] But we couldn't get it to work via puppet for /home because it exceeded max_files [14:45:00] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 31 Aug 2022 21:23:58 GMT +0000. [14:45:00] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Thu 22 Sep 2022 04:19:34 GMT +0000. [14:45:02] RECOVERY - wiki.misslessfighters.club - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.misslessfighters.club' will expire on Tue 23 Aug 2022 13:06:24 GMT +0000. [14:45:07] that's fair [14:45:08] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Thu 01 Sep 2022 00:46:39 GMT +0000. [14:45:08] RECOVERY - theapplesecretswiki.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'theapplesecretswiki.ml' will expire on Thu 04 Aug 2022 20:37:27 GMT +0000. [14:45:12] RECOVERY - wiki.chouverse.cludisciples.cyou - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chouverse.cludisciples.cyou' will expire on Tue 23 Aug 2022 00:10:19 GMT +0000. [14:45:13] RECOVERY - source.erag.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'source.erag.eu.org' will expire on Sun 09 Oct 2022 13:40:55 GMT +0000. [14:45:13] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Sun 02 Oct 2022 10:20:46 GMT +0000. [14:45:19] RECOVERY - wiki.eversd.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Thu 29 Sep 2022 08:32:27 GMT +0000. [14:45:21] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Sat 17 Sep 2022 22:08:07 GMT +0000. [14:45:21] RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Tue 09 Aug 2022 06:17:55 GMT +0000. [14:45:21] RECOVERY - miraheze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.com' will expire on Wed 12 Oct 2022 13:21:03 GMT +0000. [14:45:22] The PR is probably best completely backing out and then git reset --hard [14:45:23] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 17 Sep 2022 21:39:52 GMT +0000. [14:45:27] RECOVERY - mcdev.grantlmul.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'mcdev.grantlmul.xyz' will expire on Mon 01 Aug 2022 16:42:41 GMT +0000. [14:45:28] RECOVERY - wiki.mcpirevival.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcpirevival.tk' will expire on Wed 24 Aug 2022 03:07:55 GMT +0000. [14:45:28] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 24 Aug 2022 02:12:36 GMT +0000. [14:45:30] RECOVERY - miraheze.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ml' will expire on Tue 23 Aug 2022 02:37:13 GMT +0000. [14:45:31] And then we need to work out what's best [14:45:32] max_files can be changed [14:45:32] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Sat 01 Oct 2022 16:25:42 GMT +0000. [14:45:32] RECOVERY - www.cgradegames.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.cgradegames.net' will expire on Sat 20 Aug 2022 20:03:51 GMT +0000. [14:45:32] RECOVERY - biblestrength.net - LetsEncrypt on sslhost is OK: OK - Certificate 'biblestrength.net' will expire on Wed 07 Sep 2022 18:16:09 GMT +0000. [14:45:34] RECOVERY - grayravens.com - LetsEncrypt on sslhost is OK: OK - Certificate 'grayravens.com' will expire on Sun 18 Sep 2022 13:03:26 GMT +0000. [14:45:34] RECOVERY - scum-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'scum-wiki.com' will expire on Tue 09 Aug 2022 06:16:18 GMT +0000. [14:45:42] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Thu 08 Sep 2022 06:29:43 GMT +0000. [14:45:45] RECOVERY - jp.popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'jp.popira.info' will expire on Mon 10 Oct 2022 06:13:03 GMT +0000. [14:45:59] paladox: is upping max_files appropriate [14:46:02] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Tue 23 Aug 2022 08:34:15 GMT +0000. [14:46:07] RECOVERY - wiki.funkey-project.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Tue 23 Aug 2022 08:26:19 GMT +0000. [14:46:12] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Wed 24 Aug 2022 05:10:07 GMT +0000. [14:46:13] The only thing to actually revert would be the entry for /etc/puppetlabs/puppet/ssl-keys [14:46:18] And then adding recurse to make puppet enforce [14:46:22] The other was already there [14:46:24] RECOVERY - repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'repository.archiopedia.org' will expire on Mon 10 Oct 2022 06:46:02 GMT +0000. [14:46:30] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:46:31] RECOVERY - wiki.dereric.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dereric.gq' will expire on Tue 09 Aug 2022 23:18:28 GMT +0000. [14:46:31] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Wed 31 Aug 2022 20:26:03 GMT +0000. [14:46:32] RECOVERY - sdiy.info - LetsEncrypt on sslhost is OK: OK - Certificate 'sdiy.info' will expire on Sat 17 Sep 2022 21:29:52 GMT +0000. [14:46:32] RECOVERY - wiki.ssangyongsports.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssangyongsports.eu.org' will expire on Wed 31 Aug 2022 22:12:11 GMT +0000. [14:46:32] RECOVERY - wiki.minebox.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minebox.fr' will expire on Fri 16 Sep 2022 18:05:33 GMT +0000. [14:46:35] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Sun 02 Oct 2022 10:22:40 GMT +0000. [14:46:35] RECOVERY - proficientprofit.click - LetsEncrypt on sslhost is OK: OK - Certificate 'www.proficientprofit.click' will expire on Sat 08 Oct 2022 06:14:24 GMT +0000. [14:46:35] RECOVERY - www.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikiretet.ga' will expire on Tue 11 Oct 2022 04:42:43 GMT +0000. [14:46:38] RECOVERY - wiki.horizonsend.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.horizonsend.net' will expire on Thu 25 Aug 2022 17:01:32 GMT +0000. [14:46:39] RECOVERY - recaptime.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:46:39] RECOVERY - central.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'central.ballmedia.org' will expire on Mon 10 Oct 2022 06:35:15 GMT +0000. [14:46:41] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Mon 22 Aug 2022 23:09:56 GMT +0000. [14:46:49] RECOVERY - dnd.bellinrattin.it - LetsEncrypt on sslhost is OK: OK - Certificate 'dnd.bellinrattin.it' will expire on Sun 31 Jul 2022 05:03:24 GMT +0000. [14:46:49] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Sun 18 Sep 2022 20:39:54 GMT +0000. [14:46:50] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Mon 10 Oct 2022 04:01:57 GMT +0000. [14:46:50] RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Wed 31 Aug 2022 21:26:51 GMT +0000. [14:46:52] yes if you want. [14:46:55] RECOVERY - mrlove.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mrlove.wiki' will expire on Mon 22 Aug 2022 23:58:24 GMT +0000. [14:47:03] https://puppet.com/docs/puppet/7/types/file.html#file-attribute-max_files [14:47:04] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Mon 22 Aug 2022 23:45:19 GMT +0000. [14:47:05] [url] Resource Type: file | puppet.com [14:47:06] RECOVERY - wikicryptos.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikicryptos.org' will expire on Wed 24 Aug 2022 05:45:16 GMT +0000. [14:47:11] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Thu 29 Sep 2022 02:01:07 GMT +0000. [14:47:12] RECOVERY - patternarchive.online - LetsEncrypt on sslhost is OK: OK - Certificate 'patternarchive.online' will expire on Sat 17 Sep 2022 18:49:33 GMT +0000. [14:47:16] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 1.93, 1.83 [14:47:18] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Tue 23 Aug 2022 09:10:11 GMT +0000. [14:47:23] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:47:23] RECOVERY - test3.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:47:23] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Wed 24 Aug 2022 02:44:01 GMT +0000. [14:47:29] RECOVERY - studyabroad.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'studyabroad.wiki' will expire on Fri 19 Aug 2022 16:50:34 GMT +0000. [14:47:33] RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 0.324 second response time [14:47:39] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [14:47:42] PROBLEM - usaha.click - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'usaha.click' expires in 13 day(s) (Thu 28 Jul 2022 07:26:59 GMT +0000). [14:47:43] RECOVERY - en.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:04:52 GMT +0000. [14:47:48] RECOVERY - www.wikitipp.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.wikitipp.de' will expire on Sun 07 Aug 2022 04:23:09 GMT +0000. [14:48:03] RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Mon 22 Aug 2022 21:21:00 GMT +0000. [14:48:06] RECOVERY - royal-wiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'royal-wiki.org' will expire on Wed 28 Sep 2022 11:34:56 GMT +0000. [14:48:09] !log reinstalled nginx on cp31 [14:48:10] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Sun 02 Oct 2022 10:05:42 GMT +0000. [14:48:13] RECOVERY - scpvncommunitywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'scpvncommunitywiki.xyz' will expire on Tue 23 Aug 2022 02:34:15 GMT +0000. [14:48:13] RECOVERY - tech.teojingyao.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tech.teojingyao.com' will expire on Sat 08 Oct 2022 04:57:57 GMT +0000. [14:48:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:48:21] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Tue 23 Aug 2022 08:44:56 GMT +0000. [14:48:21] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 23 Aug 2022 08:41:48 GMT +0000. [14:48:32] RECOVERY - en.omniversalis.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.omniversalis.org' will expire on Sat 13 Aug 2022 21:32:36 GMT +0000. [14:48:33] RECOVERY - wiki.ssij.se - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Mon 22 Aug 2022 22:25:45 GMT +0000. [14:48:33] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Tue 23 Aug 2022 01:21:14 GMT +0000. [14:48:33] RECOVERY - itora.bunnies1230.com - LetsEncrypt on sslhost is OK: OK - Certificate 'itora.bunnies1230.com' will expire on Tue 20 Sep 2022 01:50:31 GMT +0000. [14:48:33] RECOVERY - mystiverse.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'mystiverse.wiki' will expire on Sat 01 Oct 2022 04:26:41 GMT +0000. [14:48:40] RECOVERY - pt.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.polandballwiki.com' will expire on Thu 29 Sep 2022 08:59:53 GMT +0000. [14:48:40] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Tue 23 Aug 2022 01:40:20 GMT +0000. [14:48:44] MacFan4000: do that then [14:48:46] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 24 Aug 2022 04:11:00 GMT +0000. [14:48:49] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Wed 24 Aug 2022 03:34:26 GMT +0000. [14:48:49] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Tue 23 Aug 2022 00:16:11 GMT +0000. [14:48:49] RECOVERY - commons.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'commons.polandballwiki.com' will expire on Mon 10 Oct 2022 05:30:54 GMT +0000. [14:48:49] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Wed 24 Aug 2022 02:17:23 GMT +0000. [14:48:51] RECOVERY - wiki.apico.buzz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apico.buzz' will expire on Thu 01 Sep 2022 00:43:13 GMT +0000. [14:48:53] and let's make sure etc and home match [14:48:54] RECOVERY - wiki.veloren.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.veloren.net' will expire on Tue 02 Aug 2022 16:20:30 GMT +0000. [14:48:55] RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Sat 17 Sep 2022 20:00:57 GMT +0000. [14:48:56] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Sat 24 Sep 2022 13:17:51 GMT +0000. [14:49:09] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 17 Sep 2022 18:56:39 GMT +0000. [14:49:24] RECOVERY - croc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:49:25] RECOVERY - drowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'drowiki.com' will expire on Tue 04 Oct 2022 18:56:56 GMT +0000. [14:49:27] RECOVERY - aresrocket.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'aresrocket.ml' will expire on Mon 01 Aug 2022 05:59:42 GMT +0000. [14:49:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:49:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:49:41] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.polandballwiki.com' will expire on Thu 29 Sep 2022 08:52:53 GMT +0000. [14:49:42] RECOVERY - www.connorjwat.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.connorjwat.net' will expire on Mon 22 Aug 2022 22:20:46 GMT +0000. [14:49:47] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Mon 19 Sep 2022 03:34:09 GMT +0000. [14:49:47] RECOVERY - lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lgbtqia.wiki' will expire on Fri 07 Oct 2022 11:35:52 GMT +0000. [14:49:47] RECOVERY - deepdark.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deepdark.wiki' will expire on Mon 03 Oct 2022 20:17:33 GMT +0000. [14:49:47] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Mon 22 Aug 2022 15:22:46 GMT +0000. [14:49:48] RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Tue 23 Aug 2022 13:11:15 GMT +0000. [14:49:50] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Sat 17 Sep 2022 18:42:17 GMT +0000. [14:49:50] RECOVERY - wiki.erisly.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.erisly.com' will expire on Wed 31 Aug 2022 21:21:26 GMT +0000. [14:49:50] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Mon 10 Oct 2022 05:23:51 GMT +0000. [14:49:52] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 24 Aug 2022 04:30:45 GMT +0000. [14:49:54] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Tue 23 Aug 2022 01:42:45 GMT +0000. [14:49:54] RECOVERY - slupe.fp64.org - LetsEncrypt on sslhost is OK: OK - Certificate 'slupe.fp64.org' will expire on Tue 04 Oct 2022 18:47:51 GMT +0000. [14:49:55] RECOVERY - tensegritywiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'tensegritywiki.com' will expire on Tue 23 Aug 2022 08:10:17 GMT +0000. [14:49:55] RECOVERY - wiki.zergmafia.top - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zergmafia.top' will expire on Tue 06 Sep 2022 20:57:49 GMT +0000. [14:50:07] RECOVERY - nenawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nenawiki.org' will expire on Tue 23 Aug 2022 01:35:44 GMT +0000. [14:50:07] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [14:50:07] RECOVERY - wildbowiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wildbowiki.net' will expire on Tue 06 Sep 2022 20:59:02 GMT +0000. [14:50:16] RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Tue 23 Aug 2022 02:51:23 GMT +0000. [14:50:19] RECOVERY - wiki.rebirthofthenight.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rebirthofthenight.com' will expire on Wed 24 Aug 2022 02:49:21 GMT +0000. [14:50:24] RECOVERY - atnarsia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'atnarsia.com' will expire on Sun 25 Sep 2022 20:45:25 GMT +0000. [14:50:24] RECOVERY - wiki.evscope.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.evscope.org' will expire on Fri 07 Oct 2022 11:21:52 GMT +0000. [14:50:32] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Fri 12 Aug 2022 07:48:29 GMT +0000. [14:51:02] RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Sun 02 Oct 2022 10:59:59 GMT +0000. [14:51:02] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Tue 23 Aug 2022 09:31:28 GMT +0000. [14:51:04] RECOVERY - wiki.wisrail.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wisrail.de' will expire on Tue 06 Sep 2022 21:01:07 GMT +0000. [14:51:04] RECOVERY - es.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'es.gyaanipedia.com' will expire on Thu 08 Sep 2022 07:18:15 GMT +0000. [14:51:05] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Tue 06 Sep 2022 20:51:36 GMT +0000. [14:51:05] RECOVERY - pokemonwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'pokemonwiki.info' will expire on Sat 06 Aug 2022 07:26:57 GMT +0000. [14:51:17] RECOVERY - unmade.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:51:19] RECOVERY - en.wikiretet.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiretet.ga' will expire on Sun 02 Oct 2022 05:37:01 GMT +0000. [14:51:20] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Sat 17 Sep 2022 19:04:53 GMT +0000. [14:51:21] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Sep 2022 00:46:11 GMT +0000. [14:51:30] RECOVERY - wiki.virtualjet.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.virtualjet.net' will expire on Sun 25 Sep 2022 21:20:59 GMT +0000. [14:51:30] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Tue 23 Aug 2022 08:40:10 GMT +0000. [14:51:30] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Sat 24 Sep 2022 13:42:28 GMT +0000. [14:51:30] RECOVERY - mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mwcosmos.com' will expire on Mon 22 Aug 2022 15:03:04 GMT +0000. [14:51:36] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Mon 22 Aug 2022 22:32:53 GMT +0000. [14:51:40] RECOVERY - miraheze.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.tk' will expire on Tue 23 Aug 2022 07:48:08 GMT +0000. [14:51:49] RECOVERY - wiki.timewisely.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.timewisely.net' will expire on Mon 22 Aug 2022 22:01:18 GMT +0000. [14:51:57] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Tue 23 Aug 2022 02:08:17 GMT +0000. [14:51:57] RECOVERY - sharqedia.ml - LetsEncrypt on sslhost is OK: OK - Certificate 'sharqedia.ml' will expire on Mon 03 Oct 2022 13:40:45 GMT +0000. [14:51:57] RECOVERY - wikipariksha.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipariksha.com' will expire on Wed 24 Aug 2022 05:18:49 GMT +0000. [14:51:59] RECOVERY - staff.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'staff.polandballwiki.com' will expire on Sun 02 Oct 2022 10:43:17 GMT +0000. [14:52:00] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Tue 23 Aug 2022 03:06:24 GMT +0000. [14:52:02] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Tue 23 Aug 2022 01:13:17 GMT +0000. [14:52:02] RECOVERY - wiki-tintin.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki-tintin.cf' will expire on Mon 10 Oct 2022 03:55:03 GMT +0000. [14:52:10] RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Sat 03 Sep 2022 15:09:33 GMT +0000. [14:52:11] RECOVERY - ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ballmedia.org' will expire on Mon 10 Oct 2022 05:56:45 GMT +0000. [14:52:19] RECOVERY - phgalaxy.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'phgalaxy.eu.org' will expire on Sun 02 Oct 2022 05:33:18 GMT +0000. [14:52:19] RECOVERY - sbs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sbs.wiki' will expire on Thu 01 Sep 2022 00:42:07 GMT +0000. [14:52:29] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Thu 15 Sep 2022 05:19:23 GMT +0000. [14:52:30] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Sat 17 Sep 2022 20:30:32 GMT +0000. [14:52:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/59ff0c0774a8...c8670a64bcc7 [14:52:31] [url] Comparing 59ff0c0774a8...c8670a64bcc7 · miraheze/puppet · GitHub | github.com [14:52:31] [02miraheze/puppet] 07paladox 03c8670a6 - nginx: increase server_names_hash_max_size and server_names_hash_bucket_size [14:52:42] RECOVERY - nottingham.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nottingham.wiki' will expire on Mon 26 Sep 2022 06:16:48 GMT +0000. [14:52:49] RECOVERY - privacywiki.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'privacywiki.xyz' will expire on Mon 26 Sep 2022 13:36:29 GMT +0000. [14:52:49] RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Sat 17 Sep 2022 21:26:27 GMT +0000. [14:52:49] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Mon 22 Aug 2022 20:43:25 GMT +0000. [14:52:49] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Sat 01 Oct 2022 16:33:15 GMT +0000. [14:52:49] RECOVERY - famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Mon 10 Oct 2022 05:45:28 GMT +0000. [14:52:53] RECOVERY - wiki.exputra.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.exputra.com' will expire on Fri 16 Sep 2022 18:09:57 GMT +0000. [14:52:53] RECOVERY - miraheze.ga - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.ga' will expire on Wed 24 Aug 2022 04:25:02 GMT +0000. [14:52:59] RECOVERY - www.pornwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pornwiki.org' will expire on Sat 17 Sep 2022 20:10:50 GMT +0000. [14:52:59] RECOVERY - wiki.projectdiablo2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Tue 11 Oct 2022 03:46:46 GMT +0000. [14:53:00] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Mon 10 Oct 2022 06:24:27 GMT +0000. [14:53:00] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Mon 22 Aug 2022 22:32:33 GMT +0000. [14:53:17] RECOVERY - pokemon2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [14:53:18] RECOVERY - www.ballmedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ballmedia.org' will expire on Mon 10 Oct 2022 05:56:45 GMT +0000. [14:53:18] RECOVERY - christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 17 Sep 2022 18:34:21 GMT +0000. [14:53:26] RECOVERY - wiki.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yahyabd.xyz' will expire on Sun 14 Aug 2022 20:06:30 GMT +0000. [14:53:27] RECOVERY - discordian.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'discordian.wiki' will expire on Fri 16 Sep 2022 18:03:24 GMT +0000. [14:53:30] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sat 17 Sep 2022 19:46:36 GMT +0000. [14:53:38] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Wed 28 Sep 2022 05:13:53 GMT +0000. [14:53:56] RECOVERY - wiki.lu9.io - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lu9.io' will expire on Tue 06 Sep 2022 20:59:54 GMT +0000. [14:53:56] RECOVERY - phytinfo.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'phytinfo.cf' will expire on Sun 14 Aug 2022 15:45:20 GMT +0000. [14:54:00] RECOVERY - electrowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electrowiki.org' will expire on Sat 17 Sep 2022 19:15:15 GMT +0000. [14:54:01] RECOVERY - evil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'evil.wiki' will expire on Wed 10 Aug 2022 22:13:25 GMT +0000. [14:54:01] RECOVERY - wikifencing.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'wikifencing.ru' will expire on Tue 06 Sep 2022 20:55:14 GMT +0000. [14:54:01] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Sat 01 Oct 2022 16:22:56 GMT +0000. [14:54:05] RECOVERY - landofliberos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'landofliberos.com' will expire on Tue 30 Aug 2022 16:38:50 GMT +0000. [14:54:24] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 12 day(s) (Wed 27 Jul 2022 12:12:22 GMT +0000). [14:54:36] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Thu 29 Dec 2022 23:59:59 GMT +0000. [14:54:48] I'll be doing the IR since I caused it but what I wonder is what exactly was the issue with the keys? [14:54:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c8670a64bcc7...19291b824c99 [14:54:58] [url] Comparing c8670a64bcc7...19291b824c99 · miraheze/puppet · GitHub | github.com [14:54:59] [02miraheze/puppet] 07paladox 0319291b8 - Fix [14:55:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.87, 1.67, 1.78 [14:55:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.20, 1.76, 1.87 [14:55:34] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:02] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.28, 8.35 [14:56:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 5.473 second response time [14:57:25] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:57:37] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.637 second response time [14:57:53] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Wed 12 Oct 2022 13:33:40 GMT +0000. [14:58:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.87, 10.03, 8.49 [15:01:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.32, 1.59 [15:01:17] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [15:01:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.78, 9.79, 8.32 [15:01:58] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.45, 1.67 [15:03:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.39, 9.77, 7.73 [15:03:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.21, 10.72, 8.81 [15:03:54] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:05:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.12, 8.99, 7.68 [15:05:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.79, 10.13, 8.84 [15:06:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.75, 11.34, 9.72 [15:08:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:08:05] [02puppet] 07MacFan4000 opened pull request 03#2697: try recursion again with fixes - 13https://github.com/miraheze/puppet/pull/2697 [15:08:06] [url] Page not found · GitHub · GitHub | github.com [15:08:25] RhinosF1: paradox: ^^ [15:08:32] paladox* [15:08:53] ok [15:09:00] paladox: +1 [15:09:09] Reception123: they didn't get pushed [15:09:40] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.58, 3.37, 2.61 [15:09:42] [02puppet] 07paladox closed pull request 03#2697: try recursion again with fixes - 13https://github.com/miraheze/puppet/pull/2697 [15:09:43] [url] Page not found · GitHub · GitHub | github.com [15:09:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/19291b824c99...5cb7d9d38a05 [15:09:44] [url] Comparing 19291b824c99...5cb7d9d38a05 · miraheze/puppet · GitHub | github.com [15:09:45] [02miraheze/puppet] 07MacFan4000 035cb7d9d - try recursion again with fixes (#2697) [15:10:54] paladox: please make sure it gets deployed and then git is happy [15:11:01] ok [15:11:35] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.93, 3.56, 2.77 [15:13:22] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.61, 1.61 [15:15:17] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.83, 1.69 [15:17:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.97, 3.25, 2.91 [15:18:28] PROBLEM - gluster101 Puppet on gluster101 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run],File[/etc/ssl/localcerts/wildcard.miraheze.org-2020-2.crt],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [15:18:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.25, 10.89, 10.14 [15:18:43] hmm [15:19:05] it keeps failing [15:19:20] oh maybe it hit the time i restarted puppetserver [15:20:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 10.64, 10.13 [15:20:49] seems to work [15:21:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.67, 9.39, 8.10 [15:22:16] paladox: is git status clear [15:22:18] RECOVERY - gluster101 Puppet on gluster101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:22:25] on where [15:22:27] After fetching from the /etc version [15:22:35] paladox: the copy in /home [15:22:57] do you mean /home/ssl*/ssl*? [15:23:31] RhinosF1: [15:23:38] paladox: the copy of the certificates in /home [15:23:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.95, 9.30, 8.23 [15:24:02] no it's not clear [15:24:05] Under /home/ssl-admins [15:24:23] i guess that's because you change the perms (chmod) [15:24:30] That's normally used to push to /etc/puppetlabs/puppet/ssl-keys [15:25:04] *changed [15:25:11] paladox: you might need to commit the mode change ye [15:25:21] Then they should stay right [15:25:32] done [15:25:38] https://www.irccloud.com/pastebin/Gr0O67Vc/ [15:26:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 11.42, 10.65 [15:26:11] https://www.irccloud.com/pastebin/zqqkOxUQ/ [15:26:25] yeah git picked up the perms changes [15:26:33] MacFan4000: it'll need to be commuted [15:26:43] paladox: that might need fixing [15:26:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 2.00, 1.94 [15:26:53] What is the post-receive hook [15:27:39] https://www.irccloud.com/pastebin/w2OD3OvQ/ [15:27:40] It's not defined anywhere [15:27:56] https://www.irccloud.com/pastebin/S6vqCu1G/ [15:27:57] paladox: cat it [15:28:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 10.65, 10.47 [15:28:19] Why are we doing that and why out of puppet [15:28:31] That's git resetting on every run [15:28:44] !log root@puppet111:/home/ssl-admins/ssl-keys# chmod ug+x /etc/p*/p*/ssl-keys/.git/hooks/post-receive [15:28:48] maybe that fixes it [15:28:55] paladox: no [15:29:01] Puppet will overwrite it [15:29:04] https://stackoverflow.com/questions/8598639/why-is-my-git-pre-commit-hook-not-executable-by-default [15:29:04] [url] githooks - Why is my Git pre-commit hook not executable by default? - Stack Overflow | stackoverflow.com [15:29:09] oh [15:29:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:17] ok lets stop recursing pls [15:29:18] paladox: it's not executable because we just set it to be not [15:29:22] just seems to be more a pian [15:29:25] *pain [15:29:30] shouldn't be doing that for git repos [15:29:35] Why does that hook actually exist [15:29:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03revert-2697-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/029f7d0f87e5 [15:29:48] [02miraheze/puppet] 07paladox 03029f7d0 - Revert "try recursion again with fixes (#2697)" [15:29:49] [02puppet] 07paladox created branch 03revert-2697-patch-11 - 13https://github.com/miraheze/puppet [15:29:50] Because it's not in puppet [15:29:50] [url] Page not found · GitHub · GitHub | github.com [15:29:51] [02puppet] 07paladox opened pull request 03#2698: Revert "try recursion again with fixes" - 13https://github.com/miraheze/puppet/pull/2698 [15:29:51] [url] Page not found · GitHub · GitHub | github.com [15:29:52] RhinosF1: MacFan4000 ^ [15:30:11] paladox: I don't know whether that's an issue though [15:30:16] it is [15:30:20] as you can see when we push [15:30:23] If it's not in puppet, when was it added [15:30:39] it's a git repo created locally [15:30:44] ofc it's not going to be in puppet [15:30:45] And why [15:30:47] manually created [15:30:50] paladox: or anywhere [15:30:51] to hold the ssl keys... [15:30:58] You can track the hooks [15:31:00] i presume log [15:31:02] it wasn't defined in the puppet repo until now [15:31:07] i dunno, you'll have to ask SPF or John [15:31:15] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.59, 10.08, 9.39 [15:31:23] i mean that would explain why it's been failed [15:31:26] Failing [15:31:40] We were silently resetting every single time anyway on receive [15:32:07] MacFan4000: do we need to enforce the mode [15:32:11] In puppet [15:32:20] Or can we just enforce puppet:ssl-admins [15:32:42] paladox: the issue is we made a change then realised we were changing stuff we don't understand [15:32:48] i just wanted to enforce perms [15:32:55] ok [15:32:58] ownership and perms [15:33:09] should only use recursive if in a non git repo tbh [15:33:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 10.85, 9.76 [15:33:14] MacFan4000: why perks [15:33:17] Perms* [15:33:18] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.792 second response time [15:33:21] What does perms help [15:33:26] Isn't it the group that's the issue [15:33:37] anyways i'm going mobile [15:33:46] paladox: let's try and actually understand the problem and solution before just making a mess [15:33:48] got a flight tomorrow [15:33:51] Ok [15:33:51] RhinosF1: even if the group is set to ssl-admins it would still need to be group writable [15:33:54] Enjoy [15:34:01] MacFan4000: but has that ever been an issue [15:34:06] yes [15:34:14] it wasn't writable [15:34:20] before the changes [15:34:29] only by root [15:34:38] ah [15:34:44] Default is 755 [15:34:55] We need to get our head around what's actually needed [15:35:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 10.91, 9.92 [15:35:18] 770 might be better [15:35:30] But I think only /home needs the mask [15:35:41] Because push to etc should set the mask anyway [15:35:47] and should always reflect [15:35:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.90, 11.78, 11.04 [15:36:11] Tbh I'm just hating ssl at the moment and wishing we'd rebuilt the system [15:36:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.97, 1.93 [15:37:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.97, 10.08, 9.74 [15:37:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 11.41, 10.98 [15:42:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.89, 10.01, 8.89 [15:44:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.65, 9.92, 8.99 [15:47:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.95, 1.49 [15:49:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.59, 1.41 [15:53:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:55:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.01, 11.22, 10.88 [15:57:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.90, 11.10, 10.88 [16:01:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.17, 10.20 [16:05:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 12.00, 11.35 [16:09:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 11.20, 11.22 [16:10:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 11.52, 11.07 [16:20:29] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.55, 1.34 [16:21:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.14, 10.96 [16:22:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.51, 1.35 [16:22:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.87, 9.34, 10.12 [16:23:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:23:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 10.70, 10.81 [16:27:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 11.56, 11.13 [16:29:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.90, 11.16, 11.05 [16:31:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.05, 10.29, 10.31 [16:34:07] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.23, 1.81, 1.36 [16:35:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 11.09, 11.01 [16:36:05] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.17, 1.60, 1.34 [16:36:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.25, 1.60, 1.94 [16:37:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.86, 11.57, 11.21 [16:42:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.51, 2.12, 1.68 [16:43:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.10, 10.65, 9.55 [16:44:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.20, 1.84, 1.63 [16:44:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.61, 10.94, 10.62 [16:45:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.78, 11.37 [16:45:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.93, 10.10, 9.47 [16:45:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.52, 10.35, 9.57 [16:46:03] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.55, 1.54 [16:46:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.88, 1.90 [16:47:18] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.00, 1.65, 1.25 [16:47:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.40, 11.27 [16:47:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.03, 9.78, 9.45 [16:49:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.50, 1.53, 1.26 [16:49:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 12.02, 11.51 [16:50:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.65, 11.35, 10.96 [16:51:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.73, 11.86, 11.52 [16:51:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 2.10, 1.77 [16:54:41] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:38] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.88, 1.77 [16:56:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.99, 1.96, 1.99 [16:57:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.89, 12.45, 11.79 [16:57:43] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.928 second response time [16:57:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.53, 1.65 [16:58:48] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.921 second response time [16:58:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.04, 1.92, 1.97 [16:58:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.407 second response time [17:02:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.72, 1.84, 1.93 [17:03:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.66, 11.71 [17:05:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.24, 12.36, 11.94 [17:07:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 11.12, 11.55 [17:12:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 9.31, 10.14 [17:16:48] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.90, 1.85 [17:17:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.50, 8.90, 10.16 [17:23:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.55, 11.41, 10.78 [17:23:46] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/7ea75ff9c40b [17:23:48] [02miraheze/ssl] 07MacFan4000 037ea75ff - add www.replication-ops.com [17:23:49] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:23:49] [url] Page not found · GitHub · GitHub | github.com [17:23:55] [02ssl] 07MacFan4000 opened pull request 03#567: add www.replication-ops.com - 13https://github.com/miraheze/ssl/pull/567 [17:23:55] ... [17:25:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.18, 10.99, 10.71 [17:25:51] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7ea75ff9c40b...28c5352ffd3f [17:25:51] [url] Comparing 7ea75ff9c40b...28c5352ffd3f · miraheze/ssl · GitHub | github.com [17:25:52] [02miraheze/ssl] 07MacFan4000 0328c5352 - Update certs.yaml [17:25:54] [02ssl] 07MacFan4000 synchronize pull request 03#567: add www.replication-ops.com - 13https://github.com/miraheze/ssl/pull/567 [17:25:54] [url] Page not found · GitHub · GitHub | github.com [17:26:13] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b5207dd99dce...884a3e667dbd [17:26:13] [url] Comparing b5207dd99dce...884a3e667dbd · miraheze/ssl · GitHub | github.com [17:26:14] [02miraheze/ssl] 07MacFan4000 03884a3e6 - add www.replication-ops.com (#567) [17:26:16] [02ssl] 07MacFan4000 closed pull request 03#567: add www.replication-ops.com - 13https://github.com/miraheze/ssl/pull/567 [17:26:16] [url] Page not found · GitHub · GitHub | github.com [17:26:17] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:26:19] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:26:19] ... [17:29:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.37, 11.74, 11.08 [17:31:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.06, 11.31, 11.00 [17:35:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.61, 11.88, 11.24 [17:35:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 10.31, 10.03 [17:37:35] RhinosF1: paladox: Reception123: after some searching I found https://stackoverflow.com/questions/25050787/why-does-my-remote-git-repository-have-uncommitted-changes-after-pushing-to-it which seems to discribe exactly the issue we're having [17:37:35] [url] git push - Why does my remote Git repository have uncommitted changes after pushing to it? - Stack Overflow | stackoverflow.com [17:38:10] (re ssl) [17:39:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.85, 1.57 [17:41:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.66, 1.54 [17:41:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 11.66, 11.35 [17:45:06] MacFan4000: ah that's interesting [17:45:08] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.96, 10.11, 10.14 [17:51:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.76, 11.43 [17:51:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 10.21, 10.19 [17:53:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 11.76, 11.48 [17:53:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.31, 10.04, 10.14 [17:55:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 12.00, 11.59 [17:55:47] Issue is that with a bare repo you can’t access the contents via a normal means so a bare repo wouldn’t work for us [17:56:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.64, 1.52 [17:57:14] I honestly wonder if it would make more sense to just be writing keys directly to /etc/puppetlabs/puppet/ssl-keys instead of going through /home [17:57:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.74, 11.62, 11.51 [17:57:54] As long as the permissions allow ssl-admins to write it shouldn’t be an issue [17:57:54] The bare repo works for us [17:57:59] It has for countless years [17:58:36] In fact, it’s actually better as it then prevent accidental pushes in the wrong place like we experienced with the private repo [17:58:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.25, 1.39 [17:59:08] [discord] I’m not sure why its suddenly causing issues though [17:59:37] It’s probably a result of the permissions change PR as it seems problems occurred after it [17:59:46] The repo in /etc has become non-bare for some reason [18:00:02] It should be bare so someone has changed it [18:00:28] Reception123: paladox ^ [18:01:14] https://github.com/miraheze/puppet/blob/master/modules/ssl/manifests/init.pp#L38 is the current definition in puppet [18:01:14] [url] puppet/init.pp at master · miraheze/puppet · GitHub | github.com [18:01:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 10.44, 10.28 [18:02:15] My advice when I fixed it previously was always ‘don’t fuck with the permissions otherwise you’ll break it’ [18:02:47] And that’s been my stance since we moved to handle SSL in this way. It’s fragile and finicky [18:03:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.13, 10.76, 9.70 [18:04:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.32, 11.91, 11.61 [18:05:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 10.01, 10.18 [18:05:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.81, 9.32, 9.31 [18:09:29] Yeah, though it's also my fault for having done the upgrades at the same time [18:09:37] I just saw that they weren't done since the end of June and thought they should be [18:11:31] Always do upgrades in batches by package and understand the implications of each package being upgraded [18:13:26] I'll remember that in the future. I didn't do critical servers like db* and gluster* to not repeat the same mistake as last time but the unfortunate problem with the keys permissions lead to the cp issue [18:17:47] ah, I found the problem [18:17:51] the perms change 100% is the cause [18:18:05] as the git hooks have lost their +x perms [18:18:16] meaning new pushes were never made available to puppet [18:19:04] so the perms change needs reverting or fixing to include +x, and then things should work again [18:20:36] I think it's probably best to cancel the puppet changes for permissions since the issues encountered were a one time thing and before that things worked fine for puppet-users [18:20:55] unless there's really a sure way not to cause issues again [18:22:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.94, 10.11, 9.90 [18:24:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.95, 9.55, 9.71 [18:24:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.55, 1.71, 1.39 [18:26:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.51, 1.27, 1.26 [18:27:04] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 98 failures. Last run 2 minutes ago with 98 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/atnarsia.com.key],File[/etc/ssl/private/gametheorywiki.org.key],File[/etc/ssl/private/wiki.virtualjet.net.key],File[/etc/ssl/private/www.agentisai.com.key] [18:29:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:29:39] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 3 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:29:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 10.72, 10.15 [18:29:46] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:30:42] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:31:14] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[glusterfs.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:31:44] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:31:48] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[glusterfs.key],File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:33:10] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5cb7d9d38a05...ce0ad67f3a41 [18:33:10] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:33:11] [url] Comparing 5cb7d9d38a05...ce0ad67f3a41 · miraheze/puppet · GitHub | github.com [18:33:12] [02miraheze/puppet] 07JohnFLewis 03ce0ad67 - remove incorrect permissions on ssl-keys [18:34:18] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:34:53] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 3 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:35:07] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:00] Reception123: my main reason for wanting the changes was so that ssl-admins could remove keys when needed. (No longer pointed etc) the only reason it still works for people is because when adding keys via the script it is not an issue because that script does things as root [18:36:15] Which otherwise we are not [18:36:20] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:36:46] Oh, so you mean with the script ssl-admins can't remove anymore? [18:37:12] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 2 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:37:15] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.278 second response time [18:37:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 9.93, 10.04 [18:37:47] The script adds keys as root, to remove a key you have to do it manually but you need the correct permissions [18:37:48] the other solution I see is to allow “sudo git ” for ssl-admins [18:38:49] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 3 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:39:32] Or it wouldn’t be very difficult to add an option to the script that can just remove a specified key [18:40:37] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.62, 3.42, 2.95 [18:40:40] Reception123: ^^ [18:40:52] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:40:52] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/miraheze.wiki.key] [18:41:17] Another key issue? [18:41:24] But no one even touched anything afaik [18:41:29] JohnLewis: I assume you’re aware of the mass puppet fails? [18:41:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 10.55, 10.28 [18:41:38] Yes, I am trying to fix it [18:41:42] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:41:45] Reception123: John is doing stuff [18:41:53] and this time, no one should be touching it because it will only break [18:42:02] Ah ok [18:42:33] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.10, 3.49, 3.02 [18:42:35] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:42:52] I mentioned two solutions above that don’t involve touching file permissions [18:43:25] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:43:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.04, 10.14, 10.14 [18:43:38] PROBLEM - ldap111 Puppet on ldap111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [18:44:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.82, 11.50, 11.95 [18:44:29] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.80, 3.44, 3.04 [18:44:42] [02puppet] 07paladox closed pull request 03#2698: Revert "try recursion again with fixes" - 13https://github.com/miraheze/puppet/pull/2698 [18:44:43] [url] Page not found · GitHub · GitHub | github.com [18:44:44] [02miraheze/puppet] 07paladox deleted branch 03revert-2697-patch-11 [18:44:46] [02puppet] 07paladox deleted branch 03revert-2697-patch-11 - 13https://github.com/miraheze/puppet [18:44:46] [url] Page not found · GitHub · GitHub | github.com [18:45:07] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 3 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:45:17] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 432 failures. Last run 3 minutes ago with 432 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/betaheze.org.key],File[/etc/ssl/private/antiguabarbudacalypso.com.key],File[/etc/ssl/private/permanentfuturelab.wiki.key] [18:47:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.72, 2.11, 1.46 [18:48:21] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.12, 3.36, 3.10 [18:49:06] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.63, 1.36 [18:49:21] The real solution is redesigning SSL [18:49:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 10.73, 10.47 [18:49:32] Which was supposed to be a goal last year that was never worked on [18:50:54] Reception123: actually looks like non-le keys aren’t automatically added either [18:51:41] so far CosmicAlpha was going to implement automatically pushing to GitHub at some point but obviously that doesn't do anything with the keys part [18:52:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.06, 11.44, 11.56 [18:52:19] MacFan4000: yeah that was intended as they're generated when the CSR is and sometimes we don't hear back so there's no point adding the keys. Since they're so rare (I think I've seen one or two in like 3 months) they can be added manually [18:53:03] Reception123: I will when I have time, July is just busy for me, it is also a bit lower priority also, behind some user tasks, etc.. [18:53:19] Technically a part of that task did eventually get done because one of the early stages is to automate adding keys and now private keys are automatic for the most part [18:53:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.03, 9.54, 10.02 [18:53:42] yeah that's fine I just mentioned that but it wouldn't be what John has in mind since it doesn't affect the keys part [18:54:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.18, 11.45 [18:54:08] but yeah, the task is partially done but the main part would've been to make SSL somewhat similar to ImportDump and also redesign it [18:54:20] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:54:44] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:12] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.58, 3.28, 2.28 [18:55:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:32] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.26, 12.03, 11.74 [18:57:02] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:08] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:37] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:46] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:58:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.58, 11.75, 11.66 [18:58:42] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:58:54] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:59:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.81, 10.79, 10.27 [18:59:37] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:00:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 12.15, 11.83 [19:00:08] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:00:16] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.02, 9.88, 9.34 [19:01:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.65, 10.51, 10.26 [19:02:11] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.66, 10.58, 9.66 [19:02:14] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [19:02:54] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:03:09] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:06] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.54, 9.97, 9.57 [19:04:19] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:05:10] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.17, 3.79, 3.15 [19:05:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.31, 1.75, 1.47 [19:05:32] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.30, 11.14, 10.55 [19:05:40] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:55] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [19:07:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.39, 1.54, 1.42 [19:07:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 10.55, 10.41 [19:08:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:08:39] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:08:49] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [19:09:36] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.532 second response time [19:09:41] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:10:02] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.787 second response time [19:10:18] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:10:43] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.556 second response time [19:10:48] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.184 second response time [19:11:01] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:11:09] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:11:09] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.40, 3.04, 3.05 [19:11:36] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:11:37] RECOVERY - ldap111 Puppet on ldap111 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:12:32] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.825 second response time [19:13:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 10.90, 10.52 [19:15:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.48, 10.71, 10.50 [19:18:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 11.27, 11.82 [19:18:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 9.77, 9.11 [19:18:28] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.61, 1.19 [19:19:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.94, 1.89, 1.68 [19:20:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 1.64, 1.25 [19:21:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.62, 1.61 [19:22:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.49, 11.66, 11.85 [19:22:02] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.63, 3.54, 3.05 [19:22:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 9.14, 9.03 [19:22:24] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.28, 1.81, 1.35 [19:23:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 9.40, 10.11 [19:24:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.46, 11.05, 11.62 [19:24:02] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.91, 3.62, 3.14 [19:24:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.79, 1.40 [19:26:02] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.45, 3.22, 3.05 [19:26:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.67, 1.40 [19:28:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.89, 1.73 [19:29:30] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.28, 10.42, 10.25 [19:29:32] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/workflow-templates/actions/cache-3.0.5 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/f5fe343414d7 [19:29:34] [02miraheze/.github] 07dependabot[bot] 03f5fe343 - Bump actions/cache from 3.0.4 to 3.0.5 in /workflow-templates [19:29:35] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/workflow-templates/actions/cache-3.0.5 - 13https://github.com/miraheze/.github [19:29:37] [02.github] 07dependabot[bot] opened pull request 03#13: Bump actions/cache from 3.0.4 to 3.0.5 in /workflow-templates - 13https://github.com/miraheze/.github/pull/13 [19:29:37] [url] Page not found · GitHub · GitHub | github.com [19:29:38] [02.github] 07dependabot[bot] labeled pull request 03#13: Bump actions/cache from 3.0.4 to 3.0.5 in /workflow-templates - 13https://github.com/miraheze/.github/pull/13 [19:29:38] [url] Page not found · GitHub · GitHub | github.com [19:30:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.42, 1.38, 1.57 [19:31:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.83, 10.53, 10.31 [19:34:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.60, 10.82, 11.10 [19:35:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.84, 10.12 [19:36:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.99, 11.16 [19:40:51] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.70, 10.85, 9.59 [19:42:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.56, 9.61, 9.29 [19:45:04] PROBLEM - mw112 NTP time on mw112 is WARNING: NTP WARNING: Offset -0.1174950898 secs [19:47:03] [02puppet] 07MacFan4000 opened pull request 03#2699: ssl: extend revoke option to also remove key from git - 13https://github.com/miraheze/puppet/pull/2699 [19:47:03] [url] Page not found · GitHub · GitHub | github.com [19:47:40] I think ^ will be a good solution [19:48:20] this means that keys can be removed via the script via the standard process of revoking a cert [19:51:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.52, 2.48, 1.80 [19:53:12] doesn't cover non-le, but since they are so rare, I would be fine just poking infra or reception to do it in those instances [19:53:39] in fact besides the maine wildcard for mh.org there is only 1 non-le certificate [19:53:46] main* [19:55:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.96, 1.71, 1.65 [19:56:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.02, 2.03, 1.57 [19:57:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.40, 1.27, 1.49 [19:57:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [19:58:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.32, 11.24, 10.73 [19:58:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.68, 1.50 [19:59:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.08, 9.77 [20:00:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 11.09, 10.74 [20:03:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.90, 9.52, 9.60 [20:07:14] RECOVERY - mw112 NTP time on mw112 is OK: NTP OK: Offset -0.09993800521 secs [20:20:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.68, 9.84, 10.17 [20:20:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.82, 1.52, 1.30 [20:22:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.37, 1.27 [20:30:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.62, 10.28 [20:32:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:32:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.27, 9.83, 10.04 [20:36:21] PROBLEM - graylog121 NTP time on graylog121 is WARNING: NTP WARNING: Offset 0.104790777 secs [20:52:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 9.52, 9.09 [20:52:24] RECOVERY - graylog121 NTP time on graylog121 is OK: NTP OK: Offset 0.09435302019 secs [20:54:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 9.94, 9.30 [20:58:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 9.47, 9.29 [21:02:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 10.33, 9.71 [21:04:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.01, 9.87, 9.59 [21:04:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.22, 2.61, 1.85 [21:05:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.42, 1.66, 1.35 [21:07:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.36, 1.56, 1.35 [21:08:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.62, 1.98, 1.76 [21:10:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.83, 3.54, 2.42 [21:11:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 2.39, 1.76 [21:11:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.45, 9.55, 8.70 [21:12:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.52, 9.29, 8.62 [21:13:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.01, 1.85, 1.64 [21:13:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 9.13, 8.66 [21:14:43] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.92, 9.33, 8.71 [21:15:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.60, 1.57 [21:16:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.26, 10.21, 9.51 [21:16:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.51, 1.75, 1.99 [21:18:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.51, 9.30, 9.27 [21:20:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.16, 1.69 [21:21:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.65, 1.56 [21:22:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 10.27, 9.80 [21:23:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.33, 1.45 [21:24:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.11, 9.99, 9.76 [21:27:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [21:36:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 10.77, 10.19 [21:45:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.97, 1.83, 1.44 [21:46:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.21, 9.79, 10.11 [21:47:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.63, 1.40 [21:49:25] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.63, 1.91, 1.53 [21:55:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.74, 1.68, 1.60 [21:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.09, 1.64, 1.98 [22:00:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cite-277e004 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/29656bb10175 [22:00:19] [02miraheze/mediawiki] 07dependabot[bot] 0329656bb - Bump extensions/Cite from `88ef259` to `277e004` [22:00:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/EmbedVideo-a274f46 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8012fceb970e [22:00:22] [02miraheze/mediawiki] 07dependabot[bot] 038012fce - Bump extensions/EmbedVideo from `2f0a657` to `a274f46` [22:00:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cite-277e004 - 13https://github.com/miraheze/mediawiki [22:00:24] [url] Page not found · GitHub · GitHub | github.com [22:00:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/EmbedVideo-a274f46 - 13https://github.com/miraheze/mediawiki [22:00:25] ... [22:00:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Thanks-7459982 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/85c0e327f431 [22:00:28] [02miraheze/mediawiki] 07dependabot[bot] 0385c0e32 - Bump extensions/Thanks from `5ed3710` to `7459982` [22:00:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Thanks-7459982 - 13https://github.com/miraheze/mediawiki [22:00:29] ... [22:00:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/VoteNY-34274da [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e8729b905d27 [22:00:32] [02miraheze/mediawiki] 07dependabot[bot] 03e8729b9 - Bump extensions/VoteNY from `a8dd03f` to `34274da` [22:00:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/VoteNY-34274da - 13https://github.com/miraheze/mediawiki [22:00:34] ... [22:00:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ConfirmEdit-db7688f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/aceb4643361a [22:00:37] [02miraheze/mediawiki] 07dependabot[bot] 03aceb464 - Bump extensions/ConfirmEdit from `65f4626` to `db7688f` [22:00:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ConfirmEdit-db7688f - 13https://github.com/miraheze/mediawiki [22:00:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AutoCreatePage-16bbb0d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3403106f25bf [22:00:41] [02miraheze/mediawiki] 07dependabot[bot] 033403106 - Bump extensions/AutoCreatePage from `9476f36` to `16bbb0d` [22:00:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AutoCreatePage-16bbb0d - 13https://github.com/miraheze/mediawiki [22:00:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikimediaIncubator-c5c93e7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a2288a59350a [22:00:46] [02miraheze/mediawiki] 07dependabot[bot] 03a2288a5 - Bump extensions/WikimediaIncubator from `35d9875` to `c5c93e7` [22:00:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8070: Bump extensions/WikimediaIncubator from `35d9875` to `c5c93e7` - 13https://github.com/miraheze/mediawiki/pull/8070 [22:00:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikimediaIncubator-c5c93e7 - 13https://github.com/miraheze/mediawiki [22:00:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/BetaFeatures-c947142 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a24cf80c9b98 [22:00:52] [02miraheze/mediawiki] 07dependabot[bot] 03a24cf80 - Bump extensions/BetaFeatures from `6684e21` to `c947142` [22:00:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/BetaFeatures-c947142 - 13https://github.com/miraheze/mediawiki [22:00:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Flow-b6d1146 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bdb4e4e7ea0d [22:00:56] [02miraheze/mediawiki] 07dependabot[bot] 03bdb4e4e - Bump extensions/Flow from `220ad44` to `b6d1146` [22:00:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8072: Bump extensions/Flow from `220ad44` to `b6d1146` - 13https://github.com/miraheze/mediawiki/pull/8072 [22:00:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Flow-b6d1146 - 13https://github.com/miraheze/mediawiki [22:01:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-3a9ba46 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/33bd86db5908 [22:01:02] [02miraheze/mediawiki] 07dependabot[bot] 0333bd86d - Bump extensions/SemanticMediaWiki from `da63f63` to `3a9ba46` [22:01:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-3a9ba46 - 13https://github.com/miraheze/mediawiki [22:01:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UploadWizard-406c34d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8be3be83dedc [22:01:07] [02miraheze/mediawiki] 07dependabot[bot] 038be3be8 - Bump extensions/UploadWizard from `99348b2` to `406c34d` [22:01:08] [02mediawiki] 07dependabot[bot] closed pull request 03#7997: Bump extensions/SemanticMediaWiki from `da63f63` to `72458a7` - 13https://github.com/miraheze/mediawiki/pull/7997 [22:01:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UploadWizard-406c34d - 13https://github.com/miraheze/mediawiki [22:01:11] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-72458a7 - 13https://github.com/miraheze/mediawiki [22:01:13] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-72458a7 [22:01:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CentralAuth-76d00f5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/691296cfc429 [22:01:16] [02miraheze/mediawiki] 07dependabot[bot] 03691296c - Bump extensions/CentralAuth from `f558e74` to `76d00f5` [22:01:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CentralAuth-76d00f5 - 13https://github.com/miraheze/mediawiki [22:01:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Vector-56d7eb3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/18a35a4934b4 [22:01:20] [02miraheze/mediawiki] 07dependabot[bot] 0318a35a4 - Bump skins/Vector from `4a102cd` to `56d7eb3` [22:01:22] [02mediawiki] 07dependabot[bot] opened pull request 03#8076: Bump skins/Vector from `4a102cd` to `56d7eb3` - 13https://github.com/miraheze/mediawiki/pull/8076 [22:01:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Vector-56d7eb3 - 13https://github.com/miraheze/mediawiki [22:01:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Interwiki-3e240f9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2af5472bfa3b [22:01:26] [02miraheze/mediawiki] 07dependabot[bot] 032af5472 - Bump extensions/Interwiki from `223bbf8` to `3e240f9` [22:01:28] [02mediawiki] 07dependabot[bot] opened pull request 03#8077: Bump extensions/Interwiki from `223bbf8` to `3e240f9` - 13https://github.com/miraheze/mediawiki/pull/8077 [22:01:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Interwiki-3e240f9 - 13https://github.com/miraheze/mediawiki [22:01:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AbuseFilter-f3a462e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d85ccb4aaeb5 [22:01:32] [02miraheze/mediawiki] 07dependabot[bot] 03d85ccb4 - Bump extensions/AbuseFilter from `97f45ae` to `f3a462e` [22:01:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AbuseFilter-f3a462e - 13https://github.com/miraheze/mediawiki [22:01:35] [02mediawiki] 07dependabot[bot] opened pull request 03#8078: Bump extensions/AbuseFilter from `97f45ae` to `f3a462e` - 13https://github.com/miraheze/mediawiki/pull/8078 [22:01:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5eebe1a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b0a04e029da9 [22:01:38] [02miraheze/mediawiki] 07dependabot[bot] 03b0a04e0 - Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` [22:01:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8079: Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` - 13https://github.com/miraheze/mediawiki/pull/8079 [22:01:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5eebe1a - 13https://github.com/miraheze/mediawiki [22:01:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8070: Bump extensions/WikimediaIncubator from `35d9875` to `c5c93e7` - 13https://github.com/miraheze/mediawiki/pull/8070 [22:01:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Gadgets-5accbd2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b15beaa61fb7 [22:01:46] [02miraheze/mediawiki] 07dependabot[bot] 03b15beaa - Bump extensions/Gadgets from `67f128d` to `5accbd2` [22:01:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Gadgets-5accbd2 - 13https://github.com/miraheze/mediawiki [22:01:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8072: Bump extensions/Flow from `220ad44` to `b6d1146` - 13https://github.com/miraheze/mediawiki/pull/8072 [22:01:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IPInfo-2d544f8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/63ef89d84bcb [22:01:52] [02miraheze/mediawiki] 07dependabot[bot] 0363ef89d - Bump extensions/IPInfo from `d413ad2` to `2d544f8` [22:01:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IPInfo-2d544f8 - 13https://github.com/miraheze/mediawiki [22:01:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-dc01234 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fc9dcadc32fc [22:01:56] [02miraheze/mediawiki] 07dependabot[bot] 03fc9dcad - Bump extensions/MirahezeMagic from `d992b71` to `dc01234` [22:01:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8082: Bump extensions/MirahezeMagic from `d992b71` to `dc01234` - 13https://github.com/miraheze/mediawiki/pull/8082 [22:01:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-dc01234 - 13https://github.com/miraheze/mediawiki [22:02:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MassMessage-dc6a65c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bfafdb0a290f [22:02:02] [02miraheze/mediawiki] 07dependabot[bot] 03bfafdb0 - Bump extensions/MassMessage from `104b3b4` to `dc6a65c` [22:02:04] [02mediawiki] 07dependabot[bot] opened pull request 03#8083: Bump extensions/MassMessage from `104b3b4` to `dc6a65c` - 13https://github.com/miraheze/mediawiki/pull/8083 [22:02:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MassMessage-dc6a65c - 13https://github.com/miraheze/mediawiki [22:02:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Renameuser-d5a7fdf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2c83b0c23909 [22:02:08] [02miraheze/mediawiki] 07dependabot[bot] 032c83b0c - Bump extensions/Renameuser from `f7767b2` to `d5a7fdf` [22:02:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Renameuser-d5a7fdf - 13https://github.com/miraheze/mediawiki [22:02:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8084: Bump extensions/Renameuser from `f7767b2` to `d5a7fdf` - 13https://github.com/miraheze/mediawiki/pull/8084 [22:02:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ReplaceText-0a5b408 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/94fdd3a4051b [22:02:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.72, 10.20, 9.31 [22:02:14] [02miraheze/mediawiki] 07dependabot[bot] 0394fdd3a - Bump extensions/ReplaceText from `7ef7d5a` to `0a5b408` [22:02:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8085: Bump extensions/ReplaceText from `7ef7d5a` to `0a5b408` - 13https://github.com/miraheze/mediawiki/pull/8085 [22:02:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8076: Bump skins/Vector from `4a102cd` to `56d7eb3` - 13https://github.com/miraheze/mediawiki/pull/8076 [22:02:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ReplaceText-0a5b408 - 13https://github.com/miraheze/mediawiki [22:02:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Linter-4b33432 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d14e62313059 [22:02:22] [02miraheze/mediawiki] 07dependabot[bot] 03d14e623 - Bump extensions/Linter from `82570c1` to `4b33432` [22:02:23] [02mediawiki] 07dependabot[bot] opened pull request 03#8086: Bump extensions/Linter from `82570c1` to `4b33432` - 13https://github.com/miraheze/mediawiki/pull/8086 [22:02:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8077: Bump extensions/Interwiki from `223bbf8` to `3e240f9` - 13https://github.com/miraheze/mediawiki/pull/8077 [22:02:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Linter-4b33432 - 13https://github.com/miraheze/mediawiki [22:02:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-53a897b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/575dda42c450 [22:02:29] [02miraheze/mediawiki] 07dependabot[bot] 03575dda4 - Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` [22:02:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8079: Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` - 13https://github.com/miraheze/mediawiki/pull/8079 [22:02:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:02:32] [02mediawiki] 07dependabot[bot] opened pull request 03#8087: Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` - 13https://github.com/miraheze/mediawiki/pull/8087 [22:02:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-53a897b - 13https://github.com/miraheze/mediawiki [22:02:34] [url] Page not found · GitHub · GitHub | github.com [22:02:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#8082: Bump extensions/MirahezeMagic from `d992b71` to `dc01234` - 13https://github.com/miraheze/mediawiki/pull/8082 [22:02:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-af3112d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/77497d3c81f0 [22:02:38] [02miraheze/mediawiki] 07dependabot[bot] 0377497d3 - Bump extensions/FlaggedRevs from `4bde537` to `af3112d` [22:02:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8088: Bump extensions/FlaggedRevs from `4bde537` to `af3112d` - 13https://github.com/miraheze/mediawiki/pull/8088 [22:02:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-af3112d - 13https://github.com/miraheze/mediawiki [22:02:43] [02mediawiki] 07dependabot[bot] commented on pull request 03#8051: Bump extensions/FlaggedRevs from `4bde537` to `211926b` - 13https://github.com/miraheze/mediawiki/pull/8051#issuecomment-1184932996 [22:02:43] [url] Bump extensions/FlaggedRevs from `4bde537` to `211926b` by dependabot[bot] · Pull Request #8051 · miraheze/mediawiki · GitHub | github.com [22:02:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DiscordNotifications-b836b55 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bde2bac3dbbc [22:02:46] [02miraheze/mediawiki] 07dependabot[bot] 03bde2bac - Bump extensions/DiscordNotifications from `a470753` to `b836b55` [22:02:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8089: Bump extensions/DiscordNotifications from `a470753` to `b836b55` - 13https://github.com/miraheze/mediawiki/pull/8089 [22:02:48] [url] Page not found · GitHub · GitHub | github.com [22:02:49] [02mediawiki] 07dependabot[bot] closed pull request 03#8051: Bump extensions/FlaggedRevs from `4bde537` to `211926b` - 13https://github.com/miraheze/mediawiki/pull/8051 [22:02:49] [url] Page not found · GitHub · GitHub | github.com [22:02:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DiscordNotifications-b836b55 - 13https://github.com/miraheze/mediawiki [22:02:51] [url] Page not found · GitHub · GitHub | github.com [22:02:52] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-211926b [22:02:53] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-211926b - 13https://github.com/miraheze/mediawiki [22:02:54] [url] Page not found · GitHub · GitHub | github.com [22:02:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8083: Bump extensions/MassMessage from `104b3b4` to `dc6a65c` - 13https://github.com/miraheze/mediawiki/pull/8083 [22:02:55] ... [22:02:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalUsage-6f3dce4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a7a5e78aee31 [22:02:58] [02miraheze/mediawiki] 07dependabot[bot] 03a7a5e78 - Bump extensions/GlobalUsage from `59c6715` to `6f3dce4` [22:02:59] [02mediawiki] 07dependabot[bot] opened pull request 03#8090: Bump extensions/GlobalUsage from `59c6715` to `6f3dce4` - 13https://github.com/miraheze/mediawiki/pull/8090 [22:03:00] ... [22:03:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalUsage-6f3dce4 - 13https://github.com/miraheze/mediawiki [22:03:01] ... [22:03:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#8078: Bump extensions/AbuseFilter from `97f45ae` to `f3a462e` - 13https://github.com/miraheze/mediawiki/pull/8078 [22:03:03] [url] Page not found · GitHub · GitHub | github.com [22:03:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/MinervaNeue-b1c26c4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6b8807969d9b [22:03:05] [02miraheze/mediawiki] 07dependabot[bot] 036b88079 - Bump skins/MinervaNeue from `7c9bccd` to `b1c26c4` [22:03:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8091: Bump skins/MinervaNeue from `7c9bccd` to `b1c26c4` - 13https://github.com/miraheze/mediawiki/pull/8091 [22:03:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/MinervaNeue-b1c26c4 - 13https://github.com/miraheze/mediawiki [22:03:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8084: Bump extensions/Renameuser from `f7767b2` to `d5a7fdf` - 13https://github.com/miraheze/mediawiki/pull/8084 [22:03:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MobileFrontend-ac9708a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5f8077eeb518 [22:03:13] [02miraheze/mediawiki] 07dependabot[bot] 035f8077e - Bump extensions/MobileFrontend from `f4a2105` to `ac9708a` [22:03:14] [02mediawiki] 07dependabot[bot] opened pull request 03#8092: Bump extensions/MobileFrontend from `f4a2105` to `ac9708a` - 13https://github.com/miraheze/mediawiki/pull/8092 [22:03:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MobileFrontend-ac9708a - 13https://github.com/miraheze/mediawiki [22:03:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikibaseLexeme-cf2859a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/723adc55a4ee [22:03:19] [02miraheze/mediawiki] 07dependabot[bot] 03723adc5 - Bump extensions/WikibaseLexeme from `c9bdebd` to `cf2859a` [22:03:20] [02mediawiki] 07dependabot[bot] opened pull request 03#8093: Bump extensions/WikibaseLexeme from `c9bdebd` to `cf2859a` - 13https://github.com/miraheze/mediawiki/pull/8093 [22:03:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikibaseLexeme-cf2859a - 13https://github.com/miraheze/mediawiki [22:03:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8085: Bump extensions/ReplaceText from `7ef7d5a` to `0a5b408` - 13https://github.com/miraheze/mediawiki/pull/8085 [22:03:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CreateWiki-33899c0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a2eb6acd54aa [22:03:26] [02miraheze/mediawiki] 07dependabot[bot] 03a2eb6ac - Bump extensions/CreateWiki from `e96869b` to `33899c0` [22:03:28] [02mediawiki] 07dependabot[bot] opened pull request 03#8094: Bump extensions/CreateWiki from `e96869b` to `33899c0` - 13https://github.com/miraheze/mediawiki/pull/8094 [22:03:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-33899c0 - 13https://github.com/miraheze/mediawiki [22:03:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ArticleCreationWorkflow-70a8df2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2f1f93818373 [22:03:32] [02miraheze/mediawiki] 07dependabot[bot] 032f1f938 - Bump extensions/ArticleCreationWorkflow from `424d763` to `70a8df2` [22:03:34] [02mediawiki] 07dependabot[bot] opened pull request 03#8095: Bump extensions/ArticleCreationWorkflow from `424d763` to `70a8df2` - 13https://github.com/miraheze/mediawiki/pull/8095 [22:03:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ArticleCreationWorkflow-70a8df2 - 13https://github.com/miraheze/mediawiki [22:03:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ExternalData-28bcaf3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f6ef56e56b24 [22:03:38] [02miraheze/mediawiki] 07dependabot[bot] 03f6ef56e - Bump extensions/ExternalData from `52025c9` to `28bcaf3` [22:03:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8096: Bump extensions/ExternalData from `52025c9` to `28bcaf3` - 13https://github.com/miraheze/mediawiki/pull/8096 [22:03:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#8087: Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` - 13https://github.com/miraheze/mediawiki/pull/8087 [22:03:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ExternalData-28bcaf3 - 13https://github.com/miraheze/mediawiki [22:03:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DiscussionTools-5078b24 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ecce4216021c [22:03:46] [02miraheze/mediawiki] 07dependabot[bot] 03ecce421 - Bump extensions/DiscussionTools from `9ce8f3f` to `5078b24` [22:03:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8097: Bump extensions/DiscussionTools from `9ce8f3f` to `5078b24` - 13https://github.com/miraheze/mediawiki/pull/8097 [22:03:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DiscussionTools-5078b24 - 13https://github.com/miraheze/mediawiki [22:03:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiDiscover-76c8b43 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2929b8a50825 [22:03:52] [02miraheze/mediawiki] 07dependabot[bot] 032929b8a - Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` [22:03:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8098: Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` - 13https://github.com/miraheze/mediawiki/pull/8098 [22:03:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiDiscover-76c8b43 - 13https://github.com/miraheze/mediawiki [22:03:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8090: Bump extensions/GlobalUsage from `59c6715` to `6f3dce4` - 13https://github.com/miraheze/mediawiki/pull/8090 [22:03:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8088: Bump extensions/FlaggedRevs from `4bde537` to `af3112d` - 13https://github.com/miraheze/mediawiki/pull/8088 [22:03:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#8088: Bump extensions/FlaggedRevs from `4bde537` to `af3112d` - 13https://github.com/miraheze/mediawiki/pull/8088 [22:04:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Echo-9d48375 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e50e22cfefab [22:04:02] [02miraheze/mediawiki] 07dependabot[bot] 03e50e22c - Bump extensions/Echo from `0a7c3d0` to `9d48375` [22:04:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8089: Bump extensions/DiscordNotifications from `a470753` to `b836b55` - 13https://github.com/miraheze/mediawiki/pull/8089 [22:04:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#8088: Bump extensions/FlaggedRevs from `4bde537` to `af3112d` - 13https://github.com/miraheze/mediawiki/pull/8088 [22:04:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8099: Bump extensions/Echo from `0a7c3d0` to `9d48375` - 13https://github.com/miraheze/mediawiki/pull/8099 [22:04:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 10.84, 9.65 [22:04:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8088: Bump extensions/FlaggedRevs from `4bde537` to `af3112d` - 13https://github.com/miraheze/mediawiki/pull/8088 [22:04:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Echo-9d48375 - 13https://github.com/miraheze/mediawiki [22:04:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cargo-b7ed78b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1e393880b6b5 [22:04:13] [02miraheze/mediawiki] 07dependabot[bot] 031e39388 - Bump extensions/Cargo from `5ecfef7` to `b7ed78b` [22:04:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#8091: Bump skins/MinervaNeue from `7c9bccd` to `b1c26c4` - 13https://github.com/miraheze/mediawiki/pull/8091 [22:04:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8100: Bump extensions/Cargo from `5ecfef7` to `b7ed78b` - 13https://github.com/miraheze/mediawiki/pull/8100 [22:04:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cargo-b7ed78b - 13https://github.com/miraheze/mediawiki [22:04:19] [02mediawiki] 07dependabot[bot] commented on pull request 03#8040: Bump extensions/Cargo from `5ecfef7` to `43a9e87` - 13https://github.com/miraheze/mediawiki/pull/8040#issuecomment-1184933374 [22:04:20] [url] Bump extensions/Cargo from `5ecfef7` to `43a9e87` by dependabot[bot] · Pull Request #8040 · miraheze/mediawiki · GitHub | github.com [22:04:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalPreferences-3a88c28 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c84950300258 [22:04:22] [02miraheze/mediawiki] 07dependabot[bot] 03c849503 - Bump extensions/GlobalPreferences from `624ca06` to `3a88c28` [22:04:23] [02mediawiki] 07dependabot[bot] closed pull request 03#8040: Bump extensions/Cargo from `5ecfef7` to `43a9e87` - 13https://github.com/miraheze/mediawiki/pull/8040 [22:04:24] [url] Page not found · GitHub · GitHub | github.com [22:04:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8101: Bump extensions/GlobalPreferences from `624ca06` to `3a88c28` - 13https://github.com/miraheze/mediawiki/pull/8101 [22:04:25] [url] Page not found · GitHub · GitHub | github.com [22:04:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalPreferences-3a88c28 - 13https://github.com/miraheze/mediawiki [22:04:27] [url] Page not found · GitHub · GitHub | github.com [22:04:28] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-43a9e87 [22:04:29] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Cargo-43a9e87 - 13https://github.com/miraheze/mediawiki [22:04:30] [url] Page not found · GitHub · GitHub | github.com [22:04:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8092: Bump extensions/MobileFrontend from `f4a2105` to `ac9708a` - 13https://github.com/miraheze/mediawiki/pull/8092 [22:04:31] ... [22:04:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#8093: Bump extensions/WikibaseLexeme from `c9bdebd` to `cf2859a` - 13https://github.com/miraheze/mediawiki/pull/8093 [22:04:33] ... [22:04:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CentralNotice-8052c24 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/10a94f5ea310 [22:04:35] [02miraheze/mediawiki] 07dependabot[bot] 0310a94f5 - Bump extensions/CentralNotice from `57cfc5f` to `8052c24` [22:04:37] [02mediawiki] 07dependabot[bot] opened pull request 03#8102: Bump extensions/CentralNotice from `57cfc5f` to `8052c24` - 13https://github.com/miraheze/mediawiki/pull/8102 [22:04:37] ... [22:04:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CentralNotice-8052c24 - 13https://github.com/miraheze/mediawiki [22:04:39] [url] Page not found · GitHub · GitHub | github.com [22:04:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalBlocking-1294e73 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0d1e51cd6d4e [22:04:41] [02miraheze/mediawiki] 07dependabot[bot] 030d1e51c - Bump extensions/GlobalBlocking from `efd8fef` to `1294e73` [22:04:43] [02mediawiki] 07dependabot[bot] opened pull request 03#8103: Bump extensions/GlobalBlocking from `efd8fef` to `1294e73` - 13https://github.com/miraheze/mediawiki/pull/8103 [22:04:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalBlocking-1294e73 - 13https://github.com/miraheze/mediawiki [22:04:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8095: Bump extensions/ArticleCreationWorkflow from `424d763` to `70a8df2` - 13https://github.com/miraheze/mediawiki/pull/8095 [22:04:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ManageWiki-202eaec [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a1c73ab4e61c [22:04:49] [02miraheze/mediawiki] 07dependabot[bot] 03a1c73ab - Bump extensions/ManageWiki from `b0139c2` to `202eaec` [22:04:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#8095: Bump extensions/ArticleCreationWorkflow from `424d763` to `70a8df2` - 13https://github.com/miraheze/mediawiki/pull/8095 [22:04:52] [02mediawiki] 07dependabot[bot] opened pull request 03#8104: Bump extensions/ManageWiki from `b0139c2` to `202eaec` - 13https://github.com/miraheze/mediawiki/pull/8104 [22:04:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-202eaec - 13https://github.com/miraheze/mediawiki [22:04:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8096: Bump extensions/ExternalData from `52025c9` to `28bcaf3` - 13https://github.com/miraheze/mediawiki/pull/8096 [22:04:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/apex-b0f3fdd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f240e5702fd1 [22:04:58] [02miraheze/mediawiki] 07dependabot[bot] 03f240e57 - Bump skins/apex from `4a3ff0a` to `b0f3fdd` [22:04:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#8094: Bump extensions/CreateWiki from `e96869b` to `33899c0` - 13https://github.com/miraheze/mediawiki/pull/8094 [22:05:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8105: Bump skins/apex from `4a3ff0a` to `b0f3fdd` - 13https://github.com/miraheze/mediawiki/pull/8105 [22:05:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/apex-b0f3fdd - 13https://github.com/miraheze/mediawiki [22:05:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8094: Bump extensions/CreateWiki from `e96869b` to `33899c0` - 13https://github.com/miraheze/mediawiki/pull/8094 [22:05:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#8099: Bump extensions/Echo from `0a7c3d0` to `9d48375` - 13https://github.com/miraheze/mediawiki/pull/8099 [22:05:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8097: Bump extensions/DiscussionTools from `9ce8f3f` to `5078b24` - 13https://github.com/miraheze/mediawiki/pull/8097 [22:05:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GrowthExperiments-16156a8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/845b68c49aa3 [22:05:10] [02miraheze/mediawiki] 07dependabot[bot] 03845b68c - Bump extensions/GrowthExperiments from `023b910` to `16156a8` [22:05:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8106: Bump extensions/GrowthExperiments from `023b910` to `16156a8` - 13https://github.com/miraheze/mediawiki/pull/8106 [22:05:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GrowthExperiments-16156a8 - 13https://github.com/miraheze/mediawiki [22:05:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#8098: Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` - 13https://github.com/miraheze/mediawiki/pull/8098 [22:05:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CleanChanges-31e7f12 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4795086bfd1b [22:05:17] [02miraheze/mediawiki] 07dependabot[bot] 034795086 - Bump extensions/CleanChanges from `873cd45` to `31e7f12` [22:05:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8107: Bump extensions/CleanChanges from `873cd45` to `31e7f12` - 13https://github.com/miraheze/mediawiki/pull/8107 [22:05:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CleanChanges-31e7f12 - 13https://github.com/miraheze/mediawiki [22:05:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Preloader-6628779 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bfec9507b978 [22:05:23] [02miraheze/mediawiki] 07dependabot[bot] 03bfec950 - Bump extensions/Preloader from `58f20ef` to `6628779` [22:05:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8108: Bump extensions/Preloader from `58f20ef` to `6628779` - 13https://github.com/miraheze/mediawiki/pull/8108 [22:05:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Preloader-6628779 - 13https://github.com/miraheze/mediawiki [22:05:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageTriage-4aec616 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fed489222a4b [22:05:29] [02miraheze/mediawiki] 07dependabot[bot] 03fed4892 - Bump extensions/PageTriage from `43f2088` to `4aec616` [22:05:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8109: Bump extensions/PageTriage from `43f2088` to `4aec616` - 13https://github.com/miraheze/mediawiki/pull/8109 [22:05:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageTriage-4aec616 - 13https://github.com/miraheze/mediawiki [22:05:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8104: Bump extensions/ManageWiki from `b0139c2` to `202eaec` - 13https://github.com/miraheze/mediawiki/pull/8104 [22:05:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#8100: Bump extensions/Cargo from `5ecfef7` to `b7ed78b` - 13https://github.com/miraheze/mediawiki/pull/8100 [22:05:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8100: Bump extensions/Cargo from `5ecfef7` to `b7ed78b` - 13https://github.com/miraheze/mediawiki/pull/8100 [22:05:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Newsletter-fa4fc25 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a4d553b4a452 [22:05:40] [02miraheze/mediawiki] 07dependabot[bot] 03a4d553b - Bump extensions/Newsletter from `d97107b` to `fa4fc25` [22:05:41] [02mediawiki] 07dependabot[bot] opened pull request 03#8110: Bump extensions/Newsletter from `d97107b` to `fa4fc25` - 13https://github.com/miraheze/mediawiki/pull/8110 [22:05:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Newsletter-fa4fc25 - 13https://github.com/miraheze/mediawiki [22:05:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#8101: Bump extensions/GlobalPreferences from `624ca06` to `3a88c28` - 13https://github.com/miraheze/mediawiki/pull/8101 [22:05:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8102: Bump extensions/CentralNotice from `57cfc5f` to `8052c24` - 13https://github.com/miraheze/mediawiki/pull/8102 [22:05:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/StandardDialogs-8a83feb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/319c4e0f28b6 [22:05:49] [02miraheze/mediawiki] 07dependabot[bot] 03319c4e0 - Bump extensions/StandardDialogs from `66afa0c` to `8a83feb` [22:05:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8111: Bump extensions/StandardDialogs from `66afa0c` to `8a83feb` - 13https://github.com/miraheze/mediawiki/pull/8111 [22:05:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/StandardDialogs-8a83feb - 13https://github.com/miraheze/mediawiki [22:05:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#8103: Bump extensions/GlobalBlocking from `efd8fef` to `1294e73` - 13https://github.com/miraheze/mediawiki/pull/8103 [22:05:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SimpleTooltip-5986ddf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d9cf9f79df26 [22:05:56] [02miraheze/mediawiki] 07dependabot[bot] 03d9cf9f7 - Bump extensions/SimpleTooltip from `383fa88` to `5986ddf` [22:05:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8112: Bump extensions/SimpleTooltip from `383fa88` to `5986ddf` - 13https://github.com/miraheze/mediawiki/pull/8112 [22:05:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SimpleTooltip-5986ddf - 13https://github.com/miraheze/mediawiki [22:06:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CommentStreams-0c120c4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ca18a3b9843 [22:06:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.21, 10.35, 9.60 [22:06:02] [02miraheze/mediawiki] 07dependabot[bot] 037ca18a3 - Bump extensions/CommentStreams from `27c8706` to `0c120c4` [22:06:04] [02mediawiki] 07dependabot[bot] opened pull request 03#8113: Bump extensions/CommentStreams from `27c8706` to `0c120c4` - 13https://github.com/miraheze/mediawiki/pull/8113 [22:06:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CommentStreams-0c120c4 - 13https://github.com/miraheze/mediawiki [22:06:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Babel-ecaf9c4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/80912c88b605 [22:06:08] [02miraheze/mediawiki] 07dependabot[bot] 0380912c8 - Bump extensions/Babel from `3a152df` to `ecaf9c4` [22:06:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8106: Bump extensions/GrowthExperiments from `023b910` to `16156a8` - 13https://github.com/miraheze/mediawiki/pull/8106 [22:06:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8114: Bump extensions/Babel from `3a152df` to `ecaf9c4` - 13https://github.com/miraheze/mediawiki/pull/8114 [22:06:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8086: Bump extensions/Linter from `82570c1` to `4b33432` - 13https://github.com/miraheze/mediawiki/pull/8086 [22:06:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Babel-ecaf9c4 - 13https://github.com/miraheze/mediawiki [22:06:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8105: Bump skins/apex from `4a3ff0a` to `b0f3fdd` - 13https://github.com/miraheze/mediawiki/pull/8105 [22:06:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8107: Bump extensions/CleanChanges from `873cd45` to `31e7f12` - 13https://github.com/miraheze/mediawiki/pull/8107 [22:06:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8115: Bump extensions/PortableInfobox from `abc29f3` to `947cf10` - 13https://github.com/miraheze/mediawiki/pull/8115 [22:06:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#8107: Bump extensions/CleanChanges from `873cd45` to `31e7f12` - 13https://github.com/miraheze/mediawiki/pull/8107 [22:06:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PortableInfobox-947cf10 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/97eaa0242f7b [22:06:23] [02miraheze/mediawiki] 07dependabot[bot] 0397eaa02 - Bump extensions/PortableInfobox from `abc29f3` to `947cf10` [22:06:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PortableInfobox-947cf10 - 13https://github.com/miraheze/mediawiki [22:06:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#8108: Bump extensions/Preloader from `58f20ef` to `6628779` - 13https://github.com/miraheze/mediawiki/pull/8108 [22:06:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8109: Bump extensions/PageTriage from `43f2088` to `4aec616` - 13https://github.com/miraheze/mediawiki/pull/8109 [22:06:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ArticleToCategory2-854895b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0881f05e0f51 [22:06:31] [02miraheze/mediawiki] 07dependabot[bot] 030881f05 - Bump extensions/ArticleToCategory2 from `5cea246` to `854895b` [22:06:32] [02mediawiki] 07dependabot[bot] opened pull request 03#8116: Bump extensions/ArticleToCategory2 from `5cea246` to `854895b` - 13https://github.com/miraheze/mediawiki/pull/8116 [22:06:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ArticleToCategory2-854895b - 13https://github.com/miraheze/mediawiki [22:06:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#8111: Bump extensions/StandardDialogs from `66afa0c` to `8a83feb` - 13https://github.com/miraheze/mediawiki/pull/8111 [22:06:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8111: Bump extensions/StandardDialogs from `66afa0c` to `8a83feb` - 13https://github.com/miraheze/mediawiki/pull/8111 [22:06:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#8110: Bump extensions/Newsletter from `d97107b` to `fa4fc25` - 13https://github.com/miraheze/mediawiki/pull/8110 [22:06:39] [url] Page not found · GitHub · GitHub | github.com [22:06:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-87d4512 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6aa9385c70ef [22:06:41] [02miraheze/mediawiki] 07dependabot[bot] 036aa9385 - Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` [22:06:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.01, 10.31 [22:06:43] [02mediawiki] 07dependabot[bot] opened pull request 03#8117: Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` - 13https://github.com/miraheze/mediawiki/pull/8117 [22:06:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-87d4512 - 13https://github.com/miraheze/mediawiki [22:06:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikibaseQualityConstraints-b2c6062 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a40e474dd9cd [22:06:47] [02miraheze/mediawiki] 07dependabot[bot] 03a40e474 - Bump extensions/WikibaseQualityConstraints from `c6fbf36` to `b2c6062` [22:06:49] [02mediawiki] 07dependabot[bot] opened pull request 03#8118: Bump extensions/WikibaseQualityConstraints from `c6fbf36` to `b2c6062` - 13https://github.com/miraheze/mediawiki/pull/8118 [22:06:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikibaseQualityConstraints-b2c6062 - 13https://github.com/miraheze/mediawiki [22:06:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Report-7ad7b58 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/295ba8fd2310 [22:06:53] [02miraheze/mediawiki] 07dependabot[bot] 03295ba8f - Bump extensions/Report from `2161d9a` to `7ad7b58` [22:06:55] [02mediawiki] 07dependabot[bot] opened pull request 03#8119: Bump extensions/Report from `2161d9a` to `7ad7b58` - 13https://github.com/miraheze/mediawiki/pull/8119 [22:06:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Report-7ad7b58 - 13https://github.com/miraheze/mediawiki [22:06:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TimedMediaHandler-3fc8b37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de6dfcef9077 [22:06:59] [02miraheze/mediawiki] 07dependabot[bot] 03de6dfce - Bump extensions/TimedMediaHandler from `7d70b57` to `3fc8b37` [22:07:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8120: Bump extensions/TimedMediaHandler from `7d70b57` to `3fc8b37` - 13https://github.com/miraheze/mediawiki/pull/8120 [22:07:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TimedMediaHandler-3fc8b37 - 13https://github.com/miraheze/mediawiki [22:07:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/RemovePII-538bafc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6a7286dbf7a1 [22:07:05] [02miraheze/mediawiki] 07dependabot[bot] 036a7286d - Bump extensions/RemovePII from `3be5c88` to `538bafc` [22:07:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8121: Bump extensions/RemovePII from `3be5c88` to `538bafc` - 13https://github.com/miraheze/mediawiki/pull/8121 [22:07:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/RemovePII-538bafc - 13https://github.com/miraheze/mediawiki [22:07:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8112: Bump extensions/SimpleTooltip from `383fa88` to `5986ddf` - 13https://github.com/miraheze/mediawiki/pull/8112 [22:07:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5faa403 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/882f5d14eada [22:07:13] [02miraheze/mediawiki] 07dependabot[bot] 03882f5d1 - Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` [22:07:15] [02mediawiki] 07dependabot[bot] opened pull request 03#8122: Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` - 13https://github.com/miraheze/mediawiki/pull/8122 [22:07:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5faa403 - 13https://github.com/miraheze/mediawiki [22:07:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8114: Bump extensions/Babel from `3a152df` to `ecaf9c4` - 13https://github.com/miraheze/mediawiki/pull/8114 [22:07:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8113: Bump extensions/CommentStreams from `27c8706` to `0c120c4` - 13https://github.com/miraheze/mediawiki/pull/8113 [22:07:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/OAuth-f7c2faf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/10df8302c957 [22:07:22] [02miraheze/mediawiki] 07dependabot[bot] 0310df830 - Bump extensions/OAuth from `17a9fd3` to `f7c2faf` [22:07:24] [02mediawiki] 07dependabot[bot] opened pull request 03#8123: Bump extensions/OAuth from `17a9fd3` to `f7c2faf` - 13https://github.com/miraheze/mediawiki/pull/8123 [22:07:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/OAuth-f7c2faf - 13https://github.com/miraheze/mediawiki [22:07:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#8115: Bump extensions/PortableInfobox from `abc29f3` to `947cf10` - 13https://github.com/miraheze/mediawiki/pull/8115 [22:07:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiEditor-039dbc1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/90c80d173e7b [22:07:30] [02miraheze/mediawiki] 07dependabot[bot] 0390c80d1 - Bump extensions/WikiEditor from `fe7ec6d` to `039dbc1` [22:07:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8124: Bump extensions/WikiEditor from `fe7ec6d` to `039dbc1` - 13https://github.com/miraheze/mediawiki/pull/8124 [22:07:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiEditor-039dbc1 - 13https://github.com/miraheze/mediawiki [22:07:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8116: Bump extensions/ArticleToCategory2 from `5cea246` to `854895b` - 13https://github.com/miraheze/mediawiki/pull/8116 [22:07:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8116: Bump extensions/ArticleToCategory2 from `5cea246` to `854895b` - 13https://github.com/miraheze/mediawiki/pull/8116 [22:07:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8119: Bump extensions/Report from `2161d9a` to `7ad7b58` - 13https://github.com/miraheze/mediawiki/pull/8119 [22:07:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MintyDocs-38f5e0f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7da3b9e37538 [22:07:40] [02miraheze/mediawiki] 07dependabot[bot] 037da3b9e - Bump extensions/MintyDocs from `b39484a` to `38f5e0f` [22:07:42] [02mediawiki] 07dependabot[bot] opened pull request 03#8125: Bump extensions/MintyDocs from `b39484a` to `38f5e0f` - 13https://github.com/miraheze/mediawiki/pull/8125 [22:07:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-38f5e0f - 13https://github.com/miraheze/mediawiki [22:07:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Nimbus-38b1bb0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5ca11e150223 [22:07:46] [02miraheze/mediawiki] 07dependabot[bot] 035ca11e1 - Bump skins/Nimbus from `af8d8a5` to `38b1bb0` [22:07:48] [02mediawiki] 07dependabot[bot] opened pull request 03#8126: Bump skins/Nimbus from `af8d8a5` to `38b1bb0` - 13https://github.com/miraheze/mediawiki/pull/8126 [22:07:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Nimbus-38b1bb0 - 13https://github.com/miraheze/mediawiki [22:07:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8118: Bump extensions/WikibaseQualityConstraints from `c6fbf36` to `b2c6062` - 13https://github.com/miraheze/mediawiki/pull/8118 [22:07:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Commentbox-6eb89d0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4272b46fc1b7 [22:07:54] [02miraheze/mediawiki] 07dependabot[bot] 034272b46 - Bump extensions/Commentbox from `e49d615` to `6eb89d0` [22:07:55] [02mediawiki] 07dependabot[bot] opened pull request 03#8127: Bump extensions/Commentbox from `e49d615` to `6eb89d0` - 13https://github.com/miraheze/mediawiki/pull/8127 [22:07:56] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.14, 9.35, 9.32 [22:07:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Commentbox-6eb89d0 - 13https://github.com/miraheze/mediawiki [22:07:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8117: Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` - 13https://github.com/miraheze/mediawiki/pull/8117 [22:08:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#8122: Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` - 13https://github.com/miraheze/mediawiki/pull/8122 [22:08:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8122: Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` - 13https://github.com/miraheze/mediawiki/pull/8122 [22:08:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-e1c23d2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dd2a967b09c5 [22:08:04] [02miraheze/mediawiki] 07dependabot[bot] 03dd2a967 - Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` [22:08:06] [02mediawiki] 07dependabot[bot] opened pull request 03#8128: Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` - 13https://github.com/miraheze/mediawiki/pull/8128 [22:08:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8120: Bump extensions/TimedMediaHandler from `7d70b57` to `3fc8b37` - 13https://github.com/miraheze/mediawiki/pull/8120 [22:08:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-e1c23d2 - 13https://github.com/miraheze/mediawiki [22:08:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CheckUser-fe9b1a2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/763c8b120b54 [22:08:12] [02miraheze/mediawiki] 07dependabot[bot] 03763c8b1 - Bump extensions/CheckUser from `bf74001` to `fe9b1a2` [22:08:13] [02mediawiki] 07dependabot[bot] opened pull request 03#8129: Bump extensions/CheckUser from `bf74001` to `fe9b1a2` - 13https://github.com/miraheze/mediawiki/pull/8129 [22:08:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CheckUser-fe9b1a2 - 13https://github.com/miraheze/mediawiki [22:08:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8121: Bump extensions/RemovePII from `3be5c88` to `538bafc` - 13https://github.com/miraheze/mediawiki/pull/8121 [22:08:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Maps-b7798a0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/85a15dd3e740 [22:08:19] [02miraheze/mediawiki] 07dependabot[bot] 0385a15dd - Bump extensions/Maps from `4cd83f9` to `b7798a0` [22:08:21] [02mediawiki] 07dependabot[bot] opened pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130 [22:08:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Maps-b7798a0 - 13https://github.com/miraheze/mediawiki [22:08:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#8124: Bump extensions/WikiEditor from `fe7ec6d` to `039dbc1` - 13https://github.com/miraheze/mediawiki/pull/8124 [22:08:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageSchemas-fce4265 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/422a246f90dc [22:08:27] [02miraheze/mediawiki] 07dependabot[bot] 03422a246 - Bump extensions/PageSchemas from `1d16b69` to `fce4265` [22:08:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#8038: Bump extensions/Maps from `4cd83f9` to `257381b` - 13https://github.com/miraheze/mediawiki/pull/8038#issuecomment-1184939282 [22:08:29] [url] Bump extensions/Maps from `4cd83f9` to `257381b` by dependabot[bot] · Pull Request #8038 · miraheze/mediawiki · GitHub | github.com [22:08:30] [02mediawiki] 07dependabot[bot] opened pull request 03#8131: Bump extensions/PageSchemas from `1d16b69` to `fce4265` - 13https://github.com/miraheze/mediawiki/pull/8131 [22:08:30] [url] Page not found · GitHub · GitHub | github.com [22:08:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-fce4265 - 13https://github.com/miraheze/mediawiki [22:08:31] [url] Page not found · GitHub · GitHub | github.com [22:08:33] [02mediawiki] 07dependabot[bot] closed pull request 03#8038: Bump extensions/Maps from `4cd83f9` to `257381b` - 13https://github.com/miraheze/mediawiki/pull/8038 [22:08:33] [url] Page not found · GitHub · GitHub | github.com [22:08:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-257381b [22:08:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-257381b - 13https://github.com/miraheze/mediawiki [22:08:36] ... [22:08:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-c61c3cb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3c13f6c00db4 [22:08:39] [02miraheze/mediawiki] 07dependabot[bot] 033c13f6c - Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` [22:08:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8125: Bump extensions/MintyDocs from `b39484a` to `38f5e0f` - 13https://github.com/miraheze/mediawiki/pull/8125 [22:08:41] ... [22:08:42] [02mediawiki] 07dependabot[bot] opened pull request 03#8132: Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` - 13https://github.com/miraheze/mediawiki/pull/8132 [22:08:42] ... [22:08:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8123: Bump extensions/OAuth from `17a9fd3` to `f7c2faf` - 13https://github.com/miraheze/mediawiki/pull/8123 [22:08:44] [url] Page not found · GitHub · GitHub | github.com [22:08:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8126: Bump skins/Nimbus from `af8d8a5` to `38b1bb0` - 13https://github.com/miraheze/mediawiki/pull/8126 [22:08:45] [url] Page not found · GitHub · GitHub | github.com [22:08:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-c61c3cb - 13https://github.com/miraheze/mediawiki [22:08:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageForms-9dc2162 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/48b81324bb9e [22:08:49] [02miraheze/mediawiki] 07dependabot[bot] 0348b8132 - Bump extensions/PageForms from `7730130` to `9dc2162` [22:08:51] [02mediawiki] 07dependabot[bot] opened pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133 [22:08:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageForms-9dc2162 - 13https://github.com/miraheze/mediawiki [22:08:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CategoryTree-56d02d3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/78a557f3c505 [22:08:55] [02miraheze/mediawiki] 07dependabot[bot] 0378a557f - Bump extensions/CategoryTree from `ed9789d` to `56d02d3` [22:08:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8134: Bump extensions/CategoryTree from `ed9789d` to `56d02d3` - 13https://github.com/miraheze/mediawiki/pull/8134 [22:08:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CategoryTree-56d02d3 - 13https://github.com/miraheze/mediawiki [22:09:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#8034: Bump extensions/PageForms from `7730130` to `2f7b830` - 13https://github.com/miraheze/mediawiki/pull/8034#issuecomment-1184940373 [22:09:00] [url] Bump extensions/PageForms from `7730130` to `2f7b830` by dependabot[bot] · Pull Request #8034 · miraheze/mediawiki · GitHub | github.com [22:09:01] [02mediawiki] 07dependabot[bot] closed pull request 03#8034: Bump extensions/PageForms from `7730130` to `2f7b830` - 13https://github.com/miraheze/mediawiki/pull/8034 [22:09:02] [url] Page not found · GitHub · GitHub | github.com [22:09:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/QuizGame-7bd6e41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/01d6bb5442fa [22:09:04] [02miraheze/mediawiki] 07dependabot[bot] 0301d6bb5 - Bump extensions/QuizGame from `00b2fea` to `7bd6e41` [22:09:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-2f7b830 [22:09:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-2f7b830 - 13https://github.com/miraheze/mediawiki [22:09:07] [url] Page not found · GitHub · GitHub | github.com [22:09:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#8127: Bump extensions/Commentbox from `e49d615` to `6eb89d0` - 13https://github.com/miraheze/mediawiki/pull/8127 [22:09:09] [url] Page not found · GitHub · GitHub | github.com [22:09:10] [02mediawiki] 07dependabot[bot] opened pull request 03#8135: Bump extensions/QuizGame from `00b2fea` to `7bd6e41` - 13https://github.com/miraheze/mediawiki/pull/8135 [22:09:10] ... [22:09:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/QuizGame-7bd6e41 - 13https://github.com/miraheze/mediawiki [22:09:12] ... [22:09:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8127: Bump extensions/Commentbox from `e49d615` to `6eb89d0` - 13https://github.com/miraheze/mediawiki/pull/8127 [22:09:13] ... [22:09:15] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SocialProfile-acfd842 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8bcba7fcb682 [22:09:16] [02miraheze/mediawiki] 07dependabot[bot] 038bcba7f - Bump extensions/SocialProfile from `f5cab34` to `acfd842` [22:09:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#8128: Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` - 13https://github.com/miraheze/mediawiki/pull/8128 [22:09:18] [url] Page not found · GitHub · GitHub | github.com [22:09:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8136: Bump extensions/SocialProfile from `f5cab34` to `acfd842` - 13https://github.com/miraheze/mediawiki/pull/8136 [22:09:19] [url] Page not found · GitHub · GitHub | github.com [22:09:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-acfd842 - 13https://github.com/miraheze/mediawiki [22:09:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8128: Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` - 13https://github.com/miraheze/mediawiki/pull/8128 [22:09:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:09:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#8131: Bump extensions/PageSchemas from `1d16b69` to `fce4265` - 13https://github.com/miraheze/mediawiki/pull/8131 [22:09:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-26ea437 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5608f07ad3ce [22:09:27] [02miraheze/mediawiki] 07dependabot[bot] 035608f07 - Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` [22:09:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130 [22:09:30] [02mediawiki] 07dependabot[bot] opened pull request 03#8137: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` - 13https://github.com/miraheze/mediawiki/pull/8137 [22:09:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130 [22:09:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-26ea437 - 13https://github.com/miraheze/mediawiki [22:09:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130 [22:09:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Kartographer-729fa58 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/88145f54dfbb [22:09:37] [02miraheze/mediawiki] 07dependabot[bot] 0388145f5 - Bump extensions/Kartographer from `c17dffd` to `729fa58` [22:09:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Kartographer-729fa58 - 13https://github.com/miraheze/mediawiki [22:09:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8138: Bump extensions/Kartographer from `c17dffd` to `729fa58` - 13https://github.com/miraheze/mediawiki/pull/8138 [22:09:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Math-c574876 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2ad9023fb046 [22:09:43] [02miraheze/mediawiki] 07dependabot[bot] 032ad9023 - Bump extensions/Math from `62df6ae` to `c574876` [22:09:45] [02mediawiki] 07dependabot[bot] opened pull request 03#8139: Bump extensions/Math from `62df6ae` to `c574876` - 13https://github.com/miraheze/mediawiki/pull/8139 [22:09:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Math-c574876 - 13https://github.com/miraheze/mediawiki [22:09:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8129: Bump extensions/CheckUser from `bf74001` to `fe9b1a2` - 13https://github.com/miraheze/mediawiki/pull/8129 [22:09:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/wikihiero-6627df6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6903ed04ef9c [22:09:51] [02miraheze/mediawiki] 07dependabot[bot] 036903ed0 - Bump extensions/wikihiero from `97baaed` to `6627df6` [22:09:52] [02mediawiki] 07dependabot[bot] opened pull request 03#8140: Bump extensions/wikihiero from `97baaed` to `6627df6` - 13https://github.com/miraheze/mediawiki/pull/8140 [22:09:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/wikihiero-6627df6 - 13https://github.com/miraheze/mediawiki [22:09:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ImportDump-e80a736 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d9c667ba9630 [22:09:57] [02miraheze/mediawiki] 07dependabot[bot] 03d9c667b - Bump extensions/ImportDump from `1d4b14b` to `e80a736` [22:09:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8132: Bump extensions/SimpleBlogPage from `a2825df` to `c61c3cb` - 13https://github.com/miraheze/mediawiki/pull/8132 [22:10:00] [02mediawiki] 07dependabot[bot] opened pull request 03#8141: Bump extensions/ImportDump from `1d4b14b` to `e80a736` - 13https://github.com/miraheze/mediawiki/pull/8141 [22:10:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ImportDump-e80a736 - 13https://github.com/miraheze/mediawiki [22:10:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#8134: Bump extensions/CategoryTree from `ed9789d` to `56d02d3` - 13https://github.com/miraheze/mediawiki/pull/8134 [22:10:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Monaco-74cb6bc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0a3a2a23cbc9 [22:10:06] [02miraheze/mediawiki] 07dependabot[bot] 030a3a2a2 - Bump skins/Monaco from `9e202c8` to `74cb6bc` [22:10:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8142: Bump skins/Monaco from `9e202c8` to `74cb6bc` - 13https://github.com/miraheze/mediawiki/pull/8142 [22:10:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Monaco-74cb6bc - 13https://github.com/miraheze/mediawiki [22:10:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133 [22:10:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133 [22:10:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-57d3e98 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a6ddc5488ebd [22:10:15] [02miraheze/mediawiki] 07dependabot[bot] 03a6ddc54 - Bump skins/Cosmos from `48e1b8a` to `57d3e98` [22:10:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.73, 1.76 [22:10:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8143: Bump skins/Cosmos from `48e1b8a` to `57d3e98` - 13https://github.com/miraheze/mediawiki/pull/8143 [22:10:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133 [22:10:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-57d3e98 - 13https://github.com/miraheze/mediawiki [22:10:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Moderation-ab2ee04 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0bafaabb9b7c [22:10:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.65, 1.40 [22:10:22] [02miraheze/mediawiki] 07dependabot[bot] 030bafaab - Bump extensions/Moderation from `5ba98b6` to `ab2ee04` [22:10:24] [02mediawiki] 07dependabot[bot] opened pull request 03#8144: Bump extensions/Moderation from `5ba98b6` to `ab2ee04` - 13https://github.com/miraheze/mediawiki/pull/8144 [22:10:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8135: Bump extensions/QuizGame from `00b2fea` to `7bd6e41` - 13https://github.com/miraheze/mediawiki/pull/8135 [22:10:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Moderation-ab2ee04 - 13https://github.com/miraheze/mediawiki [22:10:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8138: Bump extensions/Kartographer from `c17dffd` to `729fa58` - 13https://github.com/miraheze/mediawiki/pull/8138 [22:10:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AJAXPoll-830dc94 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9b3cb04a43c4 [22:10:31] [02miraheze/mediawiki] 07dependabot[bot] 039b3cb04 - Bump extensions/AJAXPoll from `2d109af` to `830dc94` [22:10:33] [02mediawiki] 07dependabot[bot] opened pull request 03#8145: Bump extensions/AJAXPoll from `2d109af` to `830dc94` - 13https://github.com/miraheze/mediawiki/pull/8145 [22:10:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AJAXPoll-830dc94 - 13https://github.com/miraheze/mediawiki [22:10:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8136: Bump extensions/SocialProfile from `f5cab34` to `acfd842` - 13https://github.com/miraheze/mediawiki/pull/8136 [22:10:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-97c2f74 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/efd377578b4b [22:10:39] [02miraheze/mediawiki] 07dependabot[bot] 03efd3775 - Bump extensions/Translate from `d7618b5` to `97c2f74` [22:10:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146 [22:10:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-97c2f74 - 13https://github.com/miraheze/mediawiki [22:10:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8139: Bump extensions/Math from `62df6ae` to `c574876` - 13https://github.com/miraheze/mediawiki/pull/8139 [22:10:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#7983: Bump extensions/Translate from `d7618b5` to `1554c3f` - 13https://github.com/miraheze/mediawiki/pull/7983#issuecomment-1184942137 [22:10:45] [url] Bump extensions/Translate from `d7618b5` to `1554c3f` by dependabot[bot] · Pull Request #7983 · miraheze/mediawiki · GitHub | github.com [22:10:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8137: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` - 13https://github.com/miraheze/mediawiki/pull/8137 [22:10:47] [url] Page not found · GitHub · GitHub | github.com [22:10:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8141: Bump extensions/ImportDump from `1d4b14b` to `e80a736` - 13https://github.com/miraheze/mediawiki/pull/8141 [22:10:48] [url] Page not found · GitHub · GitHub | github.com [22:10:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8141: Bump extensions/ImportDump from `1d4b14b` to `e80a736` - 13https://github.com/miraheze/mediawiki/pull/8141 [22:10:49] [url] Page not found · GitHub · GitHub | github.com [22:10:51] [02mediawiki] 07dependabot[bot] closed pull request 03#7983: Bump extensions/Translate from `d7618b5` to `1554c3f` - 13https://github.com/miraheze/mediawiki/pull/7983 [22:10:51] ... [22:10:52] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-1554c3f [22:10:54] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-1554c3f - 13https://github.com/miraheze/mediawiki [22:10:54] ... [22:10:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MultimediaViewer-ba72a39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2989f239827f [22:10:57] [02miraheze/mediawiki] 07dependabot[bot] 032989f23 - Bump extensions/MultimediaViewer from `59b641d` to `ba72a39` [22:10:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8147: Bump extensions/MultimediaViewer from `59b641d` to `ba72a39` - 13https://github.com/miraheze/mediawiki/pull/8147 [22:10:59] ... [22:11:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MultimediaViewer-ba72a39 - 13https://github.com/miraheze/mediawiki [22:11:00] [url] Page not found · GitHub · GitHub | github.com [22:11:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8140: Bump extensions/wikihiero from `97baaed` to `6627df6` - 13https://github.com/miraheze/mediawiki/pull/8140 [22:11:02] [url] Page not found · GitHub · GitHub | github.com [22:11:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Evelution-f08f4e4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0b3f8e99f20a [22:11:04] [02miraheze/mediawiki] 07dependabot[bot] 030b3f8e9 - Bump skins/Evelution from `631cf5e` to `f08f4e4` [22:11:06] [02mediawiki] 07dependabot[bot] opened pull request 03#8148: Bump skins/Evelution from `631cf5e` to `f08f4e4` - 13https://github.com/miraheze/mediawiki/pull/8148 [22:11:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Evelution-f08f4e4 - 13https://github.com/miraheze/mediawiki [22:11:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#8048: Bump skins/Evelution from `631cf5e` to `57584c2` - 13https://github.com/miraheze/mediawiki/pull/8048#issuecomment-1184942222 [22:11:09] [url] Bump skins/Evelution from `631cf5e` to `57584c2` by dependabot[bot] · Pull Request #8048 · miraheze/mediawiki · GitHub | github.com [22:11:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AdminLinks-025c33f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6651b741aca2 [22:11:12] [02miraheze/mediawiki] 07dependabot[bot] 036651b74 - Bump extensions/AdminLinks from `5e98fda` to `025c33f` [22:11:13] [02mediawiki] 07dependabot[bot] closed pull request 03#8048: Bump skins/Evelution from `631cf5e` to `57584c2` - 13https://github.com/miraheze/mediawiki/pull/8048 [22:11:14] [url] Page not found · GitHub · GitHub | github.com [22:11:15] [02mediawiki] 07dependabot[bot] opened pull request 03#8149: Bump extensions/AdminLinks from `5e98fda` to `025c33f` - 13https://github.com/miraheze/mediawiki/pull/8149 [22:11:15] [url] Page not found · GitHub · GitHub | github.com [22:11:16] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-57584c2 [22:11:17] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.477 second response time [22:11:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AdminLinks-025c33f - 13https://github.com/miraheze/mediawiki [22:11:18] [url] Page not found · GitHub · GitHub | github.com [22:11:19] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-57584c2 - 13https://github.com/miraheze/mediawiki [22:11:19] ... [22:11:20] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.160 second response time [22:11:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Timeless-081976a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a77b7af74c9e [22:11:22] [02miraheze/mediawiki] 07dependabot[bot] 03a77b7af - Bump skins/Timeless from `1556d82` to `081976a` [22:11:24] [02mediawiki] 07dependabot[bot] opened pull request 03#8150: Bump skins/Timeless from `1556d82` to `081976a` - 13https://github.com/miraheze/mediawiki/pull/8150 [22:11:24] ... [22:11:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Timeless-081976a - 13https://github.com/miraheze/mediawiki [22:11:25] ... [22:11:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#8142: Bump skins/Monaco from `9e202c8` to `74cb6bc` - 13https://github.com/miraheze/mediawiki/pull/8142 [22:11:27] [url] Page not found · GitHub · GitHub | github.com [22:11:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8143: Bump skins/Cosmos from `48e1b8a` to `57d3e98` - 13https://github.com/miraheze/mediawiki/pull/8143 [22:11:29] [url] Page not found · GitHub · GitHub | github.com [22:11:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#8143: Bump skins/Cosmos from `48e1b8a` to `57d3e98` - 13https://github.com/miraheze/mediawiki/pull/8143 [22:11:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8143: Bump skins/Cosmos from `48e1b8a` to `57d3e98` - 13https://github.com/miraheze/mediawiki/pull/8143 [22:11:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MsCatSelect-e5734f8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d80748394e64 [22:11:34] [02miraheze/mediawiki] 07dependabot[bot] 03d807483 - Bump extensions/MsCatSelect from `9a47a08` to `e5734f8` [22:11:36] [02mediawiki] 07dependabot[bot] opened pull request 03#8151: Bump extensions/MsCatSelect from `9a47a08` to `e5734f8` - 13https://github.com/miraheze/mediawiki/pull/8151 [22:11:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MsCatSelect-e5734f8 - 13https://github.com/miraheze/mediawiki [22:11:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/VisualEditor-ea1c59b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/afbc45308b3e [22:11:40] [02miraheze/mediawiki] 07dependabot[bot] 03afbc453 - Bump extensions/VisualEditor from `942e773` to `ea1c59b` [22:11:42] [02mediawiki] 07dependabot[bot] opened pull request 03#8152: Bump extensions/VisualEditor from `942e773` to `ea1c59b` - 13https://github.com/miraheze/mediawiki/pull/8152 [22:11:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/VisualEditor-ea1c59b - 13https://github.com/miraheze/mediawiki [22:11:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8144: Bump extensions/Moderation from `5ba98b6` to `ab2ee04` - 13https://github.com/miraheze/mediawiki/pull/8144 [22:11:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8145: Bump extensions/AJAXPoll from `2d109af` to `830dc94` - 13https://github.com/miraheze/mediawiki/pull/8145 [22:11:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ProofreadPage-76a7630 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c929ca17f0c9 [22:11:49] [02miraheze/mediawiki] 07dependabot[bot] 03c929ca1 - Bump extensions/ProofreadPage from `1063c18` to `76a7630` [22:11:51] [02mediawiki] 07dependabot[bot] opened pull request 03#8153: Bump extensions/ProofreadPage from `1063c18` to `76a7630` - 13https://github.com/miraheze/mediawiki/pull/8153 [22:11:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ProofreadPage-76a7630 - 13https://github.com/miraheze/mediawiki [22:11:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UserMerge-0ea0ee6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2b2b71da7a53 [22:11:55] [02miraheze/mediawiki] 07dependabot[bot] 032b2b71d - Bump extensions/UserMerge from `87c2df3` to `0ea0ee6` [22:11:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8154: Bump extensions/UserMerge from `87c2df3` to `0ea0ee6` - 13https://github.com/miraheze/mediawiki/pull/8154 [22:11:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UserMerge-0ea0ee6 - 13https://github.com/miraheze/mediawiki [22:12:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/RevisionSlider-94b62eb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7d18e887df60 [22:12:01] [02miraheze/mediawiki] 07dependabot[bot] 037d18e88 - Bump extensions/RevisionSlider from `3d74a38` to `94b62eb` [22:12:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/RevisionSlider-94b62eb - 13https://github.com/miraheze/mediawiki [22:12:04] [02mediawiki] 07dependabot[bot] opened pull request 03#8155: Bump extensions/RevisionSlider from `3d74a38` to `94b62eb` - 13https://github.com/miraheze/mediawiki/pull/8155 [22:12:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#8147: Bump extensions/MultimediaViewer from `59b641d` to `ba72a39` - 13https://github.com/miraheze/mediawiki/pull/8147 [22:12:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8148: Bump skins/Evelution from `631cf5e` to `f08f4e4` - 13https://github.com/miraheze/mediawiki/pull/8148 [22:12:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Citizen-7b2a046 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a8379db852ba [22:12:10] [02miraheze/mediawiki] 07dependabot[bot] 03a8379db - Bump skins/Citizen from `e53afa5` to `7b2a046` [22:12:12] [02mediawiki] 07dependabot[bot] opened pull request 03#8156: Bump skins/Citizen from `e53afa5` to `7b2a046` - 13https://github.com/miraheze/mediawiki/pull/8156 [22:12:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Citizen-7b2a046 - 13https://github.com/miraheze/mediawiki [22:12:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8150: Bump skins/Timeless from `1556d82` to `081976a` - 13https://github.com/miraheze/mediawiki/pull/8150 [22:12:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Scribunto-ecff608 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/28ab70f571d0 [22:12:18] [02miraheze/mediawiki] 07dependabot[bot] 0328ab70f - Bump extensions/Scribunto from `9cd5941` to `ecff608` [22:12:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8157: Bump extensions/Scribunto from `9cd5941` to `ecff608` - 13https://github.com/miraheze/mediawiki/pull/8157 [22:12:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Scribunto-ecff608 - 13https://github.com/miraheze/mediawiki [22:12:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.41, 1.33 [22:12:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146 [22:12:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146 [22:12:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146 [22:12:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Wikibase-318cb70 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7e7a52ebe87e [22:12:28] [02miraheze/mediawiki] 07dependabot[bot] 037e7a52e - Bump extensions/Wikibase from `073030c` to `318cb70` [22:12:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#8149: Bump extensions/AdminLinks from `5e98fda` to `025c33f` - 13https://github.com/miraheze/mediawiki/pull/8149 [22:12:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.14, 11.22, 10.65 [22:12:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8158: Bump extensions/Wikibase from `073030c` to `318cb70` - 13https://github.com/miraheze/mediawiki/pull/8158 [22:12:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Wikibase-318cb70 - 13https://github.com/miraheze/mediawiki [22:12:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8151: Bump extensions/MsCatSelect from `9a47a08` to `e5734f8` - 13https://github.com/miraheze/mediawiki/pull/8151 [22:12:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8153: Bump extensions/ProofreadPage from `1063c18` to `76a7630` - 13https://github.com/miraheze/mediawiki/pull/8153 [22:12:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8154: Bump extensions/UserMerge from `87c2df3` to `0ea0ee6` - 13https://github.com/miraheze/mediawiki/pull/8154 [22:12:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8156: Bump skins/Citizen from `e53afa5` to `7b2a046` - 13https://github.com/miraheze/mediawiki/pull/8156 [22:12:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8154: Bump extensions/UserMerge from `87c2df3` to `0ea0ee6` - 13https://github.com/miraheze/mediawiki/pull/8154 [22:12:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8155: Bump extensions/RevisionSlider from `3d74a38` to `94b62eb` - 13https://github.com/miraheze/mediawiki/pull/8155 [22:12:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8152: Bump extensions/VisualEditor from `942e773` to `ea1c59b` - 13https://github.com/miraheze/mediawiki/pull/8152 [22:12:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8157: Bump extensions/Scribunto from `9cd5941` to `ecff608` - 13https://github.com/miraheze/mediawiki/pull/8157 [22:12:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8158: Bump extensions/Wikibase from `073030c` to `318cb70` - 13https://github.com/miraheze/mediawiki/pull/8158 [22:20:13] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 9.44, 10.06 [22:22:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.88, 1.52, 1.36 [22:24:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.97, 1.97 [22:24:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.77, 1.19, 1.25 [22:24:41] [02puppet] 07MacFan4000 synchronize pull request 03#2699: ssl: extend revoke option to also remove key from git - 13https://github.com/miraheze/puppet/pull/2699 [22:24:41] [url] Page not found · GitHub · GitHub | github.com [22:32:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:34:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.27, 2.05, 1.97 [22:36:06] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.60, 2.04, 1.55 [22:36:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 10.14, 9.95 [22:38:00] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.62, 1.45 [22:38:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.98, 9.76, 9.82 [22:51:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.62, 1.84, 1.64 [22:52:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [22:55:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.17, 1.56, 1.57 [22:56:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.60, 1.33 [22:58:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.49, 1.32 [23:03:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.77, 9.55, 8.57 [23:05:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.66, 8.97, 8.46 [23:07:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.69, 10.10 [23:09:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.46, 9.77, 9.82 [23:20:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.16, 2.05, 1.57 [23:22:23] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.60, 1.46 [23:30:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.71, 2.00 [23:37:07] PROBLEM - graylog121 NTP time on graylog121 is WARNING: NTP WARNING: Offset 0.1039990485 secs [23:38:57] [02puppet] 07Universal-Omega opened pull request 03#2700: matomo: use debian package for composer - 13https://github.com/miraheze/puppet/pull/2700 [23:38:58] [url] Page not found · GitHub · GitHub | github.com [23:39:02] [02puppet] 07Universal-Omega opened pull request 03#2701: reports: use debian package for composer - 13https://github.com/miraheze/puppet/pull/2701 [23:43:08] RECOVERY - graylog121 NTP time on graylog121 is OK: NTP OK: Offset 0.08006441593 secs [23:44:15] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.46, 1.68 [23:50:15] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.99, 2.12, 1.87 [23:56:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.96, 1.90 [23:58:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.86, 2.04, 1.65