[00:03:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.84, 1.72, 1.40 [00:05:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.61, 1.40 [00:05:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.72, 10.57, 10.03 [00:06:45] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:09:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.10, 9.98, 9.95 [00:17:03] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:20:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 10.28, 10.10 [00:22:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.90, 9.93, 9.98 [00:22:46] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=taimapediawiki --no-updates --report=1 /mnt/mediawiki-static/metawiki/ImportDump/taimapediawiki-20220622030515.xml (END - exit=0) [00:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:27] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=taimapediawiki (START) [00:23:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:06] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.74, 1.52 [00:28:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.81, 1.51, 1.47 [00:32:03] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:35:45] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=taimapediawiki (END - exit=0) [00:35:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:35] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.90, 1.56 [00:50:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.32, 1.60, 1.48 [00:56:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 1.66, 1.44 [01:02:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.84, 1.58, 1.41 [01:04:14] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.43, 1.54, 1.42 [01:06:19] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.00, 1.91, 1.81 [01:08:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.61, 1.46 [01:10:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.39, 1.40 [01:12:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.30, 2.04, 1.87 [01:25:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.88, 2.00 [01:28:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 9.68, 9.21 [01:30:22] PROBLEM - id.altilunium.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'id.altilunium.xyz' expires in 15 day(s) (Mon 11 Jul 2022 01:10:38 GMT +0000). [01:32:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 8.91, 9.03 [01:33:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9312baaafc78...e1d1ee90da27 [01:33:43] [url] Comparing 9312baaafc78...e1d1ee90da27 · miraheze/ssl · GitHub | github.com [01:33:44] [02miraheze/ssl] 07MirahezeSSLBot 03e1d1ee9 - Bot: Update SSL cert for id.altilunium.xyz [01:35:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.38, 1.70 [01:38:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 10.81, 9.75 [01:39:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.02, 1.96, 1.85 [01:40:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.72, 10.66, 9.83 [01:43:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.40, 1.92, 1.88 [01:44:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.70, 9.78, 9.68 [01:51:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.35, 1.64 [02:00:14] RECOVERY - id.altilunium.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'id.altilunium.xyz' will expire on Fri 23 Sep 2022 00:33:37 GMT +0000. [02:06:34] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 10.92, 9.79 [02:08:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.87, 10.75, 9.86 [02:12:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.75, 9.70, 9.67 [02:15:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.61, 2.70, 1.85 [02:19:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.10, 1.95, 1.74 [02:21:35] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.73, 1.56, 1.62 [02:55:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.68, 3.41, 2.14 [02:58:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.30, 3.96 [03:00:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.65, 4.25, 4.23 [03:01:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.79, 1.82 [03:05:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.82, 1.35, 1.64 [03:13:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.35, 3.20, 3.78 [03:15:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.27, 4.22, 4.09 [03:17:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.40, 3.51, 3.85 [03:19:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.83, 3.92 [03:22:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.69, 3.96 [03:23:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.97, 4.00 [03:26:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.26, 3.21, 3.82 [03:32:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 10.61, 9.83 [03:32:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.61, 2.76, 3.32 [03:32:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.23, 4.06, 3.98 [03:34:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.76, 3.87 [03:35:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.15, 12.00, 10.52 [03:37:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.45, 10.44, 10.14 [03:42:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.68, 3.73 [03:43:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.69, 9.41, 9.85 [03:44:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.22, 3.55 [03:45:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.53, 3.80, 3.46 [03:47:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.42, 3.35 [03:48:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.56, 3.60 [03:49:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.77, 3.49 [03:49:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.50, 3.51 [03:51:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.39, 3.39 [03:51:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.19, 3.38 [03:52:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.91, 3.75 [03:54:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.83, 3.89, 3.74 [03:57:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.50, 2.03, 1.47 [03:57:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.66, 3.51 [03:59:26] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.62, 1.58, 1.38 [04:01:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.75, 3.63 [04:04:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.22, 3.58, 3.87 [04:05:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 2.96, 3.31 [04:15:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.39, 3.15 [04:17:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.14, 3.08 [04:20:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.41, 4.87, 4.02 [04:22:48] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:20] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 7.18, 4.33, 2.57 [04:23:24] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:32] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:42] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:23:55] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:57] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.69, 3.34 [04:26:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.30, 4.13, 3.62 [04:29:10] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:29:12] PROBLEM - mwtask111 APT on mwtask111 is UNKNOWN: NRPE: Command timed out after 60 seconds [04:29:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.77, 3.45 [04:30:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.64, 3.55 [04:31:03] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15897 bytes in 0.031 second response time [04:31:06] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [04:31:13] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:31:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 3.23, 3.30 [04:31:37] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15897 bytes in 0.307 second response time [04:31:56] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15897 bytes in 0.320 second response time [04:32:02] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 27096 bytes in 0.582 second response time [04:32:14] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15883 bytes in 0.030 second response time [04:32:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.09, 3.35 [04:36:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.06, 2.64, 3.81 [04:46:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.46, 3.24 [04:47:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.43, 2.01, 1.48 [04:47:57] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.78, 1.63, 3.68 [04:49:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.60, 1.84, 1.48 [04:49:56] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.66, 1.30, 3.31 [04:53:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.98, 1.53, 1.45 [05:03:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.79, 3.96, 3.25 [05:04:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.38, 3.03 [05:04:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.28, 3.10 [05:05:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 3.69, 3.23 [05:06:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 2.98, 2.92 [05:06:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.04, 3.04 [05:07:10] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=malcolmversewiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/malcolmversewiki-20220624150533.xml (START) [05:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:20] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=malcolmversewiki --no-updates --username-prefix=w /mnt/mediawiki-static/metawiki/ImportDump/malcolmversewiki-20220624150533.xml (END - exit=0) [05:07:21] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=malcolmversewiki (START) [05:07:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:07:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [05:11:21] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=malcolmversewiki (END - exit=0) [05:11:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:13:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.70, 3.30 [05:15:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.81, 3.37 [05:15:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.96, 3.59 [05:17:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.43, 3.28 [05:17:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.34, 3.41 [05:19:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 3.15, 3.20 [05:19:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.82, 2.82, 3.21 [05:23:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.56, 3.36 [05:27:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.24, 3.30 [05:38:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:46:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.75, 3.31 [05:48:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.23, 3.78, 3.38 [05:48:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 4.18, 3.54 [05:50:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.59, 3.34 [05:50:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.45, 3.35 [05:52:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 3.03, 3.17 [05:52:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 3.07, 3.22 [05:55:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 3.79, 3.25 [05:57:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.51, 3.20 [05:59:25] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.72, 2.88, 3.01 [06:09:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.57, 3.23 [06:09:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.60, 3.78 [06:13:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.82, 3.40 [06:14:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.77, 3.39, 3.29 [06:15:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.56, 3.58 [06:19:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.79, 3.13, 3.39 [06:25:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.86, 4.08, 3.46 [06:25:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.90, 3.64 [06:26:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.33, 4.43, 3.54 [06:27:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.64, 3.37 [06:28:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.70, 3.38 [06:28:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.12, 3.21 [06:30:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.82, 3.45 [06:31:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.66, 3.56 [06:34:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.61, 3.43 [06:34:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.56, 3.47 [06:35:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.46, 3.75, 3.66 [06:36:24] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:36:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.27, 3.32 [06:38:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.07, 3.29 [06:41:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.24, 2.95, 3.34 [06:52:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 9.45, 8.50 [06:58:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.15, 9.49, 8.90 [07:11:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.38, 3.02 [07:12:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.80, 3.21 [07:13:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 2.95, 2.91 [07:14:23] [02mw-config] 07Universal-Omega closed pull request 03#4794: beta: Set actor migration to WRITE_NEW - 13https://github.com/miraheze/mw-config/pull/4794 [07:14:23] [url] Page not found · GitHub · GitHub | github.com [07:14:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bbf7f3f295f2...5cc74d58d87f [07:14:25] [url] Comparing bbf7f3f295f2...5cc74d58d87f · miraheze/mw-config · GitHub | github.com [07:14:26] [02miraheze/mw-config] 07Universal-Omega 035cc74d5 - beta: Set actor migration to WRITE_NEW (#4794) [07:14:27] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [07:14:27] [url] Page not found · GitHub · GitHub | github.com [07:14:29] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [07:14:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.50, 3.17 [07:14:48] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [07:14:51] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 3s [07:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:27] miraheze/mw-config - Universal-Omega the build passed. [07:18:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 2.96, 3.00 [07:22:29] !log [@mwtask111] starting deploy of {'config': True} to all [07:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:22:40] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 10s [07:22:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d897355d43e2 [07:25:44] [02miraheze/mw-config] 07Universal-Omega 03d897355 - Set actor migration to WRITE_TEMP_AND_NEW on production [07:25:46] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [07:25:46] [url] Page not found · GitHub · GitHub | github.com [07:25:47] [02mw-config] 07Universal-Omega opened pull request 03#4795: Set actor migration to WRITE_TEMP_AND_NEW on production - 13https://github.com/miraheze/mw-config/pull/4795 [07:25:48] [url] Page not found · GitHub · GitHub | github.com [07:27:02] miraheze/mw-config - Universal-Omega the build passed. [08:42:40] [02mw-config] 07Universal-Omega closed pull request 03#4795: Set actor migration to WRITE_TEMP_AND_NEW on production - 13https://github.com/miraheze/mw-config/pull/4795 [08:42:40] [url] Page not found · GitHub · GitHub | github.com [08:42:41] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [08:42:43] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [08:42:43] ... [08:42:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5cc74d58d87f...7a8d6d3b37fd [08:42:45] [url] Comparing 5cc74d58d87f...7a8d6d3b37fd · miraheze/mw-config · GitHub | github.com [08:42:46] [02miraheze/mw-config] 07Universal-Omega 037a8d6d3 - Set actor migration to WRITE_TEMP_AND_NEW on production (#4795) [08:43:38] miraheze/mw-config - Universal-Omega the build passed. [08:52:04] !log [@mwtask111] starting deploy of {'config': True} to all [08:52:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:52:11] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 7s [08:52:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:05] !log [@test101] starting deploy of {'config': True} to all [08:58:07] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [08:58:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:04:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.75, 3.93, 3.08 [09:06:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.48, 3.01 [09:08:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.21, 2.97 [09:32:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.19, 3.65, 3.06 [09:36:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.18, 3.01 [10:37:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:43:06] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [10:55:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.19, 1.76, 1.48 [10:59:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.12, 1.59, 1.48 [11:06:19] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:11:06] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:20:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk [11:29:54] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.44, 1.77, 1.54 [11:31:50] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.25, 1.58, 1.50 [11:49:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 4.00, 3.37, 2.87 [11:51:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.64, 3.02 [11:52:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.80, 3.27 [11:53:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.38, 2.99 [11:54:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.63, 3.26 [11:54:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 3.68, 3.14 [11:55:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.60, 3.05 [11:56:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.33, 3.20 [11:56:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.53, 3.15 [11:57:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.17, 2.96 [11:58:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.13, 3.05 [12:15:37] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk [12:19:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.10, 9.26 [12:21:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.95, 9.77, 9.23 [12:25:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 10.61, 9.65 [12:26:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.65, 1.36 [12:27:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 10.54, 9.74 [12:28:14] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.57, 1.61, 1.37 [12:28:33] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.99, 7.09, 6.23 [12:30:30] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.36, 7.43, 6.45 [12:31:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.77, 9.59, 9.58 [12:32:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.43, 7.58, 6.64 [12:38:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.37, 3.36, 2.87 [12:40:14] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 11.22, 7.64, 6.87 [12:40:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.05, 2.82 [12:41:53] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.89, 2.65, 1.89 [12:42:11] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.55, 7.34, 6.85 [12:45:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.24, 1.82, 1.71 [12:46:04] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.89, 7.41, 6.93 [12:47:42] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.53, 1.66, 1.66 [12:48:01] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.78, 6.66, 6.71 [12:53:46] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=ysmwikiwiki (END - exit=65280) [12:53:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:22] Reception123: error? [13:02:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.07, 3.63, 3.06 [13:04:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.15, 2.96 [13:05:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.40, 4.42, 3.41 [13:06:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.24, 3.41 [13:07:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 9.71, 9.20 [13:08:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.64, 3.29 [13:08:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.61, 3.23 [13:09:19] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.45, 8.72, 8.89 [13:12:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.16, 3.50 [13:14:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 3.80, 3.44 [13:16:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.26, 3.29 [13:16:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.83, 3.56 [13:19:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.93, 4.00 [13:19:58] RhinosF1: memory yeah [13:20:17] Reception123: add the parameter to make it max [13:20:34] The cap might need raising again [13:20:40] yeah will do [13:20:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.66, 3.52 [13:20:51] was away for a few mins (and will be again in ~25) [13:21:04] Ok [13:21:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.16, 3.99, 4.03 [13:22:38] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=ysmwikiwiki --memory-limit=160000000 (END - exit=0) [13:22:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.83, 3.81, 3.45 [13:22:45] meh it's empty now [13:22:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:28] Reception123: they cleared at :59 [13:24:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.59, 3.42 [13:24:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.51, 3.67, 3.59 [13:25:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.76, 3.95 [13:26:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.75, 3.62 [13:28:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.07, 3.40, 3.39 [13:28:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.17, 3.27, 3.46 [13:30:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.95, 3.69 [13:31:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.36, 4.35, 4.14 [13:32:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.83, 3.60 [13:32:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.65, 3.60 [13:34:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 4.25, 3.79 [13:34:44] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:36:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.66, 3.65, 3.62 [13:37:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.86, 4.00 [13:38:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.53, 3.62, 3.57 [13:40:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.72, 2.88, 3.33 [13:40:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.69, 2.90, 3.31 [13:43:32] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.84, 3.18, 1.84 [13:43:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.65, 3.80, 3.81 [13:45:15] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.84, 1.95, 1.42 [13:45:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.88, 3.79, 3.80 [13:46:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.54, 3.50 [13:47:15] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.73, 1.54, 1.34 [13:49:17] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.94, 1.85, 1.68 [13:49:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.41, 4.08, 3.90 [13:51:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.11, 1.59, 1.60 [13:51:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 10.40, 9.73 [13:51:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.73, 3.78 [13:51:41] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d1b2993f5307...c9521a13f56b [13:51:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.90, 3.56 [13:51:42] [url] Comparing d1b2993f5307...c9521a13f56b · miraheze/puppet · GitHub | github.com [13:51:43] [02miraheze/puppet] 07Universal-Omega 03c9521a1 - Increase roundcube session lifetime to 24 hours (#2618) [13:51:44] [02puppet] 07JohnFLewis closed pull request 03#2618: Increase roundcube session lifetime to 24 hours - 13https://github.com/miraheze/puppet/pull/2618 [13:51:45] [url] Page not found · GitHub · GitHub | github.com [13:53:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.90, 4.07, 3.89 [13:55:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.75, 3.81 [13:58:21] !log increase matomo101 ram by 1gb - https://phabricator.miraheze.org/T9430 [13:58:22] [url] ⚓ T9430 [Existing] Server Resource Request for matomo101 | phabricator.miraheze.org [13:58:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:41] !log reboot matomo101 [13:58:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:59:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.73, 3.70 [13:59:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.04, 3.89 [14:00:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 2.98, 3.34 [14:00:43] [02puppet] 07JohnFLewis reviewed pull request 03#2436 commit - 13https://github.com/miraheze/puppet/pull/2436#discussion_r906687695 [14:00:44] [url] db: don't allow access to testglobal from non test servers by RhinosF1 · Pull Request #2436 · miraheze/puppet · GitHub | github.com [14:00:45] [02puppet] 07JohnFLewis reviewed pull request 03#2436 commit - 13https://github.com/miraheze/puppet/pull/2436#discussion_r906687762 [14:00:45] [url] db: don't allow access to testglobal from non test servers by RhinosF1 · Pull Request #2436 · miraheze/puppet · GitHub | github.com [14:01:13] [02puppet] 07JohnFLewis assigned pull request 03#2456: mwdeploy: async command running - 13https://github.com/miraheze/puppet/pull/2456 [14:01:13] [url] Page not found · GitHub · GitHub | github.com [14:01:39] [02puppet] 07JohnFLewis commented on pull request 03#2556: varnish: force thumb_handler.php to mwtask111 - 13https://github.com/miraheze/puppet/pull/2556#issuecomment-1166293742 [14:01:39] [url] varnish: force thumb_handler.php to mwtask111 by Universal-Omega · Pull Request #2556 · miraheze/puppet · GitHub | github.com [14:01:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.54, 3.73 [14:02:33] [02puppet] 07JohnFLewis commented on pull request 03#2597: add WikiMiniAtlas to frame-src - 13https://github.com/miraheze/puppet/pull/2597#issuecomment-1166293946 [14:02:33] [url] add WikiMiniAtlas to frame-src by ugochimobi · Pull Request #2597 · miraheze/puppet · GitHub | github.com [14:02:45] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [14:03:21] [02puppet] 07JohnFLewis closed pull request 03#2630: Cleanup check_reverse_dns - 13https://github.com/miraheze/puppet/pull/2630 [14:03:22] [url] Page not found · GitHub · GitHub | github.com [14:03:23] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9521a13f56b...9cdd03a96b84 [14:03:23] [url] Comparing c9521a13f56b...9cdd03a96b84 · miraheze/puppet · GitHub | github.com [14:03:24] [02miraheze/puppet] 07Universal-Omega 039cdd03a - Cleanup check_reverse_dns (#2630) [14:04:16] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9cdd03a96b84...a638ba6709d3 [14:04:17] [url] Comparing 9cdd03a96b84...a638ba6709d3 · miraheze/puppet · GitHub | github.com [14:04:18] [02miraheze/puppet] 07Universal-Omega 03a638ba6 - varnish: always use case-insensitive redirects for `path_redirects` (#2641) [14:04:19] [02puppet] 07JohnFLewis closed pull request 03#2641: varnish: always use case-insensitive redirects for `path_redirects` - 13https://github.com/miraheze/puppet/pull/2641 [14:04:20] [url] Page not found · GitHub · GitHub | github.com [14:04:21] [02miraheze/ssl] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1d1ee90da27...f8529470bd8f [14:04:21] [url] Comparing e1d1ee90da27...f8529470bd8f · miraheze/ssl · GitHub | github.com [14:04:22] [02miraheze/ssl] 07Universal-Omega 03f852947 - Move miraheze.wiki to the top of redirects.yaml (#541) [14:04:24] [02ssl] 07JohnFLewis closed pull request 03#541: Move miraheze.wiki to the top of redirects.yaml - 13https://github.com/miraheze/ssl/pull/541 [14:04:24] [url] Page not found · GitHub · GitHub | github.com [14:04:54] [02puppet] 07JohnFLewis commented on pull request 03#2644: archive puppet-users, create letsencrypt-admins group - 13https://github.com/miraheze/puppet/pull/2644#issuecomment-1166294469 [14:04:55] [url] archive puppet-users, create letsencrypt-admins group by Reception123 · Pull Request #2644 · miraheze/puppet · GitHub | github.com [14:05:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.44, 11.60, 10.71 [14:05:03] [02puppet] 07JohnFLewis assigned pull request 03#2644: archive puppet-users, create letsencrypt-admins group - 13https://github.com/miraheze/puppet/pull/2644 [14:05:04] [url] Page not found · GitHub · GitHub | github.com [14:05:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.95, 2.79, 3.39 [14:06:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 11.12, 10.64 [14:07:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.09, 2.28, 1.52 [14:07:17] [02puppet] 07JohnFLewis commented on pull request 03#2657: Merge letsencrypt and ssl modules; create ssl-admins group - 13https://github.com/miraheze/puppet/pull/2657#issuecomment-1166294981 [14:07:18] [url] Merge letsencrypt and ssl modules; create ssl-admins group by Universal-Omega · Pull Request #2657 · miraheze/puppet · GitHub | github.com [14:08:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/57260f8d6c2b [14:08:52] [02miraheze/puppet] 07paladox 0357260f8 - mariadb: send mysql-error.log to graylog [14:08:53] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [14:08:54] [url] Page not found · GitHub · GitHub | github.com [14:08:55] [02puppet] 07paladox opened pull request 03#2685: mariadb: send mysql-error.log to graylog - 13https://github.com/miraheze/puppet/pull/2685 [14:08:55] [url] Page not found · GitHub · GitHub | github.com [14:09:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.08, 1.77, 1.42 [14:09:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.91, 4.34, 3.72 [14:09:34] JohnLewis: do https://github.com/miraheze/puppet/pull/2682 too pls [14:09:35] [url] mwdeploy: backport a fix by RhinosF1 · Pull Request #2682 · miraheze/puppet · GitHub | github.com [14:09:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.38, 3.55 [14:10:15] [02puppet] 07paladox closed pull request 03#2670: matomo: set bulk_requests_use_transaction to 0 - 13https://github.com/miraheze/puppet/pull/2670 [14:10:15] [url] Page not found · GitHub · GitHub | github.com [14:10:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [14:10:19] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://github.com/miraheze/puppet [14:10:19] ... [14:10:48] [02puppet] 07JohnFLewis closed pull request 03#2682: mwdeploy: backport a fix - 13https://github.com/miraheze/puppet/pull/2682 [14:10:49] [url] Page not found · GitHub · GitHub | github.com [14:10:50] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/a638ba6709d3...ad5b8ca75f17 [14:10:50] [url] Comparing a638ba6709d3...ad5b8ca75f17 · miraheze/puppet · GitHub | github.com [14:10:51] [02miraheze/puppet] 07RhinosF1 03ad5b8ca - mwdeploy: backport a fix (#2682) [14:11:02] Thanks [14:11:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.53, 1.32, 1.29 [14:11:40] JohnLewis: https://github.com/miraheze/puppet/pull/2668/files too [14:11:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 3.67, 3.62 [14:11:40] [url] deploy: folders should use paths not files by RhinosF1 · Pull Request #2668 · miraheze/puppet · GitHub | github.com [14:12:03]