[00:05:20] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.55, 2.55, 1.63 [00:06:47] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.62, 10.08, 8.19 [00:07:41] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.85, 9.91, 7.91 [00:09:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.15, 2.40 [00:09:27] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.35, 2.17 [00:09:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.40, 9.25, 7.94 [00:10:41] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.72, 9.60, 8.44 [00:11:23] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.59, 2.92, 2.42 [00:11:23] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.45, 3.02, 2.18 [00:27:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.70, 3.48, 2.95 [00:29:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.63, 3.02, 2.84 [00:35:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 3.87, 3.25 [00:36:09] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.00, 8.75, 7.60 [00:36:46] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.36, 3.54, 2.88 [00:36:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:36:56] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [00:37:24] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.61, 3.53, 3.18 [00:37:46] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 3.39, 2.84 [00:38:06] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.31, 11.18, 8.68 [00:38:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:38:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.13, 11.01, 8.41 [00:38:52] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.462 second response time [00:39:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:39:10] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:39:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 4.06, 3.41 [00:39:38] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:39:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.93, 3.54, 2.97 [00:40:08] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.554 second response time [00:40:35] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw142 [00:41:04] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.599 second response time [00:41:04] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.529 second response time [00:41:05] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.542 second response time [00:41:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.29, 3.81, 3.41 [00:41:37] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.594 second response time [00:41:45] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 3.74, 3.10 [00:42:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.23, 11.92, 9.19 [00:42:35] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [00:42:40] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.64, 10.44, 8.86 [00:43:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.80, 3.09, 3.19 [00:43:45] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.57, 3.29, 3.02 [00:43:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.62, 11.42, 9.74 [00:44:25] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 11.81, 9.43 [00:44:35] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.73, 11.98, 9.56 [00:46:24] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.96, 11.00, 9.45 [00:46:31] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.35, 9.81, 9.03 [00:47:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.05, 3.78, 3.45 [00:47:52] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.47, 9.51, 9.49 [00:48:24] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.25, 9.08, 8.94 [00:49:23] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.87, 3.47, 3.38 [00:52:33] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.86, 3.37, 3.77 [00:53:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.25, 3.98, 3.57 [00:54:29] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.72, 3.99, 3.94 [00:55:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.68, 3.80, 3.55 [00:55:52] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.33, 9.11, 8.97 [00:56:24] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.11, 3.29, 3.70 [00:57:23] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.32, 3.20, 3.36 [00:57:51] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.18, 8.56, 8.77 [00:59:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:00:36] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:00:46] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [01:01:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.98, 4.51, 3.85 [01:01:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.687 second response time [01:02:12] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.47, 2.68, 3.30 [01:07:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:09:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.514 second response time [01:11:00] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.22, 10.75, 9.09 [01:12:57] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.97, 9.62, 8.88 [01:30:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:33:23] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.10, 3.01, 3.89 [01:43:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.65, 2.68, 3.35 [01:49:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 6.45, 5.86 [01:50:26] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:51:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.18, 3.60, 3.39 [01:52:24] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.603 second response time [01:53:28] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.09, 3.50, 3.40 [01:53:42] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.38, 6.30, 5.96 [01:55:25] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.03, 2.61, 3.08 [02:11:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.54, 7.68, 6.58 [02:13:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.78, 6.90, 6.42 [02:18:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.36, 7.29, 6.67 [02:22:48] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.37, 6.56, 6.52 [02:32:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.94, 6.97, 6.54 [02:34:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.24, 6.32 [02:55:08] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.54, 6.83, 6.47 [02:57:05] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.20, 6.23, 6.29 [03:00:03] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 1 seconds old and 0 bytes [03:00:23] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 22 seconds old and 93 bytes [03:25:52] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.82, 8.74, 6.89 [03:26:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 8.86, 6.69 [03:31:51] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.71, 13.51, 9.43 [03:32:37] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.61, 12.18, 8.78 [03:33:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.62, 11.72, 8.59 [03:34:51] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.47, 11.76, 9.34 [03:35:25] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.15, 11.61, 8.97 [03:36:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.10, 12.11, 9.74 [03:38:23] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.38, 11.21, 9.45 [03:38:51] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.72, 11.63, 9.83 [03:41:51] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.29, 11.87, 10.84 [03:44:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.14, 11.01, 10.00 [03:45:22] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.18, 9.89, 9.29 [03:47:59] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.63, 9.32, 9.44 [03:48:51] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.49, 11.01, 10.29 [03:49:51] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.36, 8.85, 9.85 [03:50:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:50:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.96, 10.02, 10.02 [03:51:55] PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediarobotica.org' expires in 15 day(s) (Sun 04 Feb 2024 03:32:10 AM GMT +0000). [03:52:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c5f07f892bc1...11fe148375c3 [03:52:13] [02miraheze/ssl] 07MirahezeSSLBot 0311fe148 - Bot: Update SSL cert for encyclopediarobotica.org [04:06:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:08:08] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.528 second response time [04:20:56] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Thu 18 Apr 2024 02:52:01 AM GMT +0000. [04:47:26] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:48:58] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.83, 12.05, 7.81 [04:49:24] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.608 second response time [04:50:56] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 4.46, 9.30, 7.33 [05:10:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:50:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.04, 7.13, 6.21 [05:52:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.60, 6.88, 6.24 [05:54:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.33, 6.52, 6.17 [06:12:26] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:13:01] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 1.665 second response time [06:14:24] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [06:15:01] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.818 second response time [06:59:56] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/96e139f137ca...a8a2f4a3b87f [06:59:58] [02miraheze/mw-config] 07Reception123 03a8a2f4a - add RequestSSL permissions [07:00:42] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:00:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:00] miraheze/mw-config - Reception123 the build passed. [07:01:36] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 53s [07:01:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:33] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:02:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:11] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:03:12] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 39s [07:03:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:57] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 46s [07:04:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:25] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php --wiki=metawiki (END - exit=2) [07:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php --wiki=metawiki /srv/mediawiki/1.40/extensions/RequestSSL/sql/mysql/tables-generated.sql (END - exit=0) [07:20:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:10] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (START) [07:56:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (END - exit=256) [07:56:12] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (START) [07:56:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:29] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (END - exit=0) [07:56:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=321nailswiki --active --update (END - exit=0) [07:56:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:07:26] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:07:38] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:09:24] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.579 second response time [08:09:38] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.333 second response time [08:10:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [08:10:21] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 23.29, 11.87, 7.34 [08:10:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.82, 3.82, 2.30 [08:12:08] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.513 second response time [08:12:19] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.70, 10.44, 7.40 [08:13:30] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:13:38] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:13:45] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:13:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:14:16] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:14:17] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 15.14, 9.41 [08:15:29] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.523 second response time [08:15:37] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.608 second response time [08:15:43] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.691 second response time [08:15:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.507 second response time [08:16:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.35, 3.80, 2.82 [08:16:15] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.510 second response time [08:18:13] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.61, 10.25, 8.71 [08:18:13] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.66, 3.06, 2.67 [08:19:44] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:19:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:20:11] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.31, 12.02, 9.47 [08:21:43] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.766 second response time [08:21:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.566 second response time [08:22:09] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.59, 9.23, 8.75 [08:24:10] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:50:18] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [08:50:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.73, 5.80 [08:52:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.95, 6.59, 5.61 [08:59:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.76, 3.88, 2.28 [09:01:35] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:04:26] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:06:24] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.761 second response time [09:06:29] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.39, 11.72, 8.14 [09:06:36] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures [09:06:52] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.73, 10.96, 8.03 [09:08:50] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.14, 9.63, 7.88 [09:10:23] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.52, 10.42, 8.49 [09:11:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.05, 3.37, 3.32 [09:12:20] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.13, 8.95, 8.18 [09:17:13] [02miraheze/RequestSSL] 07Reception123 pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/3c6dab4d1825...7533a705ee65 [09:17:15] [02miraheze/RequestSSL] 07Reception123 037533a70 - correct label name [09:17:36] [02miraheze/RequestSSL] 07Reception123 pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/7533a705ee65...28f0c58b2223 [09:17:38] [02miraheze/RequestSSL] 07Reception123 0328f0c58 - correct label name [09:22:10] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=thedollestatewiki (END - exit=0) [09:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:30] !log [reception@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [09:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:06] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=eleniswiki (END - exit=0) [09:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:41] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=snowballaspediawiki (END - exit=0) [09:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:57] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rushpediawiki (END - exit=0) [09:27:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:23] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=eleniswiki (END - exit=0) [09:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:28:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=rushpediawiki (END - exit=0) [09:28:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:31:30] !log [reception@mwtask181] starting deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all [09:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:32:13] !log [reception@mwtask181] finished deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 43s [09:32:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:37:15] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (START) [09:37:16] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (END - exit=256) [09:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:37:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:03:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.25, 6.19, 5.57 [10:04:59] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.04, 5.99, 5.57 [10:13:14] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (START) [10:13:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:14:58] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (END - exit=2) [10:15:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:47:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.05, 6.76, 5.97 [11:49:07] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.38, 6.16, 5.84 [12:27:10] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.71, 3.64, 3.00 [12:31:06] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.97, 3.26 [12:33:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.20, 3.70, 3.25 [12:34:27] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [12:35:07] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.71, 3.30 [12:37:51] PROBLEM - os141 SSH on os141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:58] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:38:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:39:50] RECOVERY - os141 SSH on os141 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [12:39:59] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.456 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [12:40:08] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.554 second response time [12:43:22] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.005590438843 secs [12:43:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:45:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [12:52:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [12:54:11] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.629 second response time [12:56:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.53, 2.98, 3.80 [13:00:54] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.76, 1.85, 3.14 [13:04:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:29:26] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:29:38] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:29:38] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:29:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [13:30:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.59, 3.24, 2.03 [13:31:24] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.571 second response time [13:31:37] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.595 second response time [13:31:37] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.871 second response time [13:31:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.512 second response time [13:32:21] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.71, 2.50, 1.90 [13:34:15] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:04:07] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [14:12:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.33, 6.90, 6.37 [14:14:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.39, 6.17, 6.18 [14:24:30] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.83, 9.77, 8.16 [14:24:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.16, 10.54, 8.43 [14:25:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.85, 11.32, 8.69 [14:26:27] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.63, 9.70, 8.34 [14:26:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.79, 9.59, 8.32 [14:27:06] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.03, 9.84, 8.44 [14:28:25] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:30:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.94, 6.63, 6.24 [14:32:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.25, 6.20, 6.13 [14:32:27] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:47:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.69, 6.91, 6.37 [14:49:15] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.88, 6.45, 6.26 [14:53:38] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [14:58:46] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#6: Cleanup - 13https://github.com/miraheze/RequestSSL/pull/6 [15:06:47] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.48, 6.61, 6.28 [15:08:20] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#6: Cleanup - 13https://github.com/miraheze/RequestSSL/pull/6 [15:08:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.68, 5.68, 5.97 [15:08:48] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:09:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:09:10] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:10:22] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#6: Cleanup - 13https://github.com/miraheze/RequestSSL/pull/6 [15:10:53] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.25, 11.21, 8.44 [15:10:54] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.782 second response time [15:11:04] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.738 second response time [15:11:09] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.269 second response time [15:11:11] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.38, 10.32, 7.25 [15:11:24] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:11:45] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.85, 12.56, 8.38 [15:11:46] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.11, 10.68, 7.46 [15:11:47] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 21.46, 13.47, 9.37 [15:11:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:11:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 19.47, 12.51, 9.46 [15:11:52] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 17.44, 13.01, 9.86 [15:12:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [15:13:00] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:13:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:13:13] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:13:41] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:14:28] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [15:14:35] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [15:14:40] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw141 mw142 mw133 mw143 [15:15:04] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw142 mw133 mw134 mw143 [15:15:07] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.511 second response time [15:15:09] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.817 second response time [15:16:16] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.568 second response time [15:16:27] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:16:28] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.394 second response time [15:17:03] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.573 second response time [15:17:13] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:17:23] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.397 second response time [15:17:24] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.499 second response time [15:17:51] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:19:31] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.768 second response time [15:20:01] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.581 second response time [15:20:38] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [15:21:38] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:23:10] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:23:14] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.842 second response time [15:24:32] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:24:48] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 9.959 second response time [15:25:07] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:25:24] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:26:08] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:26:31] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.650 second response time [15:27:09] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.397 second response time [15:27:22] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.850 second response time [15:28:04] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.497 second response time [15:28:09] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.814 second response time [15:28:22] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.927 second response time [15:28:23] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [15:28:35] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [15:28:46] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.395 second response time [15:28:54] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:30:53] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.174 second response time [15:30:55] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.423 second response time [15:31:18] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.973 second response time [15:31:28] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:32:15] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:32:16] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw134 [15:32:30] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:32:35] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw133 mw134 [15:34:10] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:34:14] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [15:34:16] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.289 second response time [15:34:33] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 4.639 second response time [15:35:17] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:36:14] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 6.016 second response time [15:37:04] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [15:37:14] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:37:18] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.212 second response time [15:37:41] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.026 second response time [15:38:00] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.85, 3.31, 2.71 [15:38:06] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw133 mw134 [15:38:18] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:38:22] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:38:38] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.459 second response time [15:39:00] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.229 second response time [15:39:14] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.509 second response time [15:39:22] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:39:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:39:59] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.39, 2.96, 2.66 [15:40:21] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.657 second response time [15:40:30] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [15:41:04] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw134 mw143 [15:41:05] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.583 second response time [15:41:19] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:41:24] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.921 second response time [15:41:53] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:42:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.484 second response time [15:42:30] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.211 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [15:42:53] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 8.193 second response time [15:43:20] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.174 second response time [15:43:37] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [15:43:50] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.745 second response time [15:43:59] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [15:44:35] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [15:45:04] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [15:46:43] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 3.68, 9.32, 11.82 [15:48:08] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.50, 6.89, 6.49 [15:48:51] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.53, 9.23, 11.97 [15:49:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:49:45] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 4.33, 7.64, 11.30 [15:49:46] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 3.95, 7.23, 11.24 [15:49:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.50, 8.73, 11.47 [15:50:40] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 2.94, 5.89, 9.85 [15:52:00] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.59, 6.71, 6.56 [15:53:31] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 5.79, 7.82, 11.51 [15:53:45] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.39, 6.22, 9.83 [15:53:46] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.13, 5.56, 9.64 [15:53:51] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.63, 8.25, 11.31 [15:55:50] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.60, 7.91, 10.20 [15:55:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.96, 6.84, 6.68 [15:57:51] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.44, 6.27, 6.49 [15:58:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 5.71, 8.10, 10.15 [16:01:28] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.59, 7.85, 10.07 [16:05:51] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 9.78, 10.41 [16:07:51] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.15, 9.00, 10.05 [16:11:22]