[00:00:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 7.08, 6.55 [00:00:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.70, 7.68, 6.76 [00:00:57] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 2781MiB (11% inode=69%); [00:01:40] RECOVERY - mw171 php-fpm on mw171 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [00:01:44] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 8.42, 4.52, 2.40 [00:01:47] RECOVERY - mw172 php-fpm on mw172 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [00:01:57] RECOVERY - mw172 ferm_active on mw172 is OK: OK ferm input default policy is set [00:02:06] RECOVERY - mw171 ferm_active on mw171 is OK: OK ferm input default policy is set [00:02:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.97, 6.57 [00:02:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.35, 7.54, 6.83 [00:02:40] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 15.40, 11.46, 8.76 [00:03:16] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 11 seconds ago with 3 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Service[cgroup] [00:03:19] PROBLEM - mw172 HTTPS on mw172 is WARNING: HTTP WARNING: HTTP/2 403 - 146 bytes in 0.006 second response time [00:04:02] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 33 seconds ago with 3 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Service[cgroup] [00:04:14] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [00:04:28] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [00:04:57] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 21.47, 12.63, 9.25 [00:05:04] RECOVERY - cp26 Disk Space on cp26 is OK: DISK OK - free space: / 16251MiB (17% inode=98%); [00:05:18] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:05:20] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw171, mw172] [00:05:36] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 19.71, 14.85, 10.83 [00:05:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:55] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:06:02] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.374 second response time [00:06:07] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:06:08] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:06:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.64, 6.27, 6.43 [00:06:21] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw171, mw172] - SUCCESS in 61s [00:06:32] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:06:43] PROBLEM - mw171 HTTPS on mw171 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.066 second response time [00:07:03] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:07:10] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:07:12] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw171, mw172] [00:07:26] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:07:29] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:07:40] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:07:49] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.63, 3.50, 2.90 [00:08:02] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw171, mw172] - SUCCESS in 49s [00:08:04] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:08:06] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw132 mw141 mw142 mw133 mw134 [00:08:11] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.94, 4.49, 2.52 [00:08:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:25] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.36, 6.62, 6.66 [00:08:29] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [00:08:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:59] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.845 second response time [00:09:05] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.167 second response time [00:09:12] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.137 second response time [00:09:23] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:09:31] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 8.710 second response time [00:09:44] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.65, 3.44, 2.95 [00:09:45] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw171, mw172] [00:09:47] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:10:47] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 33 minutes ago with 0 failures [00:11:14] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.333 second response time [00:11:20] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.708 second response time [00:11:36] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw171, mw172] - SUCCESS in 111s [00:11:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.59, 3.05 [00:11:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:45] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [00:11:46] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.527 second response time [00:11:50] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.691 second response time [00:11:52] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [00:12:02] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.166 second response time [00:12:03] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:12:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:19] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.228 second response time [00:12:19] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.938 second response time [00:12:22] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.207 second response time [00:12:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:23] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.952 second response time [00:12:35] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 47% [00:12:54] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:19] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [00:13:34] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.60, 3.47, 3.06 [00:14:21] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.69, 11.37, 10.70 [00:14:35] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [00:14:50] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 42% [00:15:29] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.84, 3.37, 3.08 [00:15:29] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.321 second response time [00:15:38] [02miraheze/MirahezeDebug] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/11999f62edbe...ca07bd362448 [00:15:41] [02miraheze/MirahezeDebug] 07MacFan4000 03ca07bd3 - add mw161/162/171/172 [00:15:43] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [00:15:48] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [00:16:01] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [00:16:40] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 3% [00:16:44] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 33% [00:16:48] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.81, 11.04, 10.90 [00:16:55] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [00:18:34] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.78, 10.97, 11.63 [00:18:41] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:18:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.86, 3.97, 3.90 [00:19:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e3a1ea84a2c7 [00:19:35] [02miraheze/puppet] 07paladox 03e3a1ea8 - Install phorge171 [00:19:38] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:19:41] [02puppet] 07paladox opened pull request 03#3681: Install phorge171 - 13https://github.com/miraheze/puppet/pull/3681 [00:20:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/e3a1ea84a2c7...531b037c9bf6 [00:20:03] [02miraheze/puppet] 07paladox 03531b037 - Create phab171.yaml [00:20:06] [02puppet] 07paladox synchronize pull request 03#3681: Install phorge171 - 13https://github.com/miraheze/puppet/pull/3681 [00:20:10] [02puppet] 07paladox closed pull request 03#3681: Install phorge171 - 13https://github.com/miraheze/puppet/pull/3681 [00:20:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [00:20:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/727d80286f03...b808f062545c [00:20:16] [02miraheze/puppet] 07paladox 03b808f06 - Install phorge171 (#3681) [00:20:18] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:21:13] PROBLEM - swiftac171 Swift Container Service on swiftac171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:20] PROBLEM - swiftac171 Swift Account Service on swiftac171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:37] [02miraheze/puppet] 07Universal-Omega pushed 039 commits to 03remove-mail [+4/-0/±14] 13https://github.com/miraheze/puppet/compare/8b5f00e8925a...7bf8cafef79d [00:21:40] [02miraheze/puppet] 07Universal-Omega 037bf8caf - Merge branch 'master' into remove-mail [00:21:41] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [00:21:52] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.76, 3.90, 3.73 [00:21:56] PROBLEM - swiftac171 ferm_active on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:01] PROBLEM - swiftac171 Current Load on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:02] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.63, 8.99, 10.08 [00:22:04] PROBLEM - swiftac171 NTP time on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:08] PROBLEM - phorge171 conntrack_table_size on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:08] PROBLEM - swiftac171 PowerDNS Recursor on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:08] PROBLEM - phorge171 PowerDNS Recursor on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:11] PROBLEM - phorge171 Puppet on phorge171 is UNKNOWN: NRPE: Unable to read output [00:22:14] PROBLEM - swiftac171 Disk Space on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:22] PROBLEM - phorge171 ferm_active on phorge171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:23] PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:28] PROBLEM - swiftac171 conntrack_table_size on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:30] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:31] PROBLEM - swiftac171 Backups Swift Account Container on swiftac171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:22:31] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:38] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.20, 9.52, 10.09 [00:22:51] [02puppet] 07MacFan4000 reviewed pull request 03#3681 commit - 13https://github.com/miraheze/puppet/pull/3681#discussion_r1465703245 [00:23:55] PROBLEM - swiftac171 ferm_active on swiftac171 is UNKNOWN: NRPE: Unable to read output [00:23:59] RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 0.54, 0.57, 0.27 [00:24:02] RECOVERY - swiftac171 PowerDNS Recursor on swiftac171 is OK: DNS OK: 0.121 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [00:24:04] RECOVERY - phorge171 PowerDNS Recursor on phorge171 is OK: DNS OK: 0.224 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [00:24:06] RECOVERY - swiftac171 NTP time on swiftac171 is OK: NTP OK: Offset -0.0007295310497 secs [00:24:07] PROBLEM - phorge171 conntrack_table_size on phorge171 is UNKNOWN: NRPE: Unable to read output [00:24:09] RECOVERY - swiftac171 Disk Space on swiftac171 is OK: DISK OK - free space: / 86264MiB (96% inode=98%); [00:24:18] PROBLEM - phorge171 ferm_active on phorge171 is UNKNOWN: NRPE: Unable to read output [00:24:21] PROBLEM - swiftac171 Puppet on swiftac171 is UNKNOWN: NRPE: Unable to read output [00:24:24] PROBLEM - swiftac171 conntrack_table_size on swiftac171 is UNKNOWN: NRPE: Unable to read output [00:24:25] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:24:32] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.87, 10.73, 11.16 [00:24:40] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.861 second response time [00:25:39] RECOVERY - swiftac171 ferm_active on swiftac171 is OK: OK ferm input default policy is set [00:25:59] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.53, 2.97, 3.37 [00:26:06] RECOVERY - phorge171 conntrack_table_size on phorge171 is OK: OK: nf_conntrack is 0 % full [00:26:07] PROBLEM - phorge171 phabricator.miraheze.org HTTPS on phorge171 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:26:15] PROBLEM - phorge171 php-fpm on phorge171 is UNKNOWN: NRPE: Command 'check_php-fpm' not defined [00:26:18] PROBLEM - phorge171 phab.miraheze.wiki HTTPS on phorge171 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:26:18] RECOVERY - phorge171 ferm_active on phorge171 is OK: OK ferm input default policy is set [00:26:19] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:26:21] RECOVERY - swiftac171 conntrack_table_size on swiftac171 is OK: OK: nf_conntrack is 0 % full [00:26:22] PROBLEM - phorge171 phd on phorge171 is UNKNOWN: NRPE: Command 'check_phd' not defined [00:26:43] PROBLEM - phorge171 Backups Phabricator Static on phorge171 is UNKNOWN: NRPE: Command 'check_Backups Phabricator Static' not defined [00:27:33] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.12, 10.50, 10.45 [00:28:26] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.04, 10.93, 11.28 [00:28:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/b808f062545c...67ec1c91e17c [00:28:43] [02miraheze/puppet] 07paladox 0367ec1c9 - Rename phab171.yaml to phorge171.yaml [00:29:46] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:30:04] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [00:30:07] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 12 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/srv/phab/phorge/conf/local/local.json] [00:30:13] PROBLEM - phorge171 phd on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [00:30:24] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.35, 11.32, 11.37 [00:30:29] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.76, 3.85, 3.76 [00:30:38] PROBLEM - phorge171 Backups Phabricator Static on phorge171 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/phabricator-backup.log [00:31:31] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.38, 9.81, 10.20 [00:31:41] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.733 second response time [00:31:59] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:32:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:32:18] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:32:22] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.77, 10.04, 10.93 [00:32:23] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.38, 10.99, 10.58 [00:32:24] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.15, 3.75, 3.74 [00:33:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.99, 6.95, 6.66 [00:33:26] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:33:43] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:33:46] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:34:04] RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd' [00:34:05] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [00:34:07] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:34:14] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.827 second response time [00:34:14] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw131 mw141 mw133 mw134 [00:34:20] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.46, 8.09, 10.11 [00:34:22] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.39, 8.91, 9.86 [00:34:25] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:34:36] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:44] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw132 mw141 mw143 [00:34:49] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw132 mw141 mw143 [00:34:55] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:34:56] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.461 second response time [00:35:00] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw132 mw141 mw142 mw143 [00:35:06] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 5 backends are down. mw132 mw141 mw142 mw133 mw143 [00:35:13] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:35:16] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:35:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.04, 6.32, 6.47 [00:35:40] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.511 second response time [00:36:13] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [00:36:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 3.86, 3.76 [00:36:20] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.342 second response time [00:36:37] PROBLEM - swiftobject171 Swift Object Service on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 6000: Connection refused [00:36:45] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.930 second response time [00:36:56] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 4.78, 7.74, 11.39 [00:37:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 4.93, 8.63, 11.81 [00:37:48] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.610 second response time [00:38:11] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.669 second response time [00:38:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.37, 3.72, 3.73 [00:38:23] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 50% [00:38:34] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [00:38:43] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [00:38:58] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [00:39:01] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.333 second response time [00:39:03] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [00:39:04] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.985 second response time [00:39:07] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.649 second response time [00:39:20] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.581 second response time [00:39:25] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.865 second response time [00:39:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.09, 11.06, 12.34 [00:39:50] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:39:58] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [00:40:07] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [00:40:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.93, 4.22, 3.89 [00:40:29] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 41% [00:40:45] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.85, 10.67, 11.70 [00:41:48] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.071 second response time [00:42:04] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [00:42:12] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.26, 3.85, 3.80 [00:42:16] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 30% [00:42:28] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 31% [00:42:39] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.70, 10.75, 11.62 [00:43:10] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:43:13] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:43:27] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:44:09] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.65, 4.42, 4.00 [00:44:39] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:45:05] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:45:06] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.655 second response time [00:45:12] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.609 second response time [00:45:14] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.30, 10.18, 10.00 [00:45:28] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.048 second response time [00:45:51] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:46:01] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [00:46:03] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw131 mw134 [00:46:14] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [00:46:29] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.96, 11.71, 11.78 [00:46:37] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.717 second response time [00:46:39] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:47:05] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.293 second response time [00:47:13] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.23, 11.85, 10.67 [00:47:50] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.95, 3.26 [00:47:50] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.840 second response time [00:47:53] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 9.42, 6.40, 4.33 [00:47:58] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [00:48:01] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [00:48:02] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:48:06] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.43, 3.59, 3.77 [00:48:09] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [00:48:45] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 7.999 second response time [00:49:13] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.95, 10.60, 10.32 [00:49:13] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:49:22] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:49:25] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:49:35] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:49:35] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:50:36] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw133 [00:50:48] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw133 [00:50:51] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw133 [00:50:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [00:51:13] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 2.317 second response time [00:51:23] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.097 second response time [00:51:25] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.891 second response time [00:51:34] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.601 second response time [00:51:35] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.871 second response time [00:52:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:52:08] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.581 second response time [00:52:33] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [00:52:47] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [00:52:49] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [00:52:51] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [00:53:11] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.16, 9.78, 10.11 [00:54:51] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:56:50] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.333 second response time [00:57:40] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:58:13] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:58:42] PROBLEM - mem131 Current Load on mem131 is CRITICAL: CRITICAL - load average: 4.38, 2.50, 1.29 [00:59:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/afd72e8f64db [00:59:13] [02miraheze/puppet] 07paladox 03afd72e8 - swift: Fix bind_ip [00:59:14] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:59:16] [02puppet] 07paladox opened pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [00:59:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/afd72e8f64db...95b04eeb8b27 [00:59:40] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:59:41] [02miraheze/puppet] 07paladox 0395b04ee - Update container-server.conf.erb [00:59:44] [02puppet] 07paladox synchronize pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [00:59:45] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:59:46] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 7.186 second response time [00:59:47] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [00:59:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/95b04eeb8b27...8557885634ba [00:59:52] [02miraheze/puppet] 07paladox 038557885 - Update account-server.conf.erb [00:59:52] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw132 mw134 [00:59:53] [02puppet] 07paladox synchronize pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [00:59:58] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.53, 3.05, 3.38 [01:00:11] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.706 second response time [01:00:22] RECOVERY - swiftac171 Swift Account Service on swiftac171 is OK: TCP OK - 0.000 second response time on 10.0.17.125 port 6002 [01:00:48] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 16.53, 11.00, 9.89 [01:01:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8557885634ba...fbc7395ca5fc [01:01:36] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:01:39] [02miraheze/puppet] 07paladox 03fbc7395 - Update ac.pp [01:01:40] [02puppet] 07paladox synchronize pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [01:01:44] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [01:01:45] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.603 second response time [01:01:52] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [01:01:57] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.34, 11.47, 9.83 [01:02:12] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.99, 11.53, 10.91 [01:02:18] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:02:28] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:02:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fbc7395ca5fc...33ed90a930d7 [01:02:33] [02miraheze/puppet] 07paladox 0333ed90a - Update account-server.conf.erb [01:02:36] [02puppet] 07paladox synchronize pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [01:02:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/33ed90a930d7...623470a1f613 [01:02:45] [02miraheze/puppet] 07paladox 03623470a - Update container-server.conf.erb [01:02:47] [02puppet] 07paladox synchronize pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [01:02:50] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.472 second response time [01:02:51] RECOVERY - mem131 Current Load on mem131 is OK: OK - load average: 2.59, 3.00, 1.81 [01:02:53] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.492 second response time [01:03:00] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:03:11] [02puppet] 07paladox closed pull request 03#3682: swift: Fix bind_ip - 13https://github.com/miraheze/puppet/pull/3682 [01:03:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/67ec1c91e17c...842287ff8829 [01:03:15] PROBLEM - mem131 PowerDNS Recursor on mem131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:03:17] [02miraheze/puppet] 07paladox 03842287f - swift: Fix bind_ip (#3682) [01:03:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [01:03:20] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:03:21] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:03:21] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.468 second response time [01:03:35] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:03:39] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:03:54] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:04:11] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.41, 14.34, 12.08 [01:04:22] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.528 second response time [01:04:23] RECOVERY - swiftac171 Swift Container Service on swiftac171 is OK: TCP OK - 0.000 second response time on 10.0.17.125 port 6001 [01:04:46] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [01:04:53] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.654 second response time [01:05:14] RECOVERY - mem131 PowerDNS Recursor on mem131 is OK: DNS OK: 0.522 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [01:05:21] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw141 mw142 [01:05:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw143 [01:05:31] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:05:36] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw143 [01:05:41] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw143 [01:05:41] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw143 [01:05:43] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.552 second response time [01:05:44] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw143 [01:05:49] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:52] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw143 [01:05:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.41, 3.45, 3.46 [01:06:00] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:06:28] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:07:01] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:07:33] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:07:44] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.158 second response time [01:07:46] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.697 second response time [01:07:52] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 9.249 second response time [01:07:55] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:07:55] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 8.210 second response time [01:07:58] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.96, 4.59, 3.89 [01:08:15] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 51% [01:08:23] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.680 second response time [01:08:29] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.261 second response time [01:08:57] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:09:33] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.755 second response time [01:09:53] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [01:10:01] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:10:03] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:10:04] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.524 second response time [01:10:12] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.586 second response time [01:10:13] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.077 second response time [01:10:14] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [01:10:53] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.670 second response time [01:11:07] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.192 second response time [01:11:14] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [01:11:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/490022e8fbc0 [01:11:27] [02miraheze/puppet] 07paladox 03490022e - Swift Phabricator backend to phorge171 [01:11:28] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:11:30] [02puppet] 07paladox opened pull request 03#3683: Swift Phabricator backend to phorge171 - 13https://github.com/miraheze/puppet/pull/3683 [01:11:31] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [01:11:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [01:11:31] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.341 second response time [01:11:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [01:11:36] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [01:11:41] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:11:42] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [01:11:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/490022e8fbc0...51fd251a9c41 [01:11:59] [02miraheze/puppet] 07paladox 0351fd251 - Update default.vcl [01:12:01] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.701 second response time [01:12:01] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.293 second response time [01:12:01] [02puppet] 07paladox synchronize pull request 03#3683: Swift Phabricator backend to phorge171 - 13https://github.com/miraheze/puppet/pull/3683 [01:12:13] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 56% [01:12:29] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:12:59] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:13:22] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 52% [01:13:39] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:13:45] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 2.645 second response time [01:13:48] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.378 second response time [01:14:08] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:14:09] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/8bf93016e09d [01:14:09] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:14:12] [02miraheze/dns] 07paladox 038bf9301 - Add db182-private with its private ip [01:14:14] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:14:16] [02dns] 07paladox opened pull request 03#466: Add db182-private with its private ip - 13https://github.com/miraheze/dns/pull/466 [01:14:20] [02dns] 07paladox closed pull request 03#466: Add db182-private with its private ip - 13https://github.com/miraheze/dns/pull/466 [01:14:22] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/42d0eb50de86...d39dedc4e255 [01:14:24] [02miraheze/dns] 07paladox 03d39dedc - Add db182-private with its private ip (#466) [01:14:26] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [01:14:27] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [01:14:28] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.124 second response time [01:15:06] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.107 second response time [01:15:12] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw133 mw134 mw143 [01:15:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw131 mw133 [01:15:30] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.468 second response time [01:15:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [01:15:32] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [01:15:33] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [01:15:38] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.950 second response time [01:15:39] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:15:41] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [01:15:49] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.438 second response time [01:15:49] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.954 second response time [01:15:54] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.416 second response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [01:16:11] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.830 second response time [01:16:16] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.140 second response time [01:16:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f825c6a1f323 [01:16:34] [02miraheze/puppet] 07paladox 03f825c6a - Phabricator: swift db to db182-private [01:16:37] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [01:16:39] [02puppet] 07paladox opened pull request 03#3684: Phabricator: swift db to db182-private - 13https://github.com/miraheze/puppet/pull/3684 [01:17:02] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.614 second response time [01:17:07] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.518 second response time [01:17:15] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 34% [01:17:31] PROBLEM - phorge171 phd on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [01:17:37] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 8.552 second response time [01:17:57] PROBLEM - phorge171 Puppet on phorge171 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [01:18:04] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [01:18:10] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 34% [01:18:27] PROBLEM - phab121 Puppet on phab121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [01:18:28] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: connect to address 2a10:6740::6:311 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:18:53] PROBLEM - phab121 phd on phab121 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [01:19:07] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [01:19:12] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: connect to address 2a10:6740::6:311 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [01:19:29] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [01:19:52] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [01:20:29] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:20:32] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:21:04] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:21:05] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 1.101 second response time [01:21:32] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3361 bytes in 0.526 second response time [01:21:35] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.223 second response time [01:22:15] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:22:17] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:22:24] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [01:22:29] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.764 second response time [01:22:30] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.527 second response time [01:22:31] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:22:31] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:23:05] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw134 [01:23:24] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw143 [01:23:30] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.327 second response time [01:23:36] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.347 second response time [01:23:48] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.710 second response time [01:23:52] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw141 mw133 mw134 mw143 [01:23:54] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:06] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 48% [01:24:17] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 2.861 second response time [01:24:21] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.512 second response time [01:24:25] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.05, 1.67, 1.10 [01:24:26] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.836 second response time [01:24:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:24:34] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 2.607 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [01:25:03] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 46% [01:25:10] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 8.744 second response time [01:25:55] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.116 second response time [01:26:24] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.600 second response time [01:26:30] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:27:18] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:27:46] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.383 second response time [01:27:55] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [01:28:14] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:28:18] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.472 second response time [01:28:23] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.467 second response time [01:28:25] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.70, 1.78, 1.28 [01:28:39] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:28:55] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [01:29:09] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.596 second response time [01:29:23] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [01:29:51] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.497 second response time [01:29:54] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.583 second response time [01:30:11] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.596 second response time [01:30:16] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.702 second response time [01:30:22] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.592 second response time [01:30:25] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.59, 1.31, 1.16 [01:30:27] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.494 second response time [01:30:52] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 49% [01:30:55] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [01:31:33] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 8.833 second response time [01:31:41] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:32:25] PROBLEM - os131 SSH on os131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:48] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.729 second response time [01:33:12] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:33:44] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.516 second response time [01:33:44] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:34:22] RECOVERY - os131 SSH on os131 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [01:34:45] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 37% [01:35:10] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.912 second response time [01:35:22] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw131 [01:35:44] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.027 second response time [01:35:52] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [01:36:05] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:36:20] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:36:27] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [01:36:44] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:36:59] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:37:36] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:37:40] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:37:55] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:38:00] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 37% [01:38:18] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [01:38:25] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.716 second response time [01:38:45] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.251 second response time [01:39:04] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.971 second response time [01:39:12] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [01:39:22] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [01:39:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [01:39:32] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [01:39:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [01:39:40] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:39:56] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.849 second response time [01:40:15] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.374 second response time [01:40:17] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.732 second response time [01:41:41] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.739 second response time [01:41:44] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 3.545 second response time [01:42:01] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 7.232 second response time [01:42:19] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.460 second response time [01:42:24] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:42:24] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 1.099 second response time [01:42:24] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [01:42:47] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 6 backends are down. mw131 mw141 mw142 mw133 mw134 mw143 [01:42:50] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw131 mw141 mw133 mw134 [01:42:53] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:43:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 5.168 second response time [01:43:30] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw133 [01:43:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw132 mw141 mw142 mw133 [01:43:33] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw141 mw142 mw133 [01:43:51] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:44:05] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.731 second response time [01:44:24] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.716 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [01:44:27] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.508 second response time [01:44:28] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:44:59] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.886 second response time [01:45:06] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw132 [01:45:12] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.582 second response time [01:45:23] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw131 mw132 mw141 [01:45:56] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 8.194 second response time [01:46:04] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.621 second response time [01:46:22] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.920 second response time [01:46:29] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.928 second response time [01:46:33] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 43% [01:46:42] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [01:46:45] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [01:47:05] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [01:47:23] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [01:47:28] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [01:47:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [01:47:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [01:48:33] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 34% [01:48:37] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:49:13] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:49:21] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:50:02] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:50:27] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:50:31] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:51:00] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw141 mw133 mw134 mw143 [01:51:03] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:51:04] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.471 second response time [01:51:05] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:19] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 9.827 second response time [01:51:22] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 6 backends are down. mw131 mw141 mw142 mw133 mw134 mw143 [01:51:26] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:51:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [01:51:33] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [01:52:04] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 5.541 second response time [01:52:27] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.527 second response time [01:52:29] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.630 second response time [01:52:38] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw131 [01:52:40] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw131 [01:52:42] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.507 second response time [01:52:59] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [01:53:04] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.903 second response time [01:53:06] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.313 second response time [01:53:20] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.707 second response time [01:53:22] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [01:53:25] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [01:53:31] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [01:53:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [01:54:11] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:54:15] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:54:15] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:54:23] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:54:36] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:55:21] [02puppet] 07paladox edited pull request 03#3683: Switch Phabricator backend to phorge171 - 13https://github.com/miraheze/puppet/pull/3683 [01:55:24] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:55:30] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:55:32] [02puppet] 07paladox closed pull request 03#3683: Switch Phabricator backend to phorge171 - 13https://github.com/miraheze/puppet/pull/3683 [01:55:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/842287ff8829...81c186ce7146 [01:55:37] [02miraheze/puppet] 07paladox 0381c186c - Switch Phabricator backend to phorge171 (#3683) [01:55:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [01:55:41] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:56:11] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.988 second response time [01:56:12] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.518 second response time [01:56:15] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.520 second response time [01:56:22] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.544 second response time [01:56:34] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.773 second response time [01:57:20] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.617 second response time [01:57:29] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.879 second response time [01:57:31] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 7.211 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [01:58:30] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [01:58:33] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [01:58:37] [02puppet] 07paladox reviewed pull request 03#3681 commit - 13https://github.com/miraheze/puppet/pull/3681#discussion_r1465753882 [01:59:57] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.07, 10.66, 11.98 [02:03:15] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:03:36] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:13] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:16] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:05:38] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:06:08] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:06:12] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 4 backends are down. mw131 mw132 mw134 mw143 [02:06:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mw143 [02:06:20] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:06:22] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw131 mw132 mw134 mw143 [02:06:23] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mw143 [02:06:24] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.519 second response time [02:06:26] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 5 backends are down. mw131 mw132 mw133 mw134 mw143 [02:06:26] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:06:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:07:13] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.625 second response time [02:07:13] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.000 second response time [02:07:23] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.629 second response time [02:08:11] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [02:08:12] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:08:19] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.129 second response time [02:08:22] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.623 second response time [02:08:25] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.481 second response time [02:08:26] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.742 second response time [02:08:34] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:08:42] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 22 minutes ago with 0 failures [02:08:43] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:09:48] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.612 second response time [02:09:57] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.09, 11.98, 11.81 [02:10:10] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.293 second response time [02:10:16] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.771 second response time [02:10:31] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:11:57] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.34, 10.50, 11.29 [02:12:13] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [02:12:15] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [02:12:16] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [02:12:18] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [02:12:29] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.609 second response time [02:12:37] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.858 second response time [02:12:49] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.968 second response time [02:13:14] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:13:36] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:14:06] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:14:28] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:15:12] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.853 second response time [02:15:36] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.698 second response time [02:16:02] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [02:16:04] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.623 second response time [02:16:08] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [02:16:27] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw131 mw132 mw134 [02:16:49] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:16:56] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:17:00] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw131 mw134 [02:17:01] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw131 mw134 [02:17:02] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.927 second response time [02:17:18] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:17:35] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.462 second response time [02:17:52] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:17:54] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.974 second response time [02:17:55] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:18:04] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 21.73, 12.75, 11.61 [02:18:04] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [02:18:11] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:18:29] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.78, 1.10, 0.57 [02:18:37] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.514 second response time [02:18:37] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [02:18:50] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.425 second response time [02:18:55] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.474 second response time [02:19:17] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.594 second response time [02:19:35] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.646 second response time [02:19:49] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.602 second response time [02:19:51] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 1.207 second response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [02:19:53] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.487 second response time [02:19:58] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [02:19:59] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [02:20:02] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [02:20:02] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.47, 11.17, 11.19 [02:20:06] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.615 second response time [02:20:18] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [02:20:34] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [02:20:36] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.299 second response time [02:20:54] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [02:21:01] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [02:21:39] [02puppet] 07paladox edited pull request 03#3684: Phabricator: switch db to db182-private - 13https://github.com/miraheze/puppet/pull/3684 [02:21:56] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:20] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.02, 1.55, 0.88 [02:22:51] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.61, 9.67, 11.64 [02:22:58] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:23:11] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:23:54] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw143 [02:23:56] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw132 mw143 [02:24:00] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 3.948 second response time [02:24:01] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.774 second response time [02:24:11] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [02:24:16] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.99, 1.64, 0.99 [02:24:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/81c186ce7146...e8449652fcb4 [02:24:17] [02miraheze/puppet] 07paladox 03e844965 - mariadb: allow binding to private ip [02:24:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:24:31] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [02:24:47] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.30, 12.63, 12.47 [02:24:51] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw143 [02:25:00] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw143 [02:25:02] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 7.955 second response time [02:25:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e8449652fcb4...4e0bebf91e4f [02:25:03] [02miraheze/puppet] 07paladox 034e0bebf - matomo: use db182-private [02:25:10] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.948 second response time [02:25:51] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [02:25:55] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [02:25:56] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.772 second response time [02:26:07] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [02:26:11] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.60, 1.64, 1.07 [02:26:28] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [02:26:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a74380a910cf [02:26:34] [02miraheze/puppet] 07paladox 03a74380a - icinga2: switch to db182-private [02:26:35] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:26:37] [02puppet] 07paladox opened pull request 03#3685: icinga2: switch to db182-private - 13https://github.com/miraheze/puppet/pull/3685 [02:26:48] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [02:27:00] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [02:27:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a74380a910cf...314cc12b7289 [02:27:10] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:27:10] [02miraheze/puppet] 07paladox 03314cc12 - Update init.pp [02:27:11] [02puppet] 07paladox synchronize pull request 03#3685: icinga2: switch to db182-private - 13https://github.com/miraheze/puppet/pull/3685 [02:27:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/314cc12b7289...a38595d7729b [02:27:30] [02miraheze/puppet] 07paladox 03a38595d - Update mon181.yaml [02:27:33] [02puppet] 07paladox synchronize pull request 03#3685: icinga2: switch to db182-private - 13https://github.com/miraheze/puppet/pull/3685 [02:27:36] [02puppet] 07paladox closed pull request 03#3685: icinga2: switch to db182-private - 13https://github.com/miraheze/puppet/pull/3685 [02:27:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/4e0bebf91e4f...a51655684388 [02:27:41] [02miraheze/puppet] 07paladox 03a516556 - icinga2: switch to db182-private (#3685) [02:27:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [02:27:44] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:27:58] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.28, 8.27, 9.77 [02:28:39] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.60, 10.72, 11.68 [02:29:10] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:29:15] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.698 second response time [02:29:49] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.71, 1.68, 1.20 [02:30:53] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:30:59] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:31:11] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.146 second response time [02:31:50] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:32:07] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.74, 10.32, 11.98 [02:33:08] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:33:12] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.463 second response time [02:33:17] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.373 second response time [02:33:20] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:33:21] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3361 bytes in 0.511 second response time [02:33:22] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.461 second response time [02:33:24] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.462 second response time [02:33:26] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.477 second response time [02:33:33] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:33:45] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.65, 2.10, 1.47 [02:33:54] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:34:14] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 6 backends are down. mw131 mw132 mw142 mw133 mw134 mw143 [02:34:33] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw141 mw142 mw134 mw143 [02:34:43] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:34:44] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:34:48] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw141 mw142 mw134 mw143 [02:34:49] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [02:34:54] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw141 mw142 mw133 mw143 [02:35:02] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 5.81, 7.61, 11.22 [02:35:04] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.671 second response time [02:35:10] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.788 second response time [02:35:11] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 4.285 second response time [02:35:13] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.32, 7.77, 11.57 [02:35:18] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.557 second response time [02:35:20] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 3.796 second response time [02:35:21] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.705 second response time [02:35:21] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.170 second response time [02:35:24] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.982 second response time [02:35:30] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 5.984 second response time [02:35:54] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.576 second response time [02:35:55] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.494 second response time [02:36:25] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.43, 8.51, 10.17 [02:36:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 21 backends are healthy [02:36:42] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 21 backends are healthy [02:36:42] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 21 backends are healthy [02:36:45] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 21 backends are healthy [02:36:45] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 21 backends are healthy [02:36:55] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 21 backends are healthy [02:36:55] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.20, 7.75, 11.35 [02:37:12] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.188 second response time [02:37:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.08, 9.35, 11.94 [02:37:37] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.485 second response time [02:38:09] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 21 backends are healthy [02:39:50] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.86, 7.86, 10.05 [02:40:26] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.29, 9.27, 10.24 [02:41:20] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:41:37] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.39, 1.86, 1.69 [02:42:25] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.55, 9.27, 10.13 [02:43:17] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.301 second response time [02:43:34] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.32, 1.32, 1.51 [02:44:25] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.93, 2.67, 3.88 [02:44:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.74, 8.36, 9.95 [02:46:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.39, 8.14, 10.04 [02:46:54] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': '1.40'} to mwtask181 [02:47:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 10.16, 10.98 [02:48:00] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': '1.40'} to mwtask181 - SUCCESS in 65s [02:48:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:12] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40', '1.41']} to mwtask181 [02:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.11, 9.43, 10.62 [02:50:18] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40', '1.41']} to mwtask181 - SUCCESS in 126s [02:50:18] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.59, 2.30, 3.36 [02:50:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.48, 10.76, 10.72 [02:54:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.02, 11.26, 10.51 [02:55:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 11.70, 11.20 [02:56:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.19, 10.78, 10.81 [02:56:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.68, 10.57, 10.36 [02:57:01] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:57:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.26, 11.06, 11.03 [02:57:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.38, 10.68, 10.33 [02:58:59] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3456 bytes in 1.698 second response time [02:59:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.74, 12.13, 11.46 [02:59:27] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 18.52, 13.72, 11.48 [03:00:06] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:00:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 21.91, 15.00, 12.03 [03:01:13] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 51 seconds old and 92 bytes [03:02:03] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.590 second response time [03:03:18] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.72, 10.92, 10.89 [03:03:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.58, 11.35, 11.33 [03:04:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.50, 11.81, 11.41 [03:06:49] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.17, 8.92, 9.95 [03:09:24] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [03:09:38] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:11:22] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.546 second response time [03:11:37] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.632 second response time [03:14:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.33, 9.19, 9.94 [03:14:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.68, 8.77, 9.94 [03:15:55] [02TSPortal] 07Universal-Omega closed pull request 03#12: mail: update SMTP settings - 13https://github.com/miraheze/TSPortal/pull/12 [03:15:58] [02miraheze/TSPortal] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/cddde5097e5f...b202f53e521d [03:16:00] [02miraheze/TSPortal] 07Universal-Omega 03b202f53 - mail: update SMTP settings [03:16:02] [02miraheze/TSPortal] 07Universal-Omega deleted branch 03smtp [03:16:05] [02TSPortal] 07Universal-Omega deleted branch 03smtp - 13https://github.com/miraheze/TSPortal [03:17:08] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:17:14] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:19:06] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.608 second response time [03:19:12] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.740 second response time [03:19:27] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.80, 8.08, 9.79 [03:22:07] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.05, 2.34, 3.85 [03:26:07] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.16, 1.14, 3.01 [03:26:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 1.71, 2.29, 3.85 [03:26:58] [02TSPortal] 07paladox commented on pull request 03#11: Bump guzzlehttp/psr7 from 2.4.3 to 2.5.0 - 13https://github.com/miraheze/TSPortal/pull/11#issuecomment-1909283149 [03:27:02] [02TSPortal] 07dependabot[bot] edited pull request 03#11: Bump guzzlehttp/psr7 from 2.4.3 to 2.5.0 - 13https://github.com/miraheze/TSPortal/pull/11 [03:27:19] [02TSPortal] 07dependabot[bot] edited pull request 03#11: Bump guzzlehttp/psr7 from 2.4.3 to 2.5.0 - 13https://github.com/miraheze/TSPortal/pull/11 [03:27:21] [02miraheze/TSPortal] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/guzzlehttp/psr7-2.6.2 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/commit/7265f95d3af5 [03:27:24] [02miraheze/TSPortal] 07dependabot[bot] 037265f95 - Bump guzzlehttp/psr7 from 2.4.3 to 2.6.2 [03:27:26] [02TSPortal] 07dependabot[bot] created branch 03dependabot/composer/guzzlehttp/psr7-2.6.2 - 13https://github.com/miraheze/TSPortal [03:27:29] [02TSPortal] 07dependabot[bot] opened pull request 03#13: Bump guzzlehttp/psr7 from 2.4.3 to 2.6.2 - 13https://github.com/miraheze/TSPortal/pull/13 [03:27:32] [02TSPortal] 07dependabot[bot] labeled pull request 03#13: Bump guzzlehttp/psr7 from 2.4.3 to 2.6.2 - 13https://github.com/miraheze/TSPortal/pull/13 [03:27:33] [02TSPortal] 07dependabot[bot] closed pull request 03#11: Bump guzzlehttp/psr7 from 2.4.3 to 2.5.0 - 13https://github.com/miraheze/TSPortal/pull/11 [03:27:35] [02TSPortal] 07dependabot[bot] commented on pull request 03#11: Bump guzzlehttp/psr7 from 2.4.3 to 2.5.0 - 13https://github.com/miraheze/TSPortal/pull/11#issuecomment-1909283401 [03:27:36] [02miraheze/TSPortal] 07dependabot[bot] deleted branch 03dependabot/composer/guzzlehttp/psr7-2.5.0 [03:27:38] [02TSPortal] 07dependabot[bot] deleted branch 03dependabot/composer/guzzlehttp/psr7-2.5.0 - 13https://github.com/miraheze/TSPortal [03:28:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.67, 10.04, 9.17 [03:29:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.26, 10.74, 9.97 [03:30:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.21, 10.89, 9.58 [03:30:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.97, 10.10, 9.70 [03:32:48] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [03:32:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.11, 11.11, 9.86 [03:34:07] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.65, 9.87, 8.42 [03:34:34] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.97, 2.51, 3.31 [03:34:43] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:34:44] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.505 second response time [03:34:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.47, 12.78, 10.64 [03:36:07] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.15, 9.39, 8.43 [03:36:07] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.87, 3.38, 3.12 [03:36:43] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 2.668 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [03:36:49] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.77, 11.57, 10.41 [03:36:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.34, 12.46, 10.87 [03:38:00] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.07, 10.22, 9.51 [03:38:07] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.20, 2.52, 2.83 [03:38:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.85, 11.00, 10.44 [03:40:07] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 10.80, 9.19 [03:40:14] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.20, 10.17, 9.52 [03:41:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.31, 11.05, 11.02 [03:42:07] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.27, 9.27, 8.82 [03:42:09] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.87, 9.67, 9.50 [03:42:49] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.95, 9.61, 10.06 [03:42:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.99, 9.39, 9.91 [03:42:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.82, 11.74, 11.17 [03:47:27] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.86, 8.65, 9.95 [03:54:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.25, 11.05, 10.80 [03:56:49] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.91, 9.91, 9.57 [03:56:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.07, 10.77, 10.74 [03:56:59] [02puppet] 07paladox closed pull request 03#3684: Phabricator: switch db to db182-private - 13https://github.com/miraheze/puppet/pull/3684 [03:57:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a51655684388...f5893e18d72c [03:57:01] [02miraheze/puppet] 07paladox 03f5893e1 - Phabricator: switch db to db182-private (#3684) [03:57:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14 [03:57:07] [02puppet] 07paladox deleted branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [03:57:20] RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd' [03:57:23] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:58:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.36, 9.72, 9.57 [03:58:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.27, 11.86, 11.12 [04:00:10] [02puppet] 07Universal-Omega closed pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [04:00:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-33/±14] 13https://github.com/miraheze/puppet/compare/f5893e18d72c...615104d408df [04:00:13] [02miraheze/puppet] 07Universal-Omega 03615104d - Remove mail121, and roundcubemail, dovecot, and postfix modules (#3669) [04:00:16] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-mail [04:00:19] [02puppet] 07Universal-Omega deleted branch 03remove-mail - 13https://github.com/miraheze/puppet [04:00:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.32, 11.55, 11.12 [04:02:54] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.55, 11.90, 11.30 [04:03:38] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:04:17] !log remove mail121 from puppet and shut down [04:04:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.95, 12.05, 10.49 [04:05:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.31, 11.61, 10.37 [04:05:37] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.640 second response time [04:05:43] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-mail121 [+0/-1/±0] 13https://github.com/miraheze/dns/commit/4a3af85df831 [04:05:44] [02miraheze/dns] 07Universal-Omega 034a3af85 - Remove mail121 [04:05:45] [02dns] 07Universal-Omega created branch 03remove-mail121 - 13https://github.com/miraheze/dns [04:05:48] [02dns] 07Universal-Omega opened pull request 03#467: Remove mail121 - 13https://github.com/miraheze/dns/pull/467 [04:06:13] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:06:17] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0df7989f939...78d5f6e40f44 [04:06:20] [02miraheze/mw-config] 07AgentIsai 0378d5f6e - Update sitenotice [04:06:27] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-mail121 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4a3af85df831...bd7e7d57ccfe [04:06:28] [02miraheze/dns] 07Universal-Omega 03bd7e7d5 - Remove [04:06:30] [02dns] 07Universal-Omega synchronize pull request 03#467: Remove mail121 - 13https://github.com/miraheze/dns/pull/467 [04:06:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.40, 11.52, 10.50 [04:06:49] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.21, 11.72, 10.58 [04:06:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.78, 11.58, 11.44 [04:06:59] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [04:07:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:18] miraheze/mw-config - AgentIsai the build passed. [04:07:19] PROBLEM - phab121 SSH on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:20] PROBLEM - cp41 Nginx Backend for mail121 on cp41 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:07:21] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-mail121 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bd7e7d57ccfe...49904104d8e8 [04:07:22] [02miraheze/dns] 07Universal-Omega 034990410 - Remove [04:07:25] [02dns] 07Universal-Omega synchronize pull request 03#467: Remove mail121 - 13https://github.com/miraheze/dns/pull/467 [04:07:27] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.02, 10.38, 10.07 [04:07:37] PROBLEM - ping6 on mail121 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:07:47] PROBLEM - mail121 IMAP on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:07:58] PROBLEM - ping6 on phab121 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:08:09] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 70s [04:08:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:30] PROBLEM - Host mail121 is DOWN: PING CRITICAL - Packet loss = 100% [04:08:57] PROBLEM - Host phab121 is DOWN: PING CRITICAL - Packet loss = 100% [04:09:27] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.45, 9.65, 9.84 [04:09:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mail121-ptr [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/138464bc80e3 [04:09:53] [02miraheze/puppet] 07Universal-Omega 03138464b - recursor: remove mail121 PTR [04:09:54] [02puppet] 07Universal-Omega created branch 03mail121-ptr - 13https://github.com/miraheze/puppet [04:09:58] [02puppet] 07Universal-Omega opened pull request 03#3686: recursor: remove mail121 PTR - 13https://github.com/miraheze/puppet/pull/3686 [04:10:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.64, 9.41, 9.88 [04:10:53] [02puppet] 07Universal-Omega closed pull request 03#3686: recursor: remove mail121 PTR - 13https://github.com/miraheze/puppet/pull/3686 [04:10:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/615104d408df...1046a369e442 [04:10:58] [02miraheze/puppet] 07Universal-Omega 031046a36 - recursor: remove mail121 PTR (#3686) [04:11:00] [02miraheze/puppet] 07Universal-Omega deleted branch 03mail121-ptr [04:11:03] [02puppet] 07Universal-Omega deleted branch 03mail121-ptr - 13https://github.com/miraheze/puppet [04:11:09] PROBLEM - cp24 Nginx Backend for mail121 on cp24 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:11:45] [02dns] 07Universal-Omega closed pull request 03#467: Remove mail121 - 13https://github.com/miraheze/dns/pull/467 [04:11:48] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±2] 13https://github.com/miraheze/dns/compare/d39dedc4e255...43b3e94115a8 [04:11:49] [02miraheze/dns] 07Universal-Omega 0343b3e94 - Remove mail121 (#467) [04:11:51] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-mail121 [04:11:53] [02dns] 07Universal-Omega deleted branch 03remove-mail121 - 13https://github.com/miraheze/dns [04:12:55] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/78d5f6e40f44...addf8c2b7d04 [04:12:57] [02miraheze/mw-config] 07AgentIsai 03addf8c2 - Fix CSS [04:12:58] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [04:13:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:13:08] PROBLEM - cp35 Nginx Backend for mail121 on cp35 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:13:55] miraheze/mw-config - AgentIsai the build passed. [04:13:57] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 58s [04:14:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:14:49] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.37, 9.22, 9.89 [04:14:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.40, 8.39, 9.87 [04:23:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:25:56] PROBLEM - cp26 Nginx Backend for mail121 on cp26 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:27:15] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.35, 10.37, 10.05 [04:27:19] PROBLEM - cp34 Nginx Backend for mail121 on cp34 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:27:41] !log delete mail component from Miraheze Status [04:27:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:29:12] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.91, 10.01, 9.94 [04:30:57] PROBLEM - cp25 Nginx Backend for mail121 on cp25 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:31:25] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.65, 11.54, 10.17 [04:31:31] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.43, 10.98, 9.91 [04:31:54] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 18.84, 12.92, 10.49 [04:33:09] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 22.09, 16.38, 12.53 [04:34:18] PROBLEM - cp51 Nginx Backend for mail121 on cp51 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:34:42] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.68, 8.95, 7.32 [04:34:49] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:35:14] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:36:47] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.750 second response time [04:37:12] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [04:37:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/8c1f5405859b...3ba89f63afe0 [04:37:19] [02miraheze/ssl] 07MirahezeSSLBot 033ba89f6 - Bot: Add SSL cert for ff8.wiki [04:37:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/3ba89f63afe0...e847b9bb7bc1 [04:37:42] [02miraheze/ssl] 07MirahezeSSLBot 03e847b9b - Bot: Add SSL cert for www.satepedia.com [04:38:41] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.30, 9.18, 8.00 [04:39:11] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.96, 11.64, 11.07 [04:39:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.29, 11.56, 11.29 [04:44:50] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.07, 11.37, 11.94 [04:44:56] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.68, 8.82, 9.92 [04:45:22] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.37, 11.29, 11.76 [04:49:43] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.09, 9.15, 10.08 [04:52:25] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [04:54:26] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 10.50, 10.96 [04:54:47] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.06, 12.33, 11.53 [04:56:23] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.38, 10.36, 10.88 [04:56:31] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.42, 10.48, 10.39 [04:56:42] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.55, 11.81, 11.44 [05:01:13] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [05:01:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.33, 11.60, 10.30 [05:02:18] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.00, 9.68, 10.09 [05:02:21] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [05:03:13] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.570 second response time [05:03:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.03, 11.23, 10.31 [05:06:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.65, 12.30, 11.35 [05:08:07] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.00, 10.40, 10.30 [05:09:16] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 12.23, 10.98 [05:10:03] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.37, 10.03, 10.18 [05:10:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.94, 11.65, 10.96 [05:11:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.83, 11.49, 10.85 [05:12:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.66, 11.65, 11.54 [05:13:14] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.98, 12.19, 11.19 [05:13:56] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.67, 10.60, 10.36 [05:14:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.26, 11.86, 11.60 [05:15:13] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.92, 11.90, 11.22 [05:16:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.64, 11.41, 11.18 [05:16:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.09, 11.42, 11.47 [05:17:12] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.46, 12.79, 11.64 [05:17:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.35, 11.33, 10.69 [05:18:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.61, 12.26, 11.51 [05:19:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.27, 10.99, 10.64 [05:22:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.69, 11.80, 11.60 [05:23:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.14, 11.83, 11.03 [05:24:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.60, 11.63, 11.53 [05:26:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.01, 11.64, 11.57 [05:26:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.72, 12.14, 11.62 [05:27:50] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:27:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.86, 11.40, 11.01 [05:28:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.62, 11.37, 11.43 [05:30:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 11.98, 12.03, 11.73 [05:34:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.15, 11.81, 11.80 [05:35:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.91, 11.21, 11.79 [05:36:09] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.18, 12.45, 11.77 [05:37:05] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 11.63, 11.87 [05:37:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.28, 10.72, 10.64 [05:38:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.28, 11.96, 11.81 [05:39:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.08, 11.61, 11.85 [05:40:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.57, 11.76, 11.76 [05:42:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.40, 12.42, 12.00 [05:43:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.54, 11.56, 11.15 [05:45:45] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.96, 3.38, 2.16 [05:45:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.61, 12.05, 11.39 [05:46:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.00, 11.62, 11.79 [05:47:45] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.84, 3.15, 2.22 [05:47:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.39, 11.72, 11.37 [05:47:51] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.18, 6.30 [05:48:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.03, 11.76, 11.83 [05:49:47] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.14, 6.33, 6.09 [05:49:52] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.49, 3.55, 3.30 [05:50:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.88, 10.90, 11.52 [05:52:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.53, 11.03, 11.83 [05:53:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.79, 7.52, 6.71 [05:53:50] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.42, 11.03, 11.10 [05:55:36] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.23, 6.49, 6.44 [05:55:46] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.96, 3.25, 3.27 [05:55:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.82, 9.36, 10.47 [06:00:31] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e6ac66a84220...a3e265940c6d [06:00:34] [02miraheze/CreateWiki] 07AgentIsai 03a3e2659 - Revert change [06:01:50] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.04, 9.36, 10.18 [06:02:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.52, 11.80, 11.66 [06:05:05] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.21, 8.91, 10.03 [06:06:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.84, 11.54, 11.57 [06:06:45] miraheze/CreateWiki - AgentIsai the build passed. [06:08:07] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.74, 9.14, 10.06 [06:09:00] [02CreateWiki] 07AgentIsai synchronize pull request 03#460: Backport further changes from WikiForge/CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/460 [06:09:03] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7893d06b5a14...3f39c53da993 [06:09:06] [02miraheze/CreateWiki] 07AgentIsai 033f39c53 - Dummy change [06:10:27] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=trafficconepeoplewiki --no-updates --username-prefix=wikia:traffic-cone-people /home/reception/ImportDump/trafficconepeoplewiki-20240125003743.xml (START) [06:10:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:54] miraheze/CreateWiki - AgentIsai the build has errored. [06:13:57] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.34, 7.23, 6.67 [06:15:54] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.49, 6.83 [06:18:08] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.60, 8.65, 9.85 [06:19:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.17, 6.85, 6.76 [06:21:41] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.98, 5.96, 6.45 [06:21:56] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3f39c53da993...29cc752756fc [06:21:56] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 11.05, 9.95 [06:21:58] [02miraheze/CreateWiki] 07AgentIsai 0329cc752 - Fix trailing space [06:22:01] [02CreateWiki] 07AgentIsai synchronize pull request 03#460: Backport further changes from WikiForge/CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/460 [06:22:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.57, 10.84, 10.15 [06:22:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.47, 10.94, 10.66 [06:24:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.33, 11.69, 10.94 [06:25:12] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:25:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.70, 11.15, 10.27 [06:26:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.28, 11.12, 10.80 [06:27:52] miraheze/CreateWiki - AgentIsai the build passed. [06:29:00] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=trafficconepeoplewiki --no-updates --username-prefix=wikia:traffic-cone-people /home/reception/ImportDump/trafficconepeoplewiki-20240125003743.xml (END - exit=0) [06:29:01] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=trafficconepeoplewiki (START) [06:29:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:07] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.38, 9.97, 10.06 [06:35:41] [02CreateWiki] 07AgentIsai closed pull request 03#460: Backport further changes from WikiForge/CreateWiki - 13https://github.com/miraheze/CreateWiki/pull/460 [06:35:43] [02miraheze/CreateWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/a3e265940c6d...3f8003973176 [06:35:44] [02miraheze/CreateWiki] 07AgentIsai 033f80039 - Backport further changes from WikiForge/CreateWiki (#460) [06:35:47] [02miraheze/CreateWiki] 07AgentIsai deleted branch 03AgentIsai-patch-1 [06:35:48] [02CreateWiki] 07AgentIsai deleted branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/CreateWiki [06:37:43] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:37:47] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.06, 8.67, 9.70 [06:38:09] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.36, 8.76, 9.83 [06:39:42] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.109 second response time [06:40:14] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.64, 9.84, 7.97 [06:41:10] miraheze/CreateWiki - AgentIsai the build passed. [06:42:13] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.42, 8.19, 7.59 [06:53:03] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:02:30] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=trafficconepeoplewiki (END - exit=0) [07:02:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=trafficconepeoplewiki --active --update (END - exit=0) [07:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:10:21] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.00, 6.76, 6.34 [07:12:03] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:17] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.37, 6.60, 6.33 [07:14:11] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.883 second response time [07:15:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.85, 10.65, 8.79 [07:17:24] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.71, 9.51, 8.61 [07:25:35] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 45% [07:26:38] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:27:33] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 23% [07:28:08] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10221MiB (10% inode=98%); [07:28:37] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.492 second response time [07:35:08] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:37:07] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.713 second response time [07:45:30] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 174.10 ms [07:47:30] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 158.21 ms [07:56:05] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:13] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.816 second response time [08:10:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.80, 10.45, 8.33 [08:11:05] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.68, 12.79, 9.32 [08:12:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 21.58, 13.75, 9.56 [08:12:43] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.69, 12.12, 9.12 [08:14:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.52, 11.60, 9.31 [08:17:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.30, 11.03, 10.06 [08:18:07] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.10, 10.12, 9.64 [08:18:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.42, 10.39, 9.79 [08:18:31] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.64, 9.04, 8.79 [08:19:05] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 5.10, 9.19, 9.52 [08:20:08] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.78, 9.09, 9.41 [08:26:23] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:26:36] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [08:27:07] PROBLEM - cp41 Disk Space on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:27:32] PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:03] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 9817MiB (10% inode=98%); [08:29:31] RECOVERY - cp41 SSH on cp41 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [08:30:29] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.426 second response time [08:32:33] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 180.34 ms [08:41:50] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 227.97 ms [08:43:51] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.44 ms [08:43:58] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.30, 7.10, 6.04 [08:47:50] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.53, 7.22, 6.30 [08:50:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.96 ms [08:51:42] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.06, 6.57, 6.27 [08:52:01] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 198.28 ms [08:55:35] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.44, 7.19, 6.66 [08:57:31] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.33, 7.67, 6.89 [09:01:23] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.61, 7.83, 7.15 [09:07:11] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.78, 7.73, 7.17 [09:09:07] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.96, 6.86, 6.93 [09:19:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 6.87, 6.91 [09:21:03] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.74, 6.40, 6.74 [09:24:35] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 210.32 ms [09:26:35] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 190.80 ms [09:30:48] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 183.33 ms [09:32:48] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.80 ms [09:40:53] PROBLEM - itadminwiki.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'itadminwiki.pl' expires in 7 day(s) (Fri 02 Feb 2024 09:29:29 AM GMT +0000). [09:55:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.47, 9.58, 8.24 [09:56:47] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.01, 6.88, 6.43 [09:58:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.60, 9.94, 8.86 [09:58:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.41, 7.49, 6.74 [10:00:36] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:00:56] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:01:50] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.04, 9.94, 8.98 [10:02:33] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.696 second response time [10:02:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.42, 3.02, 2.15 [10:02:54] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.609 second response time [10:04:38] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.35, 3.16, 2.31 [10:04:46] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.27, 11.51, 9.35 [10:06:09] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 17.08, 12.69, 10.37 [10:06:28] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.57, 6.63, 6.67 [10:08:44] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.69, 10.36, 9.46 [10:10:08] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.89, 11.64, 10.54 [10:10:23] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.40, 7.75, 7.17 [10:10:43] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.39, 11.53, 9.99 [10:12:19] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.75, 6.65, 6.84 [10:12:42] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.96, 10.76, 9.89 [10:12:43] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.94, 10.41, 9.51 [10:14:15] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.88, 6.06, 6.60 [10:14:38] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.65, 9.50, 9.28 [10:16:08] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.92, 11.34, 10.68 [10:18:09] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.47, 11.08, 10.67 [10:18:09] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.23, 7.85, 7.18 [10:18:28] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.44, 10.42, 9.69 [10:18:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.46, 9.54, 9.57 [10:20:05] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.23, 7.04, 6.95 [10:20:23] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.61, 9.96, 9.62 [10:23:38] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [10:24:14] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.24, 10.28, 9.89 [10:25:27] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.738 second response time [10:25:27] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.895 second response time [10:25:36] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.523 second response time [10:25:38] PROBLEM - db131 MariaDB on db131 is CRITICAL: Can't connect to server on 'db131.miraheze.org' (115) [10:25:39] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.899 second response time [10:25:43] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.703 second response time [10:25:52] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.737 second response time [10:26:09] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.07, 8.69, 9.35 [10:26:14] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.776 second response time [10:26:15] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.868 second response time [10:26:16] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.906 second response time [10:26:19] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.858 second response time [10:26:27] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 1.059 second response time [10:26:35] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.913 second response time [10:26:56] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.668 second response time [10:26:57] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.749 second response time [10:27:00] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 3.34, 2.84 [10:27:01] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 978 bytes in 0.693 second response time [10:27:16] PROBLEM - db131 MariaDB Connections on db131 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [10:27:16] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [10:27:49] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [10:27:50] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.42, 6.51, 6.78 [10:28:08] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.70, 9.12, 10.16 [10:29:22] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 59% [10:29:46] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.477 second response time [10:30:10] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [10:31:21] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [10:31:27] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.07, 4.97, 3.27 [10:32:01] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.58 ms [10:34:02] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.04 ms [10:34:45] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 48% [10:37:01] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [10:38:34] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 33% [10:40:56] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 48% [10:42:38] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.53, 3.75, 2.70 [10:44:37] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.29, 3.14, 2.61 [10:46:52] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 26% [10:50:52] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 52% [10:57:39] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.08, 3.41, 3.93 [11:00:52] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [11:02:52] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 53% [11:03:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.99, 6.02, 5.64 [11:03:38] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.92, 2.36, 3.29 [11:03:43] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 50% [11:04:52] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 32% [11:05:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.22, 5.63, 5.54 [11:05:43] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 34% [11:08:52] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 50% [11:10:40] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.45, 3.41, 3.52 [11:10:52] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 28% [11:12:40] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.09, 3.01, 3.36 [11:14:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.62, 3.55, 4.00 [11:16:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.70, 3.97, 4.10 [11:17:43] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 43% [11:19:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.95, 6.96, 6.05 [11:21:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.40, 5.64, 5.68 [11:22:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.94, 3.83, 3.99 [11:23:43] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 39% [11:27:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.70, 6.84, 6.14 [11:29:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.22, 6.27, 6.02 [11:29:43] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 43% [11:30:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.72, 3.85 [11:31:43] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 37% [11:38:15] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 42% [11:38:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.47, 3.85, 3.94 [11:39:43] PROBLEM - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is WARNING: WARNING - NGINX Error Rate is 53% [11:40:12] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 32% [11:42:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.79, 3.88 [11:43:43] RECOVERY - cp35 HTTP 4xx/5xx ERROR Rate on cp35 is OK: OK - NGINX Error Rate is 30% [11:44:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.24, 3.53, 3.77 [11:47:16] RECOVERY - db131 MariaDB Connections on db131 is OK: OK connection usage: 16.8%Current connections: 84 [11:47:24] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.442 second response time [11:47:35] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.120 second response time [11:47:39] RECOVERY - db131 MariaDB on db131 is OK: Uptime: 53 Threads: 66 Questions: 19628 Slow queries: 0 Opens: 685 Open tables: 679 Queries per second avg: 370.339 [11:47:43] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.926 second response time [11:48:27] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.482 second response time [11:48:33] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.295 second response time [11:48:43] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.185 second response time [11:48:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/721616f37e13 [11:48:46] [02miraheze/puppet] 07paladox 03721616f - Fix mariadb config for mariadb 10.5 [11:48:48] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [11:48:50] [02puppet] 07paladox opened pull request 03#3687: Fix mariadb config for mariadb 10.5 - 13https://github.com/miraheze/puppet/pull/3687 [11:48:56] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.967 second response time [11:49:13] [02puppet] 07paladox closed pull request 03#3687: Fix mariadb config for mariadb 10.5 - 13https://github.com/miraheze/puppet/pull/3687 [11:49:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [11:49:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1046a369e442...e82c1f448f1c [11:49:19] [02miraheze/puppet] 07paladox 03e82c1f4 - Fix mariadb config for mariadb 10.5 (#3687) [11:49:20] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [11:49:37] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.937 second response time [11:49:47] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.429 second response time [11:49:56] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.089 second response time [11:50:09] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 54% [11:50:21] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.205 second response time [11:50:23] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.957 second response time [11:50:55] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.054 second response time [11:50:58] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.52, 2.80, 3.34 [11:51:16] PROBLEM - mw132 Disk Space on mw132 is WARNING: DISK WARNING - free space: / 2607MiB (10% inode=69%); [11:51:21] PROBLEM - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is WARNING: WARNING - NGINX Error Rate is 51% [11:52:10] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 33% [11:53:21] RECOVERY - cp24 HTTP 4xx/5xx ERROR Rate on cp24 is OK: OK - NGINX Error Rate is 31% [11:53:26] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.806 second response time [11:55:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.71, 7.77, 6.14 [11:56:56] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [11:58:54] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.600 second response time [12:05:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.19, 7.40, 6.94 [12:05:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e82c1f448f1c...9a4b6d4467f7 [12:06:00] [02miraheze/puppet] 07paladox 039a4b6d4 - dns: use hint-file in pdns-recursor config [12:07:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 8.15, 7.29 [12:09:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.13, 7.48, 7.16 [12:11:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.75, 7.65, 7.24 [12:11:30] PROBLEM - swiftac171 NTP time on swiftac171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.58, 7.69, 7.31 [12:15:12] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/3f8003973176...36df915f0f53 [12:15:15] [02miraheze/CreateWiki] 07translatewiki 0336df915 - Localisation updates from https://translatewiki.net. [12:15:17] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/landing/compare/e88412990cc6...7c62d896b5e2 [12:15:18] [02miraheze/landing] 07translatewiki 037c62d89 - Localisation updates from https://translatewiki.net. [12:15:19] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/MirahezeMagic/compare/bcc26dca03f1...1ba4d2728be9 [12:15:22] [02miraheze/MirahezeMagic] 07translatewiki 031ba4d27 - Localisation updates from https://translatewiki.net. [12:15:24] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±17] 13https://github.com/miraheze/ManageWiki/compare/8f27ba5b9ad3...3cd78164a5fc [12:15:27] [02miraheze/ManageWiki] 07translatewiki 033cd7816 - Localisation updates from https://translatewiki.net. [12:16:08] miraheze/landing - translatewiki the build passed. [12:17:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.67, 8.51, 7.62 [12:19:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.94, 7.36, 7.31 [12:20:04] miraheze/ManageWiki - translatewiki the build passed. [12:20:44] miraheze/MirahezeMagic - translatewiki the build passed. [12:21:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.68, 7.97, 7.54 [12:21:04] miraheze/CreateWiki - translatewiki the build passed. [12:23:38] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [12:23:59] !log [@test151] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to test151 [12:24:00] !log [@test151] finished deploy of {'folders': '1.40/extensions/MirahezeMagic'} to test151 - SUCCESS in 1s [12:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:09] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:24:10] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 1s [12:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:18] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:19] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 1s [12:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:36] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.667 second response time [12:26:58] RECOVERY - swiftac171 NTP time on swiftac171 is OK: NTP OK: Offset 5.140900612e-05 secs [12:30:36] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [12:31:07] PROBLEM - cp41 Disk Space on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:32:32] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.321 second response time [12:33:03] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 7960MiB (8% inode=98%); [12:33:17] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 3.15, 6.24, 7.48 [12:33:52] !log [@mwtask181] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all [12:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:55] !log [@mwtask181] finished deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all - SUCCESS in 62s [12:35:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:35:02] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:35:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:35:13] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.91, 7.35, 7.72 [12:36:03] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 61s [12:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:09] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.77, 7.39, 7.68 [12:39:39] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 209.54 ms [12:41:39] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 145.16 ms [12:46:21] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:48:20] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.617 second response time [12:49:49] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.88, 10.24, 8.74 [12:49:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.74, 9.41, 7.77 [12:51:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.84, 6.23, 6.80 [12:52:33] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.46, 3.19, 2.99 [12:52:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:52:58] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.49, 10.48, 8.95 [12:53:47] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.20, 9.63, 8.93 [12:53:50] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.49, 8.86, 8.04 [12:54:31] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.47, 3.11 [12:55:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.05, 6.97, 6.97 [12:56:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.78, 2.85, 2.93 [12:56:47] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.21, 9.65, 8.97 [12:57:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.25, 6.78, 6.88 [12:58:56] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [12:59:13] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:00:54] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.594 second response time [13:01:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.03, 7.85, 7.28 [13:03:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.83, 7.39, 7.17 [13:03:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/602681250dfe [13:03:15] [02miraheze/puppet] 07paladox 036026812 - dns: set query-local-address to :: [13:03:18] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:03:20] [02puppet] 07paladox opened pull request 03#3688: dns: set query-local-address to :: - 13https://github.com/miraheze/puppet/pull/3688 [13:06:24] [02puppet] 07paladox edited pull request 03#3688: dns: set query-local-address to :: - 13https://github.com/miraheze/puppet/pull/3688 [13:06:35] [02puppet] 07paladox closed pull request 03#3688: dns: set query-local-address to :: - 13https://github.com/miraheze/puppet/pull/3688 [13:06:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9a4b6d4467f7...e1ced62f411e [13:06:40] [02miraheze/puppet] 07paladox 03e1ced62 - dns: set query-local-address to :: (#3688) [13:06:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [13:06:43] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:07:03] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.52, 8.64, 7.68 [13:08:37] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.64, 10.71, 9.60 [13:10:16] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.82, 3.67, 3.16 [13:11:10] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.63, 11.14, 9.84 [13:12:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.80, 2.99, 2.97 [13:12:35] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.93, 9.26, 9.32 [13:13:05] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.97, 9.58, 9.42 [13:21:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.35, 7.08, 7.82 [13:22:26] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:28:18] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:29:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.97, 8.17, 7.98 [13:32:43] RECOVERY - swiftobject171 Swift Object Service on swiftobject171 is OK: TCP OK - 0.000 second response time on 10.0.17.126 port 6000 [13:34:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e1ced62f411e...c46d8aa0bbff [13:34:30] [02miraheze/puppet] 07paladox 03c46d8aa - Phabricator: fix monitoring using private ip [13:38:53] RECOVERY - phorge171 phab.miraheze.wiki HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17685 bytes in 0.052 second response time [13:38:59] RECOVERY - phorge171 phabricator.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 20296 bytes in 0.054 second response time [13:41:59] RECOVERY - db171 MariaDB on db171 is OK: Uptime: 146 Threads: 2 Questions: 117 Slow queries: 0 Opens: 32 Open tables: 26 Queries per second avg: 0.801 [13:43:16] RECOVERY - db171 MariaDB Connections on db171 is OK: OK connection usage: 0.4%Current connections: 2 [13:43:55] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:44:14] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [13:45:56] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.887 second response time [13:46:11] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.576 second response time [13:46:16] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.54, 3.33, 2.70 [13:48:05] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.37, 11.46, 10.07 [13:48:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.63, 11.29, 9.84 [13:48:15] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.42, 3.03, 2.67 [13:48:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.54, 9.92, 8.85 [13:48:24] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.82, 9.85, 8.71 [13:49:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:50:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.05, 11.29, 10.19 [13:50:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.45, 11.47, 10.10 [13:50:20] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.66, 9.67, 8.88 [13:50:21] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.86, 9.45, 8.71 [13:51:16] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 16.67, 11.89, 8.76 [13:51:37] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.97, 11.03, 8.17 [13:51:47] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:52:14] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.59, 3.81, 3.06 [13:53:12] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.45, 11.29, 8.96 [13:53:12] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.665 second response time [13:53:29] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 3.66, 2.99 [13:53:31] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 9.23, 9.86, 8.08 [13:53:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.677 second response time [13:54:03] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.16, 12.22, 10.81 [13:54:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.47, 12.31, 10.68 [13:54:14] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.76, 3.74, 3.12 [13:54:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.85, 11.75, 9.91 [13:55:08] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 11.89, 12.38, 9.67 [13:55:26] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.19, 6.66, 7.83 [13:56:20] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.02, 13.93, 10.90 [13:56:47] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:03] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.11, 10.38, 9.26 [13:57:26] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.97, 7.57, 8.02 [13:58:12] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [13:58:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.69, 5.25, 3.82 [13:58:54] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.958 second response time [13:59:02] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 18.88, 13.12, 10.34 [13:59:09] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:59:14] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:59:17] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.13, 3.71, 3.27 [13:59:25] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.42, 7.49, 7.94 [13:59:26] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:59:34] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:00:09] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.544 second response time [14:00:15] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.48, 11.64, 9.49 [14:00:55] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 220.75 ms [14:00:58] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.22, 10.65, 9.76 [14:01:04] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.849 second response time [14:01:13] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 3.86, 3.37 [14:01:22] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 9.088 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [14:01:32] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.548 second response time [14:02:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.01, 10.08, 9.16 [14:02:11] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.06, 11.39, 11.07 [14:02:12] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.44, 11.07, 9.93 [14:02:54] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.56, 8.91, 9.23 [14:02:58] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 215.87 ms [14:03:23] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.25, 7.93, 8.00 [14:04:48] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.38, 2.65 [14:05:02] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 169.53 ms [14:05:22] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.70, 7.94, 8.00 [14:06:06] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.92, 11.79, 10.41 [14:06:11] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.24, 14.23, 12.20 [14:07:21] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.35, 7.86, 7.95 [14:08:27] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.48, 3.91, 3.90 [14:08:47] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.60, 3.71, 2.98 [14:10:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.45, 11.98, 11.97 [14:11:19] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.96, 7.79, 7.92 [14:12:46] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.95, 3.03, 2.89 [14:13:55] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.02, 11.39, 10.93 [14:14:31] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.84, 2.77, 3.36 [14:15:08] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.13, 3.87, 3.88 [14:15:59] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.19, 10.92, 11.82 [14:16:12] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.97, 11.92, 11.98 [14:16:25] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 277.11 ms [14:17:17] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.56, 7.97, 7.76 [14:18:12] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.20, 12.08, 12.04 [14:18:30] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.52 ms [14:19:08] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.24, 3.96, 3.92 [14:20:12] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.20, 11.62, 11.86 [14:20:33] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 151.77 ms [14:21:09] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.76, 3.97, 3.93 [14:21:56] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:22:09] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:22:12] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:22:33] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.09, 4.27, 3.64 [14:22:54] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.24, 9.17, 7.80 [14:23:14] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.60, 7.62, 7.77 [14:23:15] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.20, 10.13, 8.61 [14:23:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.62, 11.33, 11.06 [14:24:05] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [14:24:06] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [14:24:09] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.684 second response time [14:24:09] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.725 second response time [14:24:53] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.88, 10.49, 8.42 [14:25:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.36, 10.03, 8.80 [14:25:57] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 12.36, 11.62 [14:26:01] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.520 second response time [14:26:05] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.932 second response time [14:26:09] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.608 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [14:26:13] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.96, 12.86, 12.22 [14:26:22] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.69, 11.82, 11.25 [14:26:52] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.98, 10.32, 8.63 [14:28:52] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.74, 8.86, 8.28 [14:28:54] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [14:29:33] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.26, 11.37, 11.29 [14:30:13] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.72, 11.60, 11.89 [14:30:37] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:30:53] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:30:56] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.53, 11.20, 9.36 [14:31:02] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:31:08] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 4.04, 3.87 [14:31:10] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.45, 7.55 [14:31:28] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.77, 11.74, 9.46 [14:31:30] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.59, 11.73, 11.40 [14:32:14] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 18.33, 13.83, 12.65 [14:32:46] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 9.834 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [14:32:48] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.783 second response time [14:32:51] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.16, 10.41, 9.29 [14:32:58] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.353 second response time [14:34:53] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:34:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.75, 10.40, 9.37 [14:35:04] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:36:48] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.153 second response time [14:37:03] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 4.387 second response time [14:37:18] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:38:07] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw141 [14:39:23] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 5.971 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [14:40:04] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 20 backends are healthy [14:40:45] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.19, 10.53, 9.78 [14:40:49] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.50, 11.86, 10.68 [14:42:45] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.20, 9.05, 9.32 [14:43:07] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 6.56, 10.43, 10.67 [14:44:49] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.22, 9.87, 10.14 [14:45:03] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.82, 9.10, 10.15 [14:45:06] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.47, 7.79, 7.96 [14:46:50] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 15.08, 9.72, 8.60 [14:48:10] PROBLEM - db161 MariaDB Connections on db161 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [14:48:10] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db161.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [14:48:30] PROBLEM - db161 conntrack_table_size on db161 is UNKNOWN: NRPE: Unable to read output [14:48:41] PROBLEM - db161 MariaDB on db161 is CRITICAL: Can't connect to server on 'db161.wikitide.net' (115) [14:48:48] PROBLEM - bast161 Puppet on bast161 is UNKNOWN: NRPE: Unable to read output [14:48:48] PROBLEM - bast161 conntrack_table_size on bast161 is UNKNOWN: NRPE: Unable to read output [14:48:52] PROBLEM - db161 Backups SQL mhglobal on db161 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-mhglobal-backup-weekly.log [14:49:04] PROBLEM - db161 Backups SQL on db161 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [14:49:36] PROBLEM - bast161 ferm_active on bast161 is UNKNOWN: NRPE: Unable to read output [14:49:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:50:07] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 42% [14:50:25] RECOVERY - db161 conntrack_table_size on db161 is OK: OK: nf_conntrack is 0 % full [14:50:42] RECOVERY - bast161 conntrack_table_size on bast161 is OK: OK: nf_conntrack is 0 % full [14:50:44] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [14:51:32] RECOVERY - bast161 ferm_active on bast161 is OK: OK ferm input default policy is set [14:51:43] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.11, 2.65, 3.66 [14:52:03] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 19% [14:53:43] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.48, 1.91, 3.27 [14:56:37] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.35 ms [14:56:48] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.87, 7.19, 7.86 [14:58:36] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 195.85 ms [14:58:47] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.66, 7.84, 8.04 [14:59:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/243caa0eed71 [14:59:48] [02miraheze/puppet] 07paladox 03243caa0 - Install os161 [14:59:51] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:59:53] [02puppet] 07paladox opened pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:01:45] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/fd7c5478cb97 [15:01:46] [02miraheze/dns] 07paladox 03fd7c547 - Add os151-private/os161-private [15:01:47] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [15:01:48] [02dns] 07paladox opened pull request 03#468: Add os151-private/os161-private - 13https://github.com/miraheze/dns/pull/468 [15:01:58] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.63, 11.02, 11.99 [15:02:14] [02dns] 07paladox closed pull request 03#468: Add os151-private/os161-private - 13https://github.com/miraheze/dns/pull/468 [15:02:15] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [15:02:18] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/43b3e94115a8...f5cf3f545cf2 [15:02:21] [02miraheze/dns] 07paladox 03f5cf3f5 - Add os151-private/os161-private (#468) [15:02:23] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [15:03:03] [02puppet] 07paladox synchronize pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:03:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/243caa0eed71...7ad22e2968ea [15:03:12] [02miraheze/puppet] 07paladox 037ad22e2 - Update os161.yaml [15:03:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7ad22e2968ea...b3bce3751c50 [15:03:44] [02miraheze/puppet] 07paladox 03b3bce37 - Update os151.yaml [15:03:46] [02puppet] 07paladox synchronize pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:04:45] PROBLEM - os161 ferm_active on os161 is UNKNOWN: NRPE: Unable to read output [15:04:47] PROBLEM - os161 conntrack_table_size on os161 is UNKNOWN: NRPE: Unable to read output [15:05:00] PROBLEM - os161 Puppet on os161 is UNKNOWN: NRPE: Unable to read output [15:05:14] [02puppet] 07paladox synchronize pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:05:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3bce3751c50...3d5a39292de9 [15:05:19] [02miraheze/puppet] 07paladox 033d5a392 - Update os151.yaml [15:05:24] [02puppet] 07paladox synchronize pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:05:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3d5a39292de9...633df02077e0 [15:05:27] [02miraheze/puppet] 07paladox 03633df02 - Update os161.yaml [15:05:54] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.18, 6.67, 5.66 [15:06:01] [02puppet] 07paladox closed pull request 03#3689: Install os161 - 13https://github.com/miraheze/puppet/pull/3689 [15:06:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [15:06:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/c46d8aa0bbff...6037a6a7f871 [15:06:07] [02miraheze/puppet] 07paladox 036037a6a - Install os161 (#3689) [15:06:09] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:06:42] RECOVERY - os161 ferm_active on os161 is OK: OK ferm input default policy is set [15:06:44] RECOVERY - os161 conntrack_table_size on os161 is OK: OK: nf_conntrack is 0 % full [15:07:50] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.41, 6.31, 5.65 [15:08:10] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.39, 11.00, 11.55 [15:08:37] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.67, 7.32, 7.81 [15:09:02] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:47] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 56% [15:11:42] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 16% [15:12:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/107dd9c58149 [15:12:25] [02miraheze/puppet] 07paladox 03107dd9c - Install graylog161 [15:12:27] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:12:30] [02puppet] 07paladox opened pull request 03#3690: Install graylog161 - 13https://github.com/miraheze/puppet/pull/3690 [15:12:34] [02puppet] 07paladox closed pull request 03#3690: Install graylog161 - 13https://github.com/miraheze/puppet/pull/3690 [15:12:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [15:12:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6037a6a7f871...1a46f6d2963d [15:12:37] [02miraheze/puppet] 07paladox 031a46f6d - Install graylog161 (#3690) [15:12:39] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:13:13] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.21, 11.00, 11.98 [15:13:57] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.23, 11.73, 11.75 [15:14:18] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.30, 11.00, 11.86 [15:14:25] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.39, 7.71 [15:15:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1a46f6d2963d...9757404129c6 [15:15:02] [02miraheze/puppet] 07paladox 039757404 - Create graylog161.yaml [15:15:52] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.15, 12.03, 11.86 [15:16:39] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 1 minute ago with 4 failures. Failed resources (up to 3 shown): Package[mongodb_server],Package[graylog-server],File[/etc/default/graylog-server] [15:17:47] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.69, 11.23, 11.57 [15:18:03] PROBLEM - graylog161 HTTPS on graylog161 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.006 second response time [15:18:15] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.02, 7.77, 7.84 [15:19:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 11.92, 11.77, 12.02 [15:19:48] [02dns] 07MacFan4000 opened pull request 03#469: add two cnames for lostidols.wiki - 13https://github.com/miraheze/dns/pull/469 [15:19:50] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:20:11] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.96, 8.65, 8.15 [15:24:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.11, 7.49, 7.80 [15:25:25] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.85, 11.46, 11.45 [15:25:59] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.74, 9.11, 8.35 [15:26:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9757404129c6...f5fc802767c2 [15:26:53] [02miraheze/puppet] 07paladox 03f5fc802 - graylog: add graylog-package-repository.s3.eu-west-1.amazonaws.com to package source [15:28:11] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.74, 11.60, 11.49 [15:28:59] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e847b9bb7bc1...cd5973c807e7 [15:29:02] [02miraheze/ssl] 07MacFan4000 03cd5973c - add removededmsongswiki redirect [15:29:52] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.36, 7.82, 7.97 [15:30:10] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.62, 11.28, 11.39 [15:31:47] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.07, 7.90, 7.97 [15:32:10] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.03, 11.70, 11.54 [15:33:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 3.80, 6.45, 7.44 [15:34:32] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:31] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.50, 6.26, 5.72 [15:35:57] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.86, 3.82, 2.46 [15:36:40] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.815 second response time [15:37:23] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [15:37:31] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.57, 5.61, 5.53 [15:37:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.55, 3.58, 2.54 [15:38:08] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.80, 10.87, 11.40 [15:38:47] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.88, 11.38, 11.76 [15:39:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.74, 7.47, 7.53 [15:39:56] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.33, 2.77, 2.36 [15:39:58] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [15:40:42] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.31, 12.48, 12.11 [15:41:19] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 10% [15:41:28] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.73, 7.01, 7.34 [15:42:05] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 8.30, 6.96, 5.16 [15:42:37] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.96, 11.90, 11.94 [15:42:45] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.26, 10.33, 11.86 [15:44:05] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.28, 6.08, 5.06 [15:44:07] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.02, 12.09, 11.67 [15:44:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.74, 12.71, 12.24 [15:45:14] !log [macfan@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/updateSpecialPages.php (START) [15:45:19] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.97, 7.34, 7.33 [15:45:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:06] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.91, 11.63, 11.56 [15:47:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.05, 7.00, 7.21 [15:47:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5fc802767c2...6fe9bf91eabe [15:47:17] [02miraheze/puppet] 07paladox 036fe9bf9 - mongodb: temporarily use bullseye dust [15:48:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6fe9bf91eabe...655e3eefbec4 [15:48:37] [02miraheze/puppet] 07paladox 03655e3ee - Revert "mongodb: temporarily use bullseye dust" [15:48:48] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=familiarlandswiki --no-updates --username-prefix=wikia:familiarlands /home/reception/ImportDump/familiarlandswiki-20240125154708.xml (START) [15:48:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:01] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.94, 11.58, 11.53 [15:50:36] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=familiarlandswiki --no-updates --username-prefix=wikia:familiarlands /home/reception/ImportDump/familiarlandswiki-20240125154708.xml (END - exit=0) [15:50:37] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=familiarlandswiki (START) [15:50:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:50:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:56] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.55, 11.26, 11.40 [15:52:08] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [15:52:45] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.28, 12.09, 11.94 [15:52:52] PROBLEM - cp41 Disk Space on cp41 is CRITICAL: DISK CRITICAL - free space: / 4475MiB (4% inode=98%); [15:52:56] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 207.33 ms [15:53:03] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 6.63, 7.55, 6.23 [15:53:52] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.16, 11.85, 11.59 [15:54:04] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 55% [15:54:05] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 20% [15:54:51] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.54, 6.20, 6.80 [15:54:57] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 159.19 ms [15:55:01] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 8.35, 7.67, 6.41 [15:55:42] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=familiarlandswiki (END - exit=0) [15:55:46] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=familiarlandswiki --active --update (END - exit=0) [15:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/67864020d3b8 [15:56:09] [02miraheze/puppet] 07paladox 036786402 - mongodb: support installing version 7 [15:56:10] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:56:11] [02puppet] 07paladox opened pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [15:56:45] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03paladox-patch-12 [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/67864020d3b8...821ea2eea764 [15:56:48] [02miraheze/puppet] 07github-actions 03821ea2e - CI: lint puppet code to standards [15:56:49] [02puppet] 07github-actions[bot] synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [15:56:58] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 4.63, 6.91, 6.32 [15:57:09] [02puppet] 07paladox synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [15:57:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/821ea2eea764...0500756beefe [15:57:13] [02miraheze/puppet] 07paladox 030500756 - Update apt.pp [15:57:57] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 37% [15:58:56] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.75, 6.28, 6.16 [15:59:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/0500756beefe...628c6579ce97 [15:59:38] [02miraheze/puppet] 07paladox 03628c657 - Add apt key [15:59:40] [02puppet] 07paladox synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [16:02:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/628c6579ce97...4f52e7aeb929 [16:02:37] [02miraheze/puppet] 07paladox 034f52e7a - Update graylog.pp [16:02:40] [02puppet] 07paladox synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [16:03:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4f52e7aeb929...16f9d240771b [16:03:03] [02miraheze/puppet] 07paladox 0316f9d24 - Update graylog161.yaml [16:03:04] [02puppet] 07paladox synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [16:04:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/655e3eefbec4...15c3ca5d23a6 [16:04:19] [02miraheze/puppet] 07paladox 0315c3ca5 - Lower cp41 storage size to 60g [16:06:07] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:08:43] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 14509MiB (15% inode=98%); [16:11:31] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3360 bytes in 0.988 second response time [16:12:36] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 41% [16:13:31] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.602 second response time [16:16:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.82, 6.83, 5.55 [16:16:29] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 21% [16:17:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.10, 6.46, 5.44 [16:18:09] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [16:19:06] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.55, 6.59, 5.62 [16:19:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.00, 5.91 [16:21:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.06, 6.57, 5.87 [16:28:30] [02miraheze/puppet] 07paladox pushed 037 commits to 03paladox-patch-12 [+2/-1/±8] 13https://github.com/miraheze/puppet/compare/16f9d240771b...934023454d7c [16:28:33] [02miraheze/puppet] 07paladox 03662436e - mongodb: support installing version 7 [16:28:34] [02miraheze/puppet] 07github-actions 032e3bfde - CI: lint puppet code to standards [16:28:36] [02miraheze/puppet] 07paladox 03b738291 - Update apt.pp [16:28:39] [02miraheze/puppet] ... and 4 more commits. [16:28:40] [02puppet] 07paladox synchronize pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [16:29:48] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [16:29:53] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:30:51] [02puppet] 07paladox closed pull request 03#3691: mongodb: support installing version 7 - 13https://github.com/miraheze/puppet/pull/3691 [16:30:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/puppet/compare/15c3ca5d23a6...c126d8b26933 [16:30:53] [02miraheze/puppet] 07paladox 03c126d8b - mongodb: support installing version 7 (#3691) [16:30:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:30:58] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:31:42] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:31:48] PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:12] PROBLEM - cp41 Nginx Backend for mw131 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:12] PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:14] PROBLEM - cp41 conntrack_table_size on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:15] PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:17] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:19] PROBLEM - cp41 Nginx Backend for puppet181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:23] PROBLEM - cp41 Nginx Backend for mwtask181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:24] PROBLEM - cp41 Nginx Backend for mw134 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:27] PROBLEM - cp41 Nginx Backend for mw133 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:35] PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:36] PROBLEM - cp41 Nginx Backend for mw152 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:37] PROBLEM - cp41 Nginx Backend for mw142 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:37] PROBLEM - cp41 Nginx Backend for mw182 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:38] PROBLEM - cp41 Nginx Backend for mw141 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:38] PROBLEM - cp41 NTP time on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:41] PROBLEM - Host cp41 is DOWN: PING CRITICAL - Packet loss = 100% [16:39:56] PROBLEM - mem161 ferm_active on mem161 is UNKNOWN: NRPE: Unable to read output [16:40:08] PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: Connection refused [16:40:43] PROBLEM - mw161 Current Load on mw161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:50] PROBLEM - mw162 php-fpm on mw162 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:50] PROBLEM - mw161 ferm_active on mw161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:50] PROBLEM - mw161 Disk Space on mw161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/379ac1ae94fa [16:40:52] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:40:52] PROBLEM - mw161 APT on mw161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:40:53] [02miraheze/puppet] 07paladox 03379ac1a - Add mw161/162 [16:40:55] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:40:57] [02puppet] 07paladox opened pull request 03#3692: Add mw161/162 - 13https://github.com/miraheze/puppet/pull/3692 [16:41:01] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 10219MiB (10% inode=98%); [16:41:06] PROBLEM - mw162 Current Load on mw162 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:13] PROBLEM - mw162 ferm_active on mw162 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:41:19] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw161.wikitide.net port 443 after 3025 ms: Couldn't connect to server [16:41:19] PROBLEM - mw161 Puppet on mw161 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [16:41:21] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: connect to address 10.0.16.133 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:41:23] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: connect to address 10.0.16.132 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:41:23] PROBLEM - mw162 APT on mw162 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [16:41:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 4.29, 3.19 [16:41:29] PROBLEM - mw161 conntrack_table_size on mw161 is UNKNOWN: NRPE: Unable to read output [16:41:32] PROBLEM - mw162 conntrack_table_size on mw162 is UNKNOWN: NRPE: Unable to read output [16:41:36] PROBLEM - mw162 Puppet on mw162 is UNKNOWN: NRPE: Unable to read output [16:41:40] PROBLEM - mw161 php-fpm on mw161 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [16:41:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/379ac1ae94fa...8e20d4b00314 [16:41:44] [02miraheze/puppet] 07paladox 038e20d4b - Update common.yaml [16:41:48] [02puppet] 07paladox synchronize pull request 03#3692: Add mw161/162 - 13https://github.com/miraheze/puppet/pull/3692 [16:41:51] RECOVERY - mem161 ferm_active on mem161 is OK: OK ferm input default policy is set [16:41:54] [02puppet] 07paladox closed pull request 03#3692: Add mw161/162 - 13https://github.com/miraheze/puppet/pull/3692 [16:41:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/c126d8b26933...9afdf17c89f6 [16:41:58] [02miraheze/puppet] 07paladox 039afdf17 - Add mw161/162 (#3692) [16:42:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [16:42:01] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [16:42:20] miraheze/puppet - paladox the build passed. [16:42:33] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.90, 3.61, 2.93 [16:42:42] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 0.37, 0.46, 0.25 [16:42:45] RECOVERY - mw161 Disk Space on mw161 is OK: DISK OK - free space: / 49837MiB (93% inode=96%); [16:42:46] PROBLEM - mw161 ferm_active on mw161 is UNKNOWN: NRPE: Unable to read output [16:42:48] RECOVERY - mw161 APT on mw161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:43:03] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 0.27, 0.49, 0.27 [16:43:08] PROBLEM - mw162 ferm_active on mw162 is UNKNOWN: NRPE: Unable to read output [16:43:11] miraheze/puppet - paladox the build passed. [16:43:24] RECOVERY - mw162 APT on mw162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:43:26] RECOVERY - mw161 conntrack_table_size on mw161 is OK: OK: nf_conntrack is 0 % full [16:43:35] RECOVERY - mw162 conntrack_table_size on mw162 is OK: OK: nf_conntrack is 0 % full [16:43:58] RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211 [16:44:18] miraheze/puppet - paladox the build passed. [16:44:57] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw161 mw162 [16:45:22] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw161 mw162 [16:45:25] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw161 mw162 [16:45:41] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw161 mw162 [16:45:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw161 mw162 [16:46:44] RECOVERY - mw162 php-fpm on mw162 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [16:46:55] RECOVERY - mw162 ferm_active on mw162 is OK: OK ferm input default policy is set [16:47:06] RECOVERY - mw161 ferm_active on mw161 is OK: OK ferm input default policy is set [16:47:10] RECOVERY - Host cp41 is UP: PING OK - Packet loss = 0%, RTA = 152.27 ms [16:47:29] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw161 mw162 [16:47:35] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.494 second response time [16:47:35] RECOVERY - cp41 Nginx Backend for matomo151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8203 [16:47:35] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:47:38] RECOVERY - mw161 php-fpm on mw161 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [16:47:40] RECOVERY - cp41 Nginx Backend for mw134 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8111 [16:47:41] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 4 seconds ago with 3 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Service[cgroup] [16:47:45] RECOVERY - cp41 Nginx Backend for puppet181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8204 [16:47:50] RECOVERY - cp41 Nginx Backend for mw133 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8110 [16:47:50] RECOVERY - cp41 Nginx Backend for mwtask181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8160 [16:47:50] RECOVERY - cp41 SSH on cp41 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [16:48:00] RECOVERY - cp41 Nginx Backend for test151 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8181 [16:48:05] RECOVERY - cp41 Nginx Backend for mw142 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8109 [16:48:15] RECOVERY - cp41 Nginx Backend for mw141 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8108 [16:48:20] RECOVERY - cp41 NTP time on cp41 is OK: NTP OK: Offset -0.009651273489 secs [16:48:20] RECOVERY - cp41 Nginx Backend for mw152 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8114 RECOVERY - cp41 Nginx Backend for mw182 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8118 [16:48:24] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [16:48:25] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 45 seconds ago with 3 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Service[cgroup] [16:48:30] RECOVERY - cp41 conntrack_table_size on cp41 is OK: OK: nf_conntrack is 6 % full [16:48:35] RECOVERY - cp41 Nginx Backend for phorge171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8202 [16:48:40] RECOVERY - cp41 Nginx Backend for mw131 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8106 [16:48:42] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:48:43] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [16:48:49] PROBLEM - mw162 MediaWiki Rendering on mw162 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [16:48:57] PROBLEM - mw162 HTTPS on mw162 is WARNING: HTTP WARNING: HTTP/2 403 - 146 bytes in 0.006 second response time [16:48:59] PROBLEM - mw161 MediaWiki Rendering on mw161 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [16:49:00] RECOVERY - cp41 PowerDNS Recursor on cp41 is OK: DNS OK: 0.052 seconds response time. miraheze.org returns 109.123.230.163,2400:d320:2161:9775::1 [16:49:01] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.52, 4.31, 3.41 [16:49:10] PROBLEM - mw161 HTTPS on mw161 is WARNING: HTTP WARNING: HTTP/2 403 - 146 bytes in 0.006 second response time [16:49:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.10 ms [16:49:17] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [16:49:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [16:49:28] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [16:49:36] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [16:49:36] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [16:50:18] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.85, 3.81, 3.59 [16:50:38] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.716 second response time [16:50:42] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.549 second response time [16:51:37] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [16:52:11] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [16:52:13] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.74, 2.80, 3.25 [16:52:32] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.13, 10.56, 11.94 [16:53:47] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw161, mw162] [16:53:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:08] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.576 second response time [16:54:34] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': '1.40'} to [mw161, mw162] - SUCCESS in 46s [16:54:47] PROBLEM - mw162 MediaWiki Rendering on mw162 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.053 second response time [16:54:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:51] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw161, mw162] [16:54:55] PROBLEM - mw161 MediaWiki Rendering on mw161 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.051 second response time [16:55:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:10] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 7.27, 5.85, 4.87 [16:55:32] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 28% [16:55:43] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw161, mw162] - SUCCESS in 51s [16:55:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:44] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw161, mw162] [16:56:57] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:03] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:57:09] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.33, 5.11, 4.71 [16:57:38] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40', '1.41']} to [mw161, mw162] - SUCCESS in 53s [16:57:38] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:57:44] [1/2] I’ll teach 15 individuals to earn $50k or more within 72 hours but you will pay me 10% of your profit when you receive it. Note only interested people should apply, drop a message let's get started by asking (HOW) [16:57:44] [2/2] ‪Telegram: Carolinamason00 [16:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:16] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [16:58:26] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.47, 12.37, 12.19 [16:59:01] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.561 second response time [16:59:07] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.530 second response time [17:00:52] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.83, 3.80, 3.79 [17:04:04] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [17:04:35] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [17:04:40] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:05:59] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.513 second response time [17:06:30] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.496 second response time [17:06:35] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3481 bytes in 0.790 second response time [17:07:27] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:07:30] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.78, 9.78, 11.74 [17:08:16] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.03, 11.04, 11.91 [17:10:13] PROBLEM - swiftobject161 NTP time on swiftobject161 is WARNING: NTP WARNING: Offset 0.1269915402 secs [17:10:20] PROBLEM - swiftproxy161 NTP time on swiftproxy161 is WARNING: NTP WARNING: Offset -0.1193463206 secs [17:10:38] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw162 [17:11:16] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.72, 3.72 [17:11:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.17, 10.92, 11.59 [17:11:21] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [17:11:28] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:11:40] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [17:11:48] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [17:11:53] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f5cf3f545cf2...6e5905a4a9d2 [17:11:55] [02miraheze/dns] 07paladox 036e5905a - Add bast181 ipv4 address [17:12:08] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.56, 11.65, 11.85 [17:12:14] RECOVERY - swiftobject161 NTP time on swiftobject161 is OK: NTP OK: Offset -0.0009850859642 secs [17:12:21] RECOVERY - swiftproxy161 NTP time on swiftproxy161 is OK: NTP OK: Offset -0.001426130533 secs [17:13:16] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3459 bytes in 0.887 second response time [17:13:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.65, 10.10, 11.24 [17:13:24] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.610 second response time [17:13:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d884a2b64b48 [17:13:36] [02miraheze/puppet] 07paladox 03d884a2b - Switch http proxy to bastion.wikitide.net [17:13:39] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [17:13:40] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [17:13:41] [02puppet] 07paladox opened pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:13:47] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.742 second response time [17:14:03] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.48, 10.48, 11.41 [17:14:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d884a2b64b48...880151ac3456 [17:14:12] [02miraheze/puppet] 07paladox 03880151a - Update .env.erb [17:14:15] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:14:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/880151ac3456...1a961aee1ba5 [17:14:30] [02miraheze/puppet] 07paladox 031a961ae - Update shared.pp [17:14:31] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:14:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1a961aee1ba5...c549b74c7d80 [17:14:45] [02miraheze/puppet] 07paladox 03c549b74 - Update init.pp [17:14:48] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:15:00] miraheze/puppet - paladox the build passed. [17:15:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c549b74c7d80...7c9e5e2c6fbc [17:15:07] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [17:15:07] [02miraheze/puppet] 07paladox 037c9e5e2 - Update ssl-certificate.py [17:15:10] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:15:12] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:15:19] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [17:15:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c9e5e2c6fbc...6ffab2e33e44 [17:15:23] [02miraheze/puppet] 07paladox 036ffab2e - Update db112.yaml [17:15:24] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:15:39] miraheze/puppet - paladox the build passed. [17:15:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6ffab2e33e44...41e525adf49e [17:15:53] miraheze/puppet - paladox the build passed. [17:15:55] [02miraheze/puppet] 07paladox 0341e525a - Update extensionsetup.pp [17:15:58] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:16:09] miraheze/puppet - paladox the build passed. [17:16:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/41e525adf49e...b4356977c45c [17:16:16] [02miraheze/puppet] 07paladox 03b435697 - Update mwdeploy.py [17:16:17] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:16:24] miraheze/puppet - paladox the build passed. [17:17:03] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.580 second response time [17:17:10] miraheze/puppet - paladox the build passed. [17:17:12] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.564 second response time [17:17:14] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3459 bytes in 0.917 second response time [17:17:19] miraheze/puppet - paladox the build passed. [17:17:20] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.63, 10.29, 10.97 [17:17:43] miraheze/puppet - paladox the build passed. [17:17:51] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.88, 11.95, 11.76 [17:19:04] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.13, 10.56, 11.49 [17:19:17] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.55, 3.90, 3.88 [17:19:20] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.52, 9.51, 10.59 [17:19:46] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.54, 10.67, 11.32 [17:20:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+2/-0/±46] 13https://github.com/miraheze/puppet/compare/b4356977c45c...a4de3cf886e4 [17:20:36] [02miraheze/puppet] 07paladox 03a4de3cf - Replace all [17:20:42] [02puppet] 07paladox synchronize pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:20:57] [02puppet] 07paladox closed pull request 03#3693: Switch http proxy to bastion.wikitide.net - 13https://github.com/miraheze/puppet/pull/3693 [17:20:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±52] 13https://github.com/miraheze/puppet/compare/9afdf17c89f6...3d5d49c699de [17:21:02] [02miraheze/puppet] 07paladox 033d5d49c - Switch http proxy to bastion.wikitide.net (#3693) [17:21:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [17:21:08] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [17:21:17] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.28, 4.45, 4.08 [17:21:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/3d5d49c699de...172c745447f9 [17:21:34] [02miraheze/puppet] 07paladox 03172c745 - Delete modules/role/.!61911!.DS_Store [17:21:40] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.70, 12.06, 11.75 [17:22:00] miraheze/puppet - paladox the build passed. [17:22:47] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:22:58] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:22:58] miraheze/puppet - paladox the build passed. [17:23:04] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.38, 12.13, 11.85 [17:23:06] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [17:23:07] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [17:23:12] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [17:23:20] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.65, 11.88, 11.25 [17:23:21] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.26, 3.97, 3.95 [17:24:47] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.454 second response time [17:24:54] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.561 second response time [17:25:01] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.527 second response time [17:25:01] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3481 bytes in 0.670 second response time [17:25:08] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.675 second response time [17:26:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/addf8c2b7d04...d6606c575082 [17:26:07] [02miraheze/mw-config] 07paladox 03d6606c5 - Add mw161/162/171/172 to wgLinterSubmitterWhitelist [17:27:19] miraheze/mw-config - paladox the build passed. [17:28:16] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:28:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 6.86, 6.35 [17:29:23] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 67s [17:29:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d6606c575082...a425dc1befad [17:30:53] [02miraheze/mw-config] 07paladox 03a425dc1 - Switch http proxy to bastion.wikitide.net [17:31:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.38, 6.26, 6.19 [17:31:11] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/559085191e39...aac51b326d9e [17:31:14] [02miraheze/phabricator-extensions] 07paladox 03aac51b3 - Switch http proxy to bastion.wikitide.net [17:31:32] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:31:51] miraheze/mw-config - paladox the build passed. [17:32:00] miraheze/phabricator-extensions - paladox the build passed. [17:32:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:59] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 86s [17:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/172c745447f9...62cd97fcdad8 [17:34:56] [02miraheze/puppet] 07paladox 0362cd97f - Update account-server.conf.erb [17:35:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/62cd97fcdad8...be8d8bef26b0 [17:35:03] [02miraheze/puppet] 07paladox 03be8d8be - Update container-server.conf.erb [17:37:05] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.67, 11.44, 12.00 [17:37:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 3.39, 2.83 [17:37:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 56% [17:39:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.25, 11.01, 11.94 [17:39:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.68, 11.03, 11.95 [17:39:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.02, 3.27, 2.86 [17:40:21] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.16 ms [17:42:22] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 236.90 ms [17:43:05] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.32, 11.90, 11.90 [17:43:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.20, 11.57, 11.96 [17:43:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.98, 2.72, 3.24 [17:44:22] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 158.40 ms [17:45:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.74, 11.46, 11.86 [17:45:56] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 37% [17:47:05] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.27, 11.75, 11.86 [17:49:24] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.90, 10.23, 11.80 [17:50:15] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 10.84, 8.24, 6.81 [17:50:31] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [17:51:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.05, 11.39, 11.57 [17:51:18] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.16, 11.75, 11.68 [17:51:23] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.27, 11.78, 12.17 [17:51:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 40% [17:52:29] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.507 second response time [17:53:05] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.29, 11.56, 11.64 [17:53:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.52, 11.60, 11.65 [17:53:56] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 8% [17:54:04] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 6.05, 7.37, 6.77 [17:55:06] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.14, 11.25, 11.51 [17:55:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.20, 12.91, 12.09 [17:57:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.96, 11.22, 11.55 [17:59:49] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 9.16, 8.35, 7.37 [18:03:22] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.70, 11.21, 12.00 [18:07:07] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.41, 8.74, 10.17 [18:07:16] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 51% [18:07:17] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.65, 10.31, 11.59 [18:07:37] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 5.01, 7.78, 7.79 [18:08:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 6.71, 6.13 [18:09:13] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 24% [18:11:06] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.40, 8.64, 9.87 [18:12:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.45, 5.87, 5.94 [18:13:37] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 3.50, 5.40, 6.75 [18:15:16] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.04, 8.03, 9.93 [18:15:20] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.18, 8.89, 10.12 [18:21:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 6.52, 6.13 [18:23:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.22, 6.25, 6.07 [18:25:15] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 10.55, 10.16 [18:25:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 44% [18:27:07] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.27, 10.37, 9.66 [18:27:15] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.84, 10.16, 10.08 [18:27:19] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.19, 10.59, 10.19 [18:27:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [18:29:18] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.17, 11.05, 10.40 [18:29:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 56% [18:31:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.76, 11.02, 10.49 [18:31:18] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.79, 11.10, 10.51 [18:31:56] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 21% [18:31:57] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03bastion [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/339eadcb53d8 [18:32:00] [02miraheze/phabricator-extensions] 07Universal-Omega 03339eadc - Switch bastion host [18:32:02] [02phabricator-extensions] 07Universal-Omega created branch 03bastion - 13https://github.com/miraheze/phabricator-extensions [18:32:43] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03bastion [18:32:45] [02phabricator-extensions] 07Universal-Omega deleted branch 03bastion - 13https://github.com/miraheze/phabricator-extensions [18:32:55] [02phabricator-extensions] 07Universal-Omega reopened pull request 03#34: Support Swift file storage for Phabricator - 13https://github.com/miraheze/phabricator-extensions/pull/34 [18:33:07] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 0326 commits to 03support-swift-filestorage [+4/-0/±22] 13https://github.com/miraheze/phabricator-extensions/compare/bcce8c533b61...9f168beaf959 [18:33:08] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.36, 9.82, 9.72 [18:33:09] [02miraheze/phabricator-extensions] 07Universal-Omega 03f8489b0 - Support Swift file storage for Phabricator [18:33:11] [02miraheze/phabricator-extensions] 07github-actions 03c646dd4 - CI: lint code to MediaWiki standards [18:33:13] [02miraheze/phabricator-extensions] 07Universal-Omega 03f7cbe51 - Create PhabricatorSwiftFileStorageEngine.php [18:33:16] [02miraheze/phabricator-extensions] ... and 23 more commits. [18:33:17] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#34: Support Swift file storage for Phabricator - 13https://github.com/miraheze/phabricator-extensions/pull/34 [18:33:20] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03support-swift-filestorage [+0/-0/±3] 13https://github.com/miraheze/phabricator-extensions/compare/9f168beaf959...e012bfe47f9f [18:33:21] [02miraheze/phabricator-extensions] 07github-actions 03e012bfe - CI: lint code to MediaWiki standards [18:33:23] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#34: Support Swift file storage for Phabricator - 13https://github.com/miraheze/phabricator-extensions/pull/34 [18:33:43] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:34:04] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:35:05] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [18:35:14] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.30, 9.71, 10.07 [18:35:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.36, 6.84, 6.20 [18:37:00] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.508 second response time [18:37:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.47, 6.02, 5.98 [18:39:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.83, 9.72, 10.07 [18:40:06] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 145.38 ms [18:42:07] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 155.25 ms [18:44:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.92, 3.41, 3.89 [18:46:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.68, 3.92 [18:48:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.88, 3.76, 3.92 [18:48:57] [02dns] 07Universal-Omega closed pull request 03#469: add two cnames for lostidols.wiki - 13https://github.com/miraheze/dns/pull/469 [18:49:00] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6e5905a4a9d2...03e83eaa1334 [18:49:03] [02miraheze/dns] 07MacFan4000 0303e83ea - add two cnames for lostidols.wiki (#469) [18:54:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 4.01, 3.95 [18:57:09] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.45, 9.96, 9.12 [18:58:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.93, 3.96, 3.93 [18:58:14] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.55, 11.25, 10.02 [18:59:09] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.32, 9.32, 8.99 [19:00:14] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.67, 9.63, 9.56 [19:00:29] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:26] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: [19:05:34] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.823 second response time [19:05:42] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/5d97eba4db4f [19:05:43] [02miraheze/.github] 07dependabot[bot] 035d97eba - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 [19:05:45] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/.github [19:05:47] [02.github] 07dependabot[bot] labeled pull request 03#44: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/.github/pull/44 [19:05:49] [02.github] 07dependabot[bot] opened pull request 03#44: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/.github/pull/44 [19:06:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 3.92, 3.87 [19:08:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.85, 3.42, 3.69 [19:14:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.56, 2.97, 3.39 [19:18:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.53, 3.43, 3.49 [19:20:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.09, 3.96, 3.67 [19:22:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 3.74, 3.63 [19:23:57] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/03e83eaa1334...f1acc9cd3329 [19:23:58] [02miraheze/dns] 07paladox 03f1acc9c - Add swiftproxy171 [19:24:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.68, 4.07, 3.76 [19:25:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.52, 6.77, 6.15 [19:25:56] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 57% [19:27:56] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 15% [19:32:00] PROBLEM - swiftproxy181 NTP time on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: No route to hostconnect to host 10.0.18.108 port 5666: No route to host [19:32:59] PROBLEM - Host swiftproxy181 is DOWN: CRITICAL - Host Unreachable (10.0.18.108) [19:32:59] PROBLEM - swiftproxy181 SSH on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 and port 22: No route to host [19:32:59] PROBLEM - swiftproxy181 Disk Space on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: No route to hostconnect to host 10.0.18.108 port 5666: No route to host [19:33:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-1/±0] 13https://github.com/miraheze/puppet/commit/ff51e4b3db58 [19:33:05] [02miraheze/puppet] 07paladox 03ff51e4b - Rename swiftproxy181.yaml to swiftproxy171.yaml [19:33:06] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:33:08] [02puppet] 07paladox opened pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:34:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff51e4b3db58...5745fa99b35b [19:34:39] [02miraheze/puppet] 07paladox 035745fa9 - Update swiftobject151.yaml [19:34:42] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:34:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5745fa99b35b...bfd4af2bac65 [19:34:48] [02miraheze/puppet] 07paladox 03bfd4af2 - Update swiftobject161.yaml [19:34:51] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:34:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bfd4af2bac65...9c159e9c3dcc [19:34:58] [02miraheze/puppet] 07paladox 039c159e9 - Update swiftobject171.yaml [19:35:01] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:35:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9c159e9c3dcc...e40739c6c5bb [19:35:09] [02miraheze/puppet] 07paladox 03e40739c - Update swiftobject181.yaml [19:35:12] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:35:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e40739c6c5bb...ebfe51f284cd [19:35:38] [02miraheze/puppet] 07paladox 03ebfe51f - Update swiftproxy151.yaml [19:35:39] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:36:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ebfe51f284cd...b898fcac82cc [19:36:46] [02miraheze/puppet] 07paladox 03b898fca - Update swiftproxy171.yaml [19:36:48] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:36:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b898fcac82cc...ddc9134b4f8e [19:36:57] [02miraheze/puppet] 07paladox 03ddc9134 - Update swiftproxy151.yaml [19:36:58] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:37:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ddc9134b4f8e...3f46999c41ab [19:37:13] [02miraheze/puppet] 07paladox 033f46999 - Update swiftac171.yaml [19:37:14] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:37:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3f46999c41ab...276e9b6ea1f1 [19:37:38] [02miraheze/puppet] 07paladox 03276e9b6 - Update swiftac171.yaml [19:37:40] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:39:23] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.35, 6.66, 6.46 [19:40:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.60, 3.95, 3.97 [19:40:23] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:40:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±7] 13https://github.com/miraheze/puppet/compare/276e9b6ea1f1...5441d5defe93 [19:40:43] [02miraheze/puppet] 07paladox 035441d5d - Fix [19:40:46] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:41:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5441d5defe93...edaeb6d30ca8 [19:41:22] [02miraheze/puppet] 07paladox 03edaeb6d - Update site.pp [19:41:24] [02puppet] 07paladox synchronize pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:41:26] [02puppet] 07paladox closed pull request 03#3694: Rename swiftproxy181.yaml to swiftproxy171.yaml - 13https://github.com/miraheze/puppet/pull/3694 [19:41:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±8] 13https://github.com/miraheze/puppet/compare/be8d8bef26b0...85b9a15dbf20 [19:41:28] [02miraheze/puppet] 07paladox 0385b9a15 - Rename swiftproxy181.yaml to swiftproxy171.yaml (#3694) [19:41:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [19:41:32] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [19:43:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.41, 6.91, 6.64 [19:43:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.26, 6.93, 6.52 [19:44:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:45:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.26, 6.80 [19:45:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.60, 7.55, 6.80 [19:45:46] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 3.91, 3.94 [19:47:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.29, 6.87 [19:47:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.37, 7.10, 6.72 [19:47:46] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.72, 3.72, 3.86 [19:49:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.59, 6.49, 6.54 [19:51:46] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.97, 4.07, 3.94 [19:52:58] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.92, 11.73, 8.97 [19:53:01] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.72, 11.53, 8.84 [19:53:02] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.89, 11.20, 9.49 [19:53:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.68, 10.86, 9.25 [19:54:40] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [19:55:42] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.30, 12.89, 10.15 [19:56:34] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.493 second response time [20:00:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.40, 6.78, 6.29 [20:02:39] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.21, 6.59, 6.28 [20:05:01] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:05:01] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:05:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.33, 6.35, 6.76 [20:06:50] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.54, 9.82, 7.85 [20:07:01] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.776 second response time [20:07:01] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.643 second response time [20:08:34] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.34, 11.61, 11.19 [20:08:48] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.42, 8.34, 7.53 [20:10:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.12, 12.10, 11.38 [20:11:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.33, 6.16, 5.84 [20:12:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.89, 7.14, 6.56 [20:13:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.18, 6.88 [20:13:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.98, 6.52, 6.21 [20:13:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.71, 5.63, 5.69 [20:15:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.17, 6.91 [20:15:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.49, 7.01, 6.42 [20:17:01] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.07, 11.66, 11.97 [20:17:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.93, 7.24, 6.57 [20:18:19] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.97, 11.20, 11.50 [20:18:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.87, 10.91, 11.95 [20:19:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.83, 6.43, 5.96 [20:20:16] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 11.60, 11.44, 12.02 [20:20:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/cd5973c807e7...92fd78d08dff [20:20:24] [02miraheze/ssl] 07MirahezeSSLBot 0392fd78d - Bot: Add SSL cert for lostidols.wiki [20:20:53] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.78, 12.08, 12.01 [20:21:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.70, 8.50, 7.49 [20:22:13] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 11.76, 11.66 [20:22:59] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/92fd78d08dff...6f043ed3abd6 [20:23:01] [02miraheze/ssl] 07MacFan4000 036f043ed - add www.lostidols.wiki redirect [20:23:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.75, 7.34 [20:23:21] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.67, 6.56, 6.51 [20:23:28] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.41, 6.73, 6.23 [20:24:06] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.31, 2.25 [20:24:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.85, 6.75, 6.77 [20:28:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.20, 3.27, 2.51 [20:28:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.03, 11.75, 11.75 [20:29:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.50, 6.37, 6.80 [20:30:01] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.16, 13.94, 12.55 [20:30:29] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:31:09] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:31:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 6.53, 5.98 [20:31:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.34, 6.88, 6.01 [20:31:31] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:32:27] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.347 second response time [20:32:31] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.33, 7.44, 6.97 [20:33:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.40, 7.11 [20:33:12] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3369 bytes in 2.944 second response time [20:33:14] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 6.999 second response time [20:33:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.03, 7.05, 6.24 [20:33:23] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.83, 7.70, 6.89 [20:33:31] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 3.716 second response time [20:33:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.30, 10.59, 8.46 [20:33:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.70, 3.36, 2.73 [20:35:10] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.616 second response time [20:35:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.28, 7.47, 6.47 [20:35:33] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 9.21, 9.58, 8.32 [20:35:55] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.32, 3.17, 2.72 [20:36:19] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:36:32] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:36:34] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:37:04] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:37:10] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.223 second response time [20:37:11] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.466 second response time [20:38:15] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.658 second response time [20:38:30] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.579 second response time [20:38:34] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 5.410 second response time [20:38:46] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [20:39:10] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.661 second response time [20:39:10] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:39:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.85, 7.98, 7.00 [20:39:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.07, 7.68, 6.86 [20:40:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.72, 3.94, 3.12 [20:40:20] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [20:41:05] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.678 second response time [20:41:13] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.634 second response time [20:41:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 8.11, 7.16 [20:41:19] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.555 second response time [20:41:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.19, 3.59, 3.09 [20:42:15] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.68, 11.15, 9.33 [20:42:16] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.585 second response time [20:44:04] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.66, 3.17 [20:44:13] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.34, 11.31, 9.60 [20:46:41] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:47:20] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:47:40] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.70, 9.40, 8.50 [20:47:59] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:49:18] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.483 second response time [20:49:38] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.36, 8.70, 8.37 [20:49:58] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.840 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [20:50:08] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.81, 10.42, 9.85 [20:51:25] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:51:34] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.88, 9.83, 11.78 [20:52:06] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 21.79, 13.75, 11.06 [20:52:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.22, 8.07, 7.46 [20:53:22] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.512 second response time [20:53:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.04, 10.59, 11.80 [20:54:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.09, 7.52, 7.35 [20:55:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.35, 7.94, 7.77 [20:56:03] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.42, 11.39, 10.73 [20:57:02] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.01, 10.40, 9.73 [20:58:57] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.85, 9.73, 9.57 [21:00:00] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.01, 8.71, 9.79 [21:00:42] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.61, 6.14, 6.77 [21:01:11] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [21:01:21] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:01:24] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [21:01:30] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:01:31] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:02:44] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:02:52] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.81, 11.37, 10.27 [21:02:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.52, 7.69, 7.98 [21:03:07] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.854 second response time [21:03:19] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.817 second response time [21:03:22] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.382 second response time [21:03:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.69, 7.77, 7.61 [21:03:29] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.619 second response time [21:03:40] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw131 [21:03:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.55, 6.85, 7.72 [21:03:55] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.64, 10.35, 10.28 [21:04:16] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.14, 11.78, 10.04 [21:04:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.47, 6.94, 7.79 [21:04:43] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.310 second response time [21:04:47] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.90, 12.54, 10.86 [21:04:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.90, 8.02 [21:05:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.81, 7.75, 7.63 [21:05:38] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.678 second response time [21:05:38] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [21:06:41] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.94, 11.10, 10.55 [21:07:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 7.90, 7.70 [21:07:52] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.08, 9.19, 9.82 [21:08:16] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.41, 9.19, 9.40 [21:08:36] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.84, 9.55, 10.06 [21:08:50] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.15, 7.72, 7.94 [21:08:50] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 53% [21:09:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.39, 10.05, 11.68 [21:09:24] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.86, 7.82, 7.69 [21:10:46] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 5% [21:11:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.53, 8.54, 7.97 [21:14:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.77, 7.70, 7.60 [21:15:44] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.83, 7.40, 7.36 [21:16:28] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.18, 7.52, 7.55 [21:17:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.66, 7.06, 7.25 [21:18:27] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.44, 10.24, 11.71 [21:19:08] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.34, 10.17, 11.82 [21:19:12] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:19:13] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:20:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.15, 7.61, 7.60 [21:21:09] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.488 second response time [21:21:10] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.146 second response time [21:21:23] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.12, 8.88, 10.17 [21:21:30] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:22:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.71, 7.38, 7.53 [21:23:28] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.522 second response time [21:23:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85b9a15dbf20...4ba780a47644 [21:23:38] [02miraheze/puppet] 07paladox 034ba780a - swift: increase conn_timeout to 1.5 for object-server [21:23:38] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.49, 11.89, 10.06 [21:23:52] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.39, 11.37, 9.92 [21:24:16] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.98, 10.91, 9.25 [21:24:27] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 16.58, 12.31, 11.94 [21:25:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 12.02, 11.88 [21:25:23] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.38, 12.36, 11.16 [21:25:37] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.78, 11.27, 10.07 [21:25:46] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.01, 10.52, 9.79 [21:26:16] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.60, 9.68, 9.00 [21:26:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.24, 7.52, 7.43 [21:26:55] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:27:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.35, 11.03, 11.54 [21:27:12] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:27:35] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.70, 13.63, 11.03 [21:27:43] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.27, 12.12, 10.46 [21:28:27] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.98, 7.24, 7.33 [21:28:52] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.712 second response time [21:29:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.61, 13.63, 12.43 [21:29:09] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [21:29:33] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.92, 11.81, 10.63 [21:29:43] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.73, 9.75, 9.77 [21:30:16] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.56, 10.53, 9.60 [21:31:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.80, 6.85, 7.86 [21:31:32] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.38, 12.02, 10.82 [21:31:36] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:32:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.55, 5.84, 6.66 [21:33:30] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.10, 10.71, 10.48 [21:33:35] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.687 second response time [21:33:43] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.10, 10.77, 10.31 [21:33:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.15, 6.39, 6.77 [21:34:05] [02TSPortal] 07paladox closed pull request 03#13: Bump guzzlehttp/psr7 from 2.4.3 to 2.6.2 - 13https://github.com/miraheze/TSPortal/pull/13 [21:34:08] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/b202f53e521d...1635e04cdffb [21:34:11] [02miraheze/TSPortal] 07dependabot[bot] 031635e04 - Bump guzzlehttp/psr7 from 2.4.3 to 2.6.2 (#13) [21:34:12] [02miraheze/TSPortal] 07paladox deleted branch 03dependabot/composer/guzzlehttp/psr7-2.6.2 [21:34:15] [02TSPortal] 07paladox deleted branch 03dependabot/composer/guzzlehttp/psr7-2.6.2 - 13https://github.com/miraheze/TSPortal [21:34:54] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/commit/9872c0ca0fbf [21:34:57] [02miraheze/TSPortal] 07paladox 039872c0c - Support php 8 [21:34:59] [02TSPortal] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [21:35:00] [02TSPortal] 07paladox opened pull request 03#14: Support php 8 - 13https://github.com/miraheze/TSPortal/pull/14 [21:35:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.35, 7.04, 7.83 [21:35:43] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.55, 9.87, 10.02 [21:36:16] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.56, 9.58, 9.62 [21:36:32] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 3.21, 2.35 [21:37:29] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/9872c0ca0fbf...91b65699997d [21:37:32] [02miraheze/TSPortal] 07paladox 0391b6569 - Update composer.json [21:37:35] [02TSPortal] 07paladox synchronize pull request 03#14: Support php 8 - 13https://github.com/miraheze/TSPortal/pull/14 [21:37:36] [02TSPortal] 07paladox closed pull request 03#14: Support php 8 - 13https://github.com/miraheze/TSPortal/pull/14 [21:37:39] [02miraheze/TSPortal] 07paladox deleted branch 03paladox-patch-1 [21:37:41] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:37:42] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/1635e04cdffb...d403c30321ac [21:37:45] [02miraheze/TSPortal] 07paladox 03d403c30 - Support php 8 (#14) [21:37:46] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [21:37:46] [02TSPortal] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [21:38:29] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.05, 2.77, 2.30 [21:38:49] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:38:57] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [21:39:19] [02miraheze/TSPortal] 07paladox tagged 03d403c30 as 03v12 13https://github.com/miraheze/TSPortal/commit/d403c30321ace4cb26b3521d828fee0a75f9fe89 [21:39:22] [02TSPortal] 07paladox created tag 03v12 - 13https://github.com/miraheze/TSPortal [21:39:40] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.504 second response time [21:39:41] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.660 second response time [21:39:43] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.98, 12.92, 11.18 [21:39:47] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 11.63, 12.38, 11.28 [21:40:45] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.24, 16.70, 12.85 [21:40:51] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.519 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [21:40:55] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [21:41:09] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [21:41:43] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.22, 10.55, 10.51 [21:41:45] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.74, 10.32, 10.66 [21:41:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-1/±0] 13https://github.com/miraheze/puppet/commit/8f6f0a5b8ec1 [21:42:01] [02miraheze/puppet] 07paladox 038f6f0a5 - Rename swiftproxy151.yaml to swiftproxy161.yaml [21:42:02] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [21:42:05] [02puppet] 07paladox opened pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:42:40] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:42:46] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:42:47] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:42:51] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:43:25] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.55, 5.54, 6.71 [21:43:43] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.72, 11.52, 11.05 [21:44:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8f6f0a5b8ec1...dad70a8d47eb [21:44:20] [02miraheze/puppet] 07paladox 03dad70a8 - Update proxy-server.conf.erb [21:44:21] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:44:27] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.48, 6.18, 6.64 [21:44:32] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.51, 3.70, 2.80 [21:44:39] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.139 second response time [21:44:46] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.317 second response time [21:44:47] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.574 second response time [21:44:49] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 3.817 second response time [21:45:00] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [21:45:11] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:45:11] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [21:45:17] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:45:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dad70a8d47eb...c13e975502ea [21:45:21] [02miraheze/puppet] 07paladox 03c13e975 - Update proxy.pp [21:45:22] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:45:43] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 12.29, 11.22 [21:45:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c13e975502ea...46120b876bb5 [21:45:56] [02miraheze/puppet] 07paladox 0346120b8 - Update swiftproxy161.yaml [21:45:58] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:46:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/46120b876bb5...1040597cbf0c [21:46:04] [02miraheze/puppet] 07paladox 031040597 - Update swiftproxy171.yaml [21:46:06] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:46:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1040597cbf0c...b40146cd1040 [21:46:17] [02miraheze/puppet] 07paladox 03b40146c - Update site.pp [21:46:18] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:46:39] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.52, 11.47, 11.76 [21:47:08] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.007 second response time [21:47:09] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.641 second response time [21:47:17] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.908 second response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [21:47:38] [02puppet] 07paladox synchronize pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:47:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b40146cd1040...f4dca9f6fead [21:47:40] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.42, 10.22, 10.76 [21:47:41] [02miraheze/puppet] 07paladox 03f4dca9f - Update proxy-server.conf.erb [21:47:43] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 6.62, 10.24, 10.61 [21:48:07] [02puppet] 07paladox closed pull request 03#3695: Rename swiftproxy151.yaml to swiftproxy161.yaml - 13https://github.com/miraheze/puppet/pull/3695 [21:48:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [21:48:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±4] 13https://github.com/miraheze/puppet/compare/4ba780a47644...04ee4b915a8f [21:48:13] [02miraheze/puppet] 07paladox 0304ee4b9 - Rename swiftproxy151.yaml to swiftproxy161.yaml (#3695) [21:48:15] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [21:48:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.41, 6.35, 6.76 [21:49:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/04ee4b915a8f...6334665b04d6 [21:49:13] [02miraheze/puppet] 07paladox 036334665 - Fix [21:49:29] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [21:49:38] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.12, 8.88, 10.20 [21:49:43] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.81, 8.83, 10.05 [21:50:08] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is WARNING: WARNING - NGINX Error Rate is 57% [21:50:29] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.73, 11.03, 11.91 [21:50:58] PROBLEM - swiftproxy161 Puppet on swiftproxy161 is CRITICAL: connect to address 10.0.16.135 port 5666: Connection refusedconnect to host 10.0.16.135 port 5666: Connection refused [21:51:27] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.691 second response time [21:52:08] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 4% [21:52:29] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.81, 12.03, 12.16 [21:52:32] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.28, 7.59, 9.94 [21:52:55] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:58] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [21:53:09] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 51.65, 26.03, 11.41 [21:54:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6334665b04d6...37ecbd0a7396 [21:54:35] [02miraheze/puppet] 07paladox 0337ecbd0 - swift: increase conn_timeout to 1.5 for object-updater [21:54:38] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:54:54] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:54:57] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.380 second response time [21:54:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.03, 6.93 [21:55:00] PROBLEM - db121 Puppet on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:55:02] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.374 second response time [21:55:04] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 1.071 second response time [21:55:13] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.39, 2.09, 3.70 [21:55:24] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3393 bytes in 0.563 second response time [21:55:28] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:55:37] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:55:46] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:56:06] PROBLEM - db121 SSH on db121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:16] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:24] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:25] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:56:28] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.225 second response time [21:56:29] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 3.94, 9.45, 11.27 [21:56:29] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:56:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:56:34] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.469 second response time [21:56:37] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.222 second response time [21:56:40] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:56:42] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3362 bytes in 0.519 second response time [21:56:52] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:56:56] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 1.63, 7.88, 11.35 [21:56:59] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.463 second response time [21:56:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 6.28, 6.68 [21:57:01] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [21:57:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 1.57, 8.07, 10.78 [21:57:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.36, 1.49, 3.28 [21:57:11] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [21:57:15] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 [21:57:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 1.25, 7.37, 10.63 [21:57:25] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:28] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.647 second response time [21:57:40] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:58:01] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [21:58:18] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.434 second response time [21:58:21] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.29, 3.10, 3.37 [21:58:23] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.134 second response time [21:58:29] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 1.16, 6.56, 9.98 [21:58:37] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.566 second response time [21:58:41] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.352 second response time [21:58:42] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.520 second response time [21:58:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b0feb6f21781 [21:58:44] [02miraheze/puppet] 07paladox 03b0feb6f - Fix [21:58:45] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [21:58:48] [02puppet] 07paladox opened pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [21:58:52] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.571 second response time [21:58:58] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [21:58:58] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [21:59:01] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [21:59:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.87, 9.12, 10.76 [21:59:06] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.653 second response time [21:59:08] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 5.164 second response time [21:59:10] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.730 second response time [21:59:20] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to server on 'db121.miraheze.org' (115) [21:59:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b0feb6f21781...ebff00a14130 [21:59:21] [02miraheze/puppet] 07paladox 03ebff00a - Update swiftobject161.yaml [21:59:21] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3481 bytes in 0.579 second response time [21:59:22] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.378 second response time [21:59:23] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.89, 9.39, 10.93 [21:59:24] [02puppet] 07paladox synchronize pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [21:59:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ebff00a14130...fd675c160bde [21:59:31] [02miraheze/puppet] 07paladox 03fd675c1 - Update swiftobject171.yaml [21:59:33] [02puppet] 07paladox synchronize pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [21:59:34] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.474 second response time [21:59:35] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 0.106 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [21:59:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fd675c160bde...fcb3342559e5 [21:59:38] [02miraheze/puppet] 07paladox 03fcb3342 - Update swiftobject181.yaml [21:59:40] [02puppet] 07paladox synchronize pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [21:59:47] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.815 second response time [21:59:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fcb3342559e5...f00ad2daca22 [21:59:49] [02miraheze/puppet] 07paladox 03f00ad2d - Update swiftproxy161.yaml [21:59:50] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:59:51] [02puppet] 07paladox synchronize pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [21:59:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f00ad2daca22...0e0b02c2dd77 [21:59:57] [02miraheze/puppet] 07paladox 030e0b02c - Update swiftproxy171.yaml [22:00:00] [02puppet] 07paladox synchronize pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [22:00:01] [02puppet] 07paladox closed pull request 03#3696: Fix - 13https://github.com/miraheze/puppet/pull/3696 [22:00:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/37ecbd0a7396...739c52a8c071 [22:00:05] [02miraheze/puppet] 07paladox 03739c52a - Fix (#3696) [22:00:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [22:00:08] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [22:00:19] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [22:00:23] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [22:00:24] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 22 backends are healthy [22:00:31] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.580 second response time [22:00:34] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.774 second response time [22:00:40] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 22 backends are healthy [22:00:49] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 22 backends are healthy [22:00:55] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 9.4%Current connections: 47 [22:01:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.67, 8.45, 10.29 [22:01:04] PROBLEM - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is WARNING: WARNING - NGINX Error Rate is 40% [22:01:06] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [22:01:09] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.76, 3.22, 3.54 [22:01:16] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 191 Threads: 39 Questions: 72510 Slow queries: 0 Opens: 3194 Open tables: 3177 Queries per second avg: 379.633 [22:01:21] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.881 second response time [22:01:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.70, 8.77, 10.50 [22:02:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/739c52a8c071...1063cf6ab277 [22:02:20] [02miraheze/puppet] 07paladox 031063cf6 - reports: upgrade to v12 [22:02:29] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.01, 9.30, 10.39 [22:02:56] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.03, 10.37, 11.04 [22:03:06] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.31, 2.98, 3.43 [22:04:29] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.71, 11.12, 10.91 [22:05:04] RECOVERY - cp25 HTTP 4xx/5xx ERROR Rate on cp25 is OK: OK - NGINX Error Rate is 38% [22:05:04] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.27, 2.34, 3.14 [22:05:46] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f1acc9cd3329...0f76aafe9d8e [22:05:49] [02miraheze/dns] 07paladox 030f76aaf - Remove swiftproxy181 from dns [22:06:18] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0f76aafe9d8e...2681f91338f6 [22:06:20] [02miraheze/dns] 07paladox 032681f91 - Add reports171 to dns [22:07:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.18, 10.56, 10.60 [22:09:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.73, 11.38, 10.93 [22:09:23] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 11.12, 10.89 [22:10:29] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.03, 11.50, 11.36 [22:11:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.00, 11.96, 11.20 [22:11:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.35, 10.66, 10.72 [22:13:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.98, 11.09, 10.97 [22:19:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.35, 8.73, 10.00 [22:19:24] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.25, 10.44, 11.53 [22:19:45] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.52, 8.72, 9.85 [22:21:00] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.15, 11.24, 10.64 [22:23:22] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.76, 12.00, 11.84 [22:24:22] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.61, 11.44, 10.95 [22:25:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:25:12] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.931 second response time [22:25:36] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.97, 12.10, 10.89 [22:26:38] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:26:41] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.05, 3.75, 2.75 [22:26:55] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.535 second response time [22:28:37] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 2.842 second response time [22:30:31] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.65, 1.64, 7.60 [22:32:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.77, 3.46, 3.04 [22:33:31] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:34:23] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.35, 2.47, 6.62 [22:34:33] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.62, 4.02, 3.29 [22:34:46] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [22:34:46] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:34:48] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:35:03] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.30, 12.48, 9.39 [22:35:06] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:35:06] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:35:59] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [22:36:13] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.42, 13.69, 10.26 [22:36:16] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [22:36:21] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw134 [22:36:22] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw134 [22:36:33] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.58, 3.23, 3.10 [22:36:41] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.88, 10.37, 8.90 [22:36:44] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.305 second response time [22:36:45] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.640 second response time [22:36:46] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.825 second response time [22:36:59] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.48, 10.96, 9.21 [22:37:01] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.591 second response time [22:37:02] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.123 second response time [22:37:29] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 0.654 second response time [22:37:56] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 22 backends are healthy [22:38:13] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.06, 11.19, 9.76 [22:38:15] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 22 backends are healthy [22:38:16] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 22 backends are healthy [22:38:21] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 22 backends are healthy [22:38:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 10.13, 9.92, 8.88 [22:38:55] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.07, 9.88, 9.01 [22:39:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.18, 11.37, 11.97 [22:39:22] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:13] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.00, 9.55, 9.34 [22:41:24] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.01, 11.29, 11.91 [22:41:28] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.779 second response time [22:42:00] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.44, 11.49, 11.95 [22:43:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.95, 12.26, 12.15 [22:43:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1e59471eb908 [22:43:45] [02miraheze/puppet] 07paladox 031e59471 - Install reports171 [22:43:47] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [22:43:49] [02puppet] 07paladox opened pull request 03#3697: Install reports171 - 13https://github.com/miraheze/puppet/pull/3697 [22:44:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1e59471eb908...081f7b7a3901 [22:44:17] [02miraheze/puppet] 07paladox 03081f7b7 - Create reports171.yaml [22:44:19] [02puppet] 07paladox synchronize pull request 03#3697: Install reports171 - 13https://github.com/miraheze/puppet/pull/3697 [22:45:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.60, 11.32, 11.81 [22:47:19] PROBLEM - reports171 conntrack_table_size on reports171 is UNKNOWN: NRPE: Unable to read output [22:47:31] PROBLEM - reports171 ferm_active on reports171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:48:13] PROBLEM - reports171 Puppet on reports171 is UNKNOWN: NRPE: Unable to read output [22:49:25] PROBLEM - reports171 ferm_active on reports171 is UNKNOWN: NRPE: Unable to read output [22:49:50] [02puppet] 07paladox synchronize pull request 03#3697: Install reports171 - 13https://github.com/miraheze/puppet/pull/3697 [22:49:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/081f7b7a3901...972fbeef7632 [22:49:55] [02miraheze/puppet] 07paladox 03972fbee - Update reports171.yaml [22:50:07] [02puppet] 07paladox closed pull request 03#3697: Install reports171 - 13https://github.com/miraheze/puppet/pull/3697 [22:50:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [22:50:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/1063cf6ab277...69de743479ce [22:50:14] [02miraheze/puppet] 07paladox 0369de743 - Install reports171 (#3697) [22:50:17] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [22:50:20] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.06, 11.89, 11.84 [22:51:06] RECOVERY - reports171 conntrack_table_size on reports171 is OK: OK: nf_conntrack is 0 % full [22:51:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 6.81, 6.29 [22:51:20] RECOVERY - reports171 ferm_active on reports171 is OK: OK ferm input default policy is set [22:52:05] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[reports_composer] [22:52:17] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.38, 11.46, 11.67 [22:53:36] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/d403c30321ac...8055e851d068 [22:53:37] [02miraheze/TSPortal] 07paladox 038055e85 - Update composer lock file [22:54:14] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.45, 12.16, 11.88 [22:54:57] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.93, 11.70, 11.71 [22:55:16] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 11.50, 11.38 [22:55:21] [02miraheze/TSPortal] 07paladox tagged 038055e85 as 03v13 13https://github.com/miraheze/TSPortal/commit/8055e851d068688786969b0af4bedd4fb107eef8 [22:55:23] [02TSPortal] 07paladox created tag 03v13 - 13https://github.com/miraheze/TSPortal [22:55:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/69de743479ce...9d6978abf3e3 [22:55:43] [02miraheze/puppet] 07paladox 039d6978a - reports: upgrade to v13 [22:55:55] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.42, 8.65, 8.21 [22:56:11] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.56, 11.87, 11.83 [22:56:56] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.16, 11.63, 11.70 [22:57:05] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.55, 6.39 [22:57:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.09, 10.95, 11.20 [22:57:50] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.39, 8.35, 8.14 [22:57:55] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [22:58:54] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.15, 12.18, 11.88 [23:00:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.69, 3.53, 4.00 [23:00:20] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_TSPortal] [23:00:53] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.26, 11.13, 11.54 [23:02:59] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 11.14, 8.23, 5.30 [23:03:13] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.12, 10.69, 10.90 [23:03:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-bast [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/eed14fb2007c [23:03:40] [02miraheze/puppet] 07Universal-Omega 03eed14fb - varnish: update bastion hosts [23:03:43] [02puppet] 07Universal-Omega created branch 03varnish-bast - 13https://github.com/miraheze/puppet [23:03:47] [02puppet] 07Universal-Omega opened pull request 03#3698: varnish: update bastion hosts - 13https://github.com/miraheze/puppet/pull/3698 [23:05:03] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:05:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.00, 3.25, 2.45 [23:05:55] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.28, 12.25, 11.71 [23:06:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.19, 3.74, 3.94 [23:06:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d6978abf3e3...6af2152adb99 [23:06:14] [02miraheze/puppet] 07paladox 036af2152 - reports: use http proxy [23:06:50] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 11.25, 11.35 [23:07:02] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.565 second response time [23:07:11] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:07:12] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.71, 10.98, 10.94 [23:07:52] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.25, 11.99, 11.69 [23:08:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/eacce97f8c79 [23:08:11] [02miraheze/puppet] 07paladox 03eacce97 - reports: switch db to db182-private [23:08:12] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:08:13] [02puppet] 07paladox opened pull request 03#3699: reports: switch db to db182-private - 13https://github.com/miraheze/puppet/pull/3699 [23:08:48] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.59, 10.76, 11.15 [23:08:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/328e5533565b [23:08:54] [02miraheze/puppet] 07paladox 03328e553 - varnish: switch reports121 to reports172 [23:08:57] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [23:08:58] [02puppet] 07paladox opened pull request 03#3700: varnish: switch reports121 to reports172 - 13https://github.com/miraheze/puppet/pull/3700 [23:09:06] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.502 second response time [23:09:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.13, 3.37, 2.70 [23:09:35] [02puppet] 07paladox edited pull request 03#3700: varnish: switch reports121 to reports171 - 13https://github.com/miraheze/puppet/pull/3700 [23:10:16] PROBLEM - reports121 Puppet on reports121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 1 failures [23:10:47] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.42, 12.94, 11.87 [23:11:20] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 242.25 ms [23:11:49] [02puppet] 07paladox closed pull request 03#3700: varnish: switch reports121 to reports171 - 13https://github.com/miraheze/puppet/pull/3700 [23:11:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6af2152adb99...f6c7efefdbc2 [23:11:53] [02miraheze/puppet] 07paladox 03f6c7efe - varnish: switch reports121 to reports171 (#3700) [23:11:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14 [23:11:56] [02puppet] 07paladox deleted branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [23:11:59] [02puppet] 07paladox closed pull request 03#3699: reports: switch db to db182-private - 13https://github.com/miraheze/puppet/pull/3699 [23:12:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f6c7efefdbc2...c1c4dc07eae6 [23:12:04] [02miraheze/puppet] 07paladox 03c1c4dc0 - reports: switch db to db182-private (#3699) [23:12:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [23:12:09] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:12:11] [02puppet] 07Universal-Omega reviewed pull request 03#3699 commit - 13https://github.com/miraheze/puppet/pull/3699#discussion_r1467062047 [23:12:46] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.39, 10.90, 11.27 [23:13:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c1c4dc07eae6...cac5ad471176 [23:13:18] [02miraheze/puppet] 07paladox 03cac5ad4 - Fix [23:13:19] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 167.90 ms [23:13:29] PROBLEM - reports121 HTTPS on reports121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to reports121.miraheze.org port 443 after 122 ms: Couldn't connect to server [23:15:10] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.57, 8.83, 9.96 [23:17:14] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:18:35] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.66, 11.79, 10.90 [23:19:05] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.53, 10.97, 11.00 [23:19:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cac5ad471176...97a92a102623 [23:19:08] [02miraheze/puppet] 07paladox 0397a92a1 - Fix [23:19:09] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.537 second response time [23:19:17] [02puppet] 07paladox reviewed pull request 03#3699 commit - 13https://github.com/miraheze/puppet/pull/3699#discussion_r1467069128 [23:21:01] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 5.43, 7.62, 7.57 [23:21:03] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.83, 9.33, 10.40 [23:21:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 6.92, 6.60 [23:21:53] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.77, 3.70, 3.94 [23:22:16] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/commit/7dd2cd75a5f9 [23:22:17] [02miraheze/TSPortal] 07paladox 037dd2cd7 - Change footer-company [23:22:20] [02TSPortal] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [23:22:21] [02TSPortal] 07paladox opened pull request 03#15: Change footer-company - 13https://github.com/miraheze/TSPortal/pull/15 [23:22:31] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.37, 11.83, 11.15 [23:23:07] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:39] [02TSPortal] 07paladox closed pull request 03#15: Change footer-company - 13https://github.com/miraheze/TSPortal/pull/15 [23:23:42] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/8055e851d068...cd5b55392386 [23:23:44] [02miraheze/TSPortal] 07paladox 03cd5b553 - Change footer-company (#15) [23:23:46] [02miraheze/TSPortal] 07paladox deleted branch 03paladox-patch-1 [23:23:47] [02TSPortal] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [23:23:52] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.77, 4.51, 4.20 [23:24:10] PROBLEM - ping6 on reports121 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:24:16] PROBLEM - Host reports121 is DOWN: PING CRITICAL - Packet loss = 100% [23:24:58] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 2.33, 5.05, 6.55 [23:25:19] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/cd5b55392386...d8942aa6e9af [23:25:20] [02miraheze/TSPortal] 07paladox 03d8942aa - Add changelings for ver 12, 13 and 14 [23:25:28] PROBLEM - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is WARNING: WARNING - NGINX Error Rate is 43% [23:25:50] [02miraheze/TSPortal] 07paladox tagged 03d8942aa as 03v14 13https://github.com/miraheze/TSPortal/commit/d8942aa6e9afcc1d224bf743aa1aa53e96993356 [23:25:53] [02TSPortal] 07paladox created tag 03v14 - 13https://github.com/miraheze/TSPortal [23:26:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/97a92a102623...59dc198dc01c [23:26:14] [02miraheze/puppet] 07paladox 0359dc198 - reports: upgrade to v14 [23:26:27] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.76, 12.77, 11.66 [23:27:08] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/commit/258738275a7f [23:27:09] [02miraheze/TSPortal] 07paladox 032587382 - Bump version number to 15 [23:27:10] [02TSPortal] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [23:27:11] [02TSPortal] 07paladox opened pull request 03#16: Bump version number to 15 - 13https://github.com/miraheze/TSPortal/pull/16 [23:27:26] RECOVERY - cp34 HTTP 4xx/5xx ERROR Rate on cp34 is OK: OK - NGINX Error Rate is 11% [23:27:36] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/258738275a7f...935fa6450213 [23:27:37] [02miraheze/TSPortal] 07paladox 03935fa64 - Update CHANGELOG.md [23:27:38] [02TSPortal] 07paladox synchronize pull request 03#16: Bump version number to 15 - 13https://github.com/miraheze/TSPortal/pull/16 [23:28:06] [02TSPortal] 07paladox closed pull request 03#16: Bump version number to 15 - 13https://github.com/miraheze/TSPortal/pull/16 [23:28:07] [02miraheze/TSPortal] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/TSPortal/compare/d8942aa6e9af...5afb00fd039b [23:28:10] [02miraheze/TSPortal] 07paladox 035afb00f - Bump version number to 15 (#16) [23:28:11] [02miraheze/TSPortal] 07paladox deleted branch 03paladox-patch-1 [23:28:14] [02TSPortal] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/TSPortal [23:28:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.59, 6.65 [23:29:21] [02miraheze/TSPortal] 07paladox tagged 035afb00f as 03v15 13https://github.com/miraheze/TSPortal/commit/5afb00fd039be6921601c23a30ec61c33a286276 [23:29:24] [02TSPortal] 07paladox created tag 03v15 - 13https://github.com/miraheze/TSPortal [23:29:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/95a030373aed [23:29:46] [02miraheze/puppet] 07paladox 0395a0303 - reports: upgrade to v15 [23:29:49] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:29:52] [02puppet] 07paladox opened pull request 03#3701: reports: upgrade to v15 - 13https://github.com/miraheze/puppet/pull/3701 [23:29:53] [02puppet] 07paladox closed pull request 03#3701: reports: upgrade to v15 - 13https://github.com/miraheze/puppet/pull/3701 [23:29:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/59dc198dc01c...148e97bd2771 [23:29:57] [02miraheze/puppet] 07paladox 03148e97b - reports: upgrade to v15 (#3701) [23:29:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [23:30:01] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:30:35] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.28, 11.26, 11.98 [23:30:58] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.50, 9.57, 10.16 [23:31:36] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [23:31:45] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:32:22] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.01, 10.91, 11.37 [23:33:16] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.45, 11.10, 8.95 [23:33:32] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 0.533 second response time [23:33:44] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.496 second response time [23:35:16] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.94, 9.45, 8.60 [23:37:18] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:37:33] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:38:16] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.08, 11.29, 11.21 [23:38:22] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.02, 12.53, 11.98 [23:38:37] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:38:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.15, 11.49, 10.64 [23:39:09] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.95, 10.04, 8.85 [23:39:16] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.36, 10.68, 9.31 [23:39:17] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.176 second response time [23:39:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.44, 11.48, 10.89 [23:39:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.12, 3.94, 3.05 [23:40:00] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.87, 11.20, 11.60 [23:40:14] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.575 second response time [23:40:36] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.980 second response time [23:40:47] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.62, 10.66, 10.41 [23:41:07] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.98, 9.56, 8.82 [23:41:16] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.98, 9.68, 9.14 [23:41:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.72, 10.91, 10.74 [23:41:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.26, 2.94, 2.79 [23:41:56] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 12.25, 11.96 [23:43:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.17, 11.27, 11.30 [23:43:56] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.96, 11.38, 11.67 [23:43:56] RECOVERY - db161 MariaDB on db161 is OK: Uptime: 3 Threads: 1 Questions: 1 Slow queries: 0 Opens: 16 Open tables: 10 Queries per second avg: 0.333 [23:44:01] RECOVERY - db161 MariaDB Connections on db161 is OK: OK connection usage: 0.2%Current connections: 1 [23:44:41] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.15, 9.45, 9.99 [23:45:38] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.14, 9.27, 10.05 [23:46:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 6.58, 6.34 [23:47:11] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [23:47:56] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.12, 10.84, 11.33 [23:48:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.20, 6.34, 6.28 [23:49:11] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.680 second response time [23:49:47] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.04, 4.62, 3.43 [23:49:56] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.66, 10.18, 11.07 [23:50:32] PROBLEM - ldap141 Puppet on ldap141 is WARNING: WARNING: Puppet is currently disabled, message: paladoxz, last run 17 minutes ago with 0 failures [23:50:43] PROBLEM - ldap141 LDAP on ldap141 is CRITICAL: Could not bind to the LDAP server [23:51:44] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.47, 3.77, 3.26 [23:53:09] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.60, 8.67, 9.88 [23:53:44] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.52, 2.59, 2.88 [23:54:34] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.17, 10.75, 10.41 [23:54:43] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:29] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 11.12, 10.44 [23:56:33] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.52, 11.37, 10.68 [23:56:43] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2681f91338f6...802c7f1923ab [23:56:44] [02miraheze/dns] 07paladox 03802c7f1 - Add ldap to miraheze pointing at ldap171 [23:57:04] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.12, 10.08, 10.18 [23:57:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.25, 10.39, 10.25 [23:58:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/802c7f1923ab...0b96ff7b0c9f [23:58:37] [02miraheze/dns] 07paladox 030b96ff7 - Fix [23:59:04] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.60, 12.12, 10.91