[00:00:15] PROBLEM - mon141 Puppet on mon141 is UNKNOWN: NRPE: Unable to read output [00:00:19] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.27, 11.14, 10.04 [00:00:29] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 13.73, 22.22, 13.00 [00:01:25] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.01, 1.39, 0.95 [00:01:41] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.25, 7.92, 7.78 [00:02:19] PROBLEM - mon141 Puppet on mon141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 1 failures [00:02:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.32, 6.03, 2.88 [00:03:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.94, 7.88, 7.78 [00:03:43] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.04, 9.37, 8.62 [00:04:07] PROBLEM - mon141 IRC-Discord Relay Bot on mon141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'relaybot' [00:04:16] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.525 second response time [00:04:34] PROBLEM - mon141 HTTPS on mon141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.398 second response time [00:04:54] PROBLEM - mon141 IRCEcho on mon141 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/local/bin/ircecho' [00:05:14] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 10.10, 10.07, 9.86 [00:05:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.66, 8.16, 7.90 [00:06:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4922e602fec...a3c0bb69ce67 [00:06:02] [02miraheze/puppet] 07Universal-Omega 03a3c0bb6 - logsalmsg: use mon181 [00:07:58] PROBLEM - ping6 on mwtask141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.81 ms [00:13:38] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bad85762ae01...8f039b17cd59 [00:13:41] [02miraheze/mw-config] 07MacFan4000 038f039b1 - switch RC to mon181 [00:13:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mon141 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/15bcc12cf924 [00:13:52] [02miraheze/puppet] 07Universal-Omega 0315bcc12 - Remove mon141 [00:13:55] [02puppet] 07Universal-Omega created branch 03remove-mon141 - 13https://github.com/miraheze/puppet [00:13:56] [02puppet] 07Universal-Omega opened pull request 03#3614: Remove mon141 - 13https://github.com/miraheze/puppet/pull/3614 [00:14:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mon141 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/15bcc12cf924...e21687d869ba [00:14:28] [02miraheze/puppet] 07Universal-Omega 03e21687d - Delete hieradata/hosts/mon141.yaml [00:14:31] [02puppet] 07Universal-Omega synchronize pull request 03#3614: Remove mon141 - 13https://github.com/miraheze/puppet/pull/3614 [00:14:38] miraheze/mw-config - MacFan4000 the build passed. [00:15:52] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [00:16:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±0] 13https://github.com/miraheze/mw-config/compare/8f039b17cd59...1c418f2c0bc9 [00:16:33] [02miraheze/mw-config] 07Universal-Omega 031c418f2 - Use mon181 for IRC RC feed [00:17:07] !log [macfan@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.wikitide.net [00:17:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mon141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e21687d869ba...b6ac3f1cc017 [00:17:22] [02miraheze/puppet] 07Universal-Omega 03b6ac3f1 - Remove from NRPE [00:17:25] [02puppet] 07Universal-Omega synchronize pull request 03#3614: Remove mon141 - 13https://github.com/miraheze/puppet/pull/3614 [00:17:36] miraheze/mw-config - Universal-Omega the build passed. [00:20:32] https://github.com/miraheze/mw-config/commit/1c418f2c0bc9bc2579291da16001cfe334a1fced how did that happen lol [00:21:49] !log [macfan@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [00:22:08] !log [macfan@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 18s [00:22:39] well with force it works, just the canary checks that fail [00:23:41] Yeah because those servers aren't in Varnish yet so proxy wont work. [00:27:10] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [00:27:12] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [00:28:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e478854773d4 [00:28:05] [02miraheze/puppet] 07paladox 03e478854 - mon181: switch db to db182 [00:28:06] [02puppet] 07paladox opened pull request 03#3615: mon181: switch db to db182 - 13https://github.com/miraheze/puppet/pull/3615 [00:28:09] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [00:28:11] [02puppet] 07paladox closed pull request 03#3615: mon181: switch db to db182 - 13https://github.com/miraheze/puppet/pull/3615 [00:28:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a3c0bb69ce67...b1de23779852 [00:28:15] [02miraheze/puppet] 07paladox 03b1de237 - mon181: switch db to db182 (#3615) [00:28:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [00:28:18] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [00:29:27] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.48, 3.60, 4.91 [00:29:29] PROBLEM - mw182 MediaWiki Rendering on mw182 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.097 second response time [00:29:33] PROBLEM - ping6 on db112 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.67 ms [00:29:38] PROBLEM - mw181 MediaWiki Rendering on mw181 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 6693 bytes in 0.085 second response time [00:29:40] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:29:51] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:30:01] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.09, 6.32, 7.11 [00:30:02] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:30:04] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 28522 Threads: 1 Questions: 1906 Slow queries: 0 Opens: 108 Open tables: 26 Queries per second avg: 0.066 [00:30:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.76, 7.10, 7.40 [00:30:22] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:30:28] PROBLEM - mon141 php-fpm on mon141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:30:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1de23779852...48aa2404f26e [00:30:40] [02miraheze/puppet] 07paladox 0348aa240 - Update mon181.yaml [00:30:43] PROBLEM - mon141 SSH on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:30:49] PROBLEM - ping6 on mwtask141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [00:30:56] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:30:56] PROBLEM - mon141 icinga.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:00] PROBLEM - Host mon141 is DOWN: PING CRITICAL - Packet loss = 100% [00:33:49] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.849 second response time [00:34:07] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.590 second response time [00:34:33] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.646 second response time [00:34:45] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.548 second response time [00:35:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03icinga-no-db-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bb39ce0176fe [00:35:25] [02miraheze/puppet] 07Universal-Omega 03bb39ce0 - icinga: remove SSL for db [00:35:28] [02puppet] 07Universal-Omega created branch 03icinga-no-db-ssl - 13https://github.com/miraheze/puppet [00:35:30] [02puppet] 07Universal-Omega opened pull request 03#3616: icinga: remove SSL for db - 13https://github.com/miraheze/puppet/pull/3616 [00:36:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03icinga-no-db-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bb39ce0176fe...ab3b0a39dc71 [00:36:37] [02miraheze/puppet] 07Universal-Omega 03ab3b0a3 - No SSL [00:36:39] [02puppet] 07Universal-Omega synchronize pull request 03#3616: icinga: remove SSL for db - 13https://github.com/miraheze/puppet/pull/3616 [00:38:27] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.88, 3.07, 2.53 [00:39:46] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:39:47] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:40:02] PROBLEM - ping6 on swiftobject121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.07 ms [00:40:06] [02puppet] 07Universal-Omega closed pull request 03#3616: icinga: remove SSL for db - 13https://github.com/miraheze/puppet/pull/3616 [00:40:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/48aa2404f26e...a9cf0fd8288e [00:40:10] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 3.65, 2.88 [00:40:11] [02miraheze/puppet] 07Universal-Omega 03a9cf0fd - icinga: remove SSL for db (#3616) [00:40:14] [02miraheze/puppet] 07Universal-Omega deleted branch 03icinga-no-db-ssl [00:40:15] [02puppet] 07Universal-Omega deleted branch 03icinga-no-db-ssl - 13https://github.com/miraheze/puppet [00:40:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.48, 7.53, 7.30 [00:40:22] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.86, 2.94, 2.55 [00:41:40] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.558 second response time [00:41:43] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [00:42:01] PROBLEM - ping6 on swiftobject121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.67 ms [00:42:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.37, 3.60, 2.97 [00:42:17] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.62, 7.49, 7.31 [00:42:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.80, 7.71, 7.43 [00:44:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.32, 7.50, 7.37 [00:44:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.51, 7.91, 7.49 [00:45:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.06, 7.91, 7.55 [00:46:41] [02puppet] 07paladox closed pull request 03#3614: Remove mon141 - 13https://github.com/miraheze/puppet/pull/3614 [00:46:43] [02miraheze/puppet] 07paladox deleted branch 03remove-mon141 [00:46:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±2] 13https://github.com/miraheze/puppet/compare/a9cf0fd8288e...aa6e91ac3e8a [00:46:47] [02miraheze/puppet] 07Universal-Omega 03aa6e91a - Remove mon141 (#3614) [00:46:49] [02puppet] 07paladox deleted branch 03remove-mon141 - 13https://github.com/miraheze/puppet [00:47:33] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.85, 3.29, 3.03 [00:48:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.65, 6.87, 6.22 [00:49:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.99, 7.94, 7.64 [00:50:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.46, 5.99, 5.98 [00:50:25] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [00:51:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.00, 8.06, 7.72 [00:51:30] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 4.05, 3.41 [00:51:37] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 8418 MB (6% inode=99%); [00:52:22] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.536 second response time [00:53:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.25, 7.55, 7.59 [00:53:30] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.82, 3.52, 3.29 [00:54:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.45, 7.68, 7.84 [00:55:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.97, 7.20, 7.92 [00:55:28] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 4.02, 3.51 [00:57:27] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.34, 3.73, 3.46 [00:58:51] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 130.47 ms [00:59:11] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.42, 7.47 [00:59:29] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.35, 4.62, 3.79 [01:00:50] PROBLEM - ping6 on prometheus131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.99 ms [01:01:09] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.94, 7.36, 7.43 [01:03:11] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [01:03:20] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.35, 7.70, 7.81 [01:03:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.44, 8.55, 8.06 [01:05:20] PROBLEM - ping6 on db112 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.97 ms [01:05:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.11, 3.77, 3.76 [01:07:19] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.74, 7.73, 7.79 [01:07:22] PROBLEM - ping6 on db112 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.14 ms [01:09:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.05, 7.65, 7.74 [01:09:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.27, 4.17, 3.93 [01:10:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.35, 7.07, 7.21 [01:11:32] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.13, 3.44, 3.70 [01:12:50] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.70, 7.19, 7.23 [01:13:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.52, 7.86, 7.82 [01:13:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.64, 7.93, 7.98 [01:15:23] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.20 ms [01:15:45] [02dns] 07Universal-Omega created branch 03remove-mon141 - 13https://github.com/miraheze/dns [01:15:46] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-mon141 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/499e1c82ca57 [01:15:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.19, 7.97, 7.98 [01:15:47] [02miraheze/dns] 07Universal-Omega 03499e1c8 - Remove mon141 from DNS [01:15:49] [02dns] 07Universal-Omega opened pull request 03#456: Remove mon141 from DNS - 13https://github.com/miraheze/dns/pull/456 [01:16:05] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.80 ms [01:16:36] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-mon141 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/499e1c82ca57...9e75563ad586 [01:16:39] [02miraheze/dns] 07Universal-Omega 039e75563 - Remove [01:16:39] [02dns] 07Universal-Omega synchronize pull request 03#456: Remove mon141 from DNS - 13https://github.com/miraheze/dns/pull/456 [01:16:50] [02dns] 07Universal-Omega closed pull request 03#456: Remove mon141 from DNS - 13https://github.com/miraheze/dns/pull/456 [01:16:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/d5046dc897ef...cb95758c4895 [01:16:55] [02miraheze/dns] 07Universal-Omega 03cb95758 - Remove mon141 from DNS (#456) [01:16:57] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-mon141 [01:16:59] [02dns] 07Universal-Omega deleted branch 03remove-mon141 - 13https://github.com/miraheze/dns [01:17:27] PROBLEM - ping6 on db131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.78 ms [01:17:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.59, 7.66, 7.88 [01:17:54] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:17:59] PROBLEM - ping6 on swiftac111 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.93 ms [01:18:01] [02miraheze/mediawiki-repos] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/5536a8835017...2da6c29453e0 [01:18:04] PROBLEM - ping6 on graylog131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [01:18:04] [02miraheze/mediawiki-repos] 07paladox 032da6c29 - Update Nimbus to master [01:18:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.48, 7.83, 7.48 [01:18:56] PROBLEM - mon181 Puppet on mon181 is UNKNOWN: NRPE: Unable to read output [01:19:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.34, 7.85, 7.83 [01:19:30] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.53, 3.00, 3.39 [01:19:59] PROBLEM - ping6 on swiftac111 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.90 ms [01:20:17] !log [paladox@mwtask141] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:20:22] !log [paladox@mwtask141] finished deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 5s [01:20:23] !log [paladox@mwtask141] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:20:59] PROBLEM - ping6 on cloud14 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.99 ms [01:21:23] !log [paladox@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [01:22:08] !log [paladox@mwtask141] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:22:10] !log [paladox@mwtask141] finished deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 2s [01:22:11] !log [paladox@mwtask141] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:23:00] PROBLEM - ping6 on cloud14 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [01:23:41] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [01:23:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.41, 7.93, 7.90 [01:24:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.74, 7.97, 7.73 [01:25:01] !log [paladox@mwtask181] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mwtask181, mw181, mw182] [01:25:03] !log [paladox@mwtask181] finished deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mwtask181, mw181, mw182] - SUCCESS in 1s [01:25:04] !log [paladox@mwtask181] starting deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mwtask181, mw181, mw182] [01:25:40] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.66 ms [01:25:50] !log [paladox@mwtask141] finished deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 222s [01:26:00] !log [paladox@mwtask181] finished deploy of {'world': True, 'force': True, 'versions': ['1.40']} to [mwtask181, mw181, mw182] - SUCCESS in 59s [01:26:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1c418f2c0bc9...709b9ec53649 [01:26:33] [02miraheze/mw-config] 07Universal-Omega 03709b9ec - Fix [01:26:56] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [01:26:57] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [01:27:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.73, 7.99, 7.90 [01:27:25] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:27:30] miraheze/mw-config - Universal-Omega the build passed. [01:27:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.18, 7.88, 7.91 [01:28:44] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 78s [01:29:24] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [01:30:31] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.07 ms [01:30:35] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:31:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/709b9ec53649...130d5b3e675e [01:31:02] [02miraheze/mw-config] 07Universal-Omega 03130d5b3 - Revert [01:31:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.27, 8.06, 7.93 [01:31:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-5019-patch-104 [+0/-0/±3] 13https://github.com/miraheze/mw-config/commit/77de843cf826 [01:31:34] [02miraheze/mw-config] 07Universal-Omega 0377de843 - Revert "Wikibase client and repo settings that are currently missing (#5019)" [01:31:35] [02mw-config] 07Universal-Omega created branch 03revert-5019-patch-104 - 13https://github.com/miraheze/mw-config [01:31:36] [02mw-config] 07Universal-Omega opened pull request 03#5431: Revert "Wikibase client and repo settings that are currently missing" - 13https://github.com/miraheze/mw-config/pull/5431 [01:31:38] !log [@mwtask181] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:31:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.76, 8.70, 8.21 [01:31:49] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw131.miraheze.org [01:32:01] miraheze/mw-config - Universal-Omega the build passed. [01:32:30] PROBLEM - ping6 on prometheus131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.98 ms [01:32:34] miraheze/mw-config - Universal-Omega the build passed. [01:32:50] [02mw-config] 07Universal-Omega edited pull request 03#5431: Revert "Wikibase client and repo settings that are currently missing" - 13https://github.com/miraheze/mw-config/pull/5431 [01:33:11] [02mw-config] 07Universal-Omega closed pull request 03#5431: Revert "Wikibase client and repo settings that are currently missing" - 13https://github.com/miraheze/mw-config/pull/5431 [01:33:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/130d5b3e675e...4ff63e19ca12 [01:33:15] [02miraheze/mw-config] 07Universal-Omega 034ff63e1 - Revert "Wikibase client and repo settings that are currently missing" (#5431) [01:33:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-5019-patch-104 [01:33:19] [02mw-config] 07Universal-Omega deleted branch 03revert-5019-patch-104 - 13https://github.com/miraheze/mw-config [01:33:33] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:33:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.62, 7.46, 7.82 [01:34:00] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:34:02] miraheze/mw-config - Universal-Omega the build passed. [01:34:58] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:35:18] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'versions': ['1.40']} to [mwtask181] [01:35:21] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'versions': ['1.40']} to [mwtask181] - SUCCESS in 2s [01:35:24] PROBLEM - mwtask181 Puppet on mwtask181 is UNKNOWN: NRPE: Unable to read output [01:35:27] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mwtask181] [01:35:29] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 2s [01:35:30] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [01:35:38] !log [paladox@mwtask181] starting deploy of {'config': True, 'versions': ['1.40']} to [mwtask181] [01:35:39] !log [paladox@mwtask181] finished deploy of {'config': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 1s [01:35:46] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.01, 4.45, 3.70 [01:35:56] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.550 second response time [01:35:58] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:36:41] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [01:36:51] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.33, 10.39, 8.15 [01:37:08] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.wikitide.net [01:37:15] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:37:24] !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mwtask181] [01:37:28] !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 4s [01:37:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.25, 7.57, 7.76 [01:37:55] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 39s [01:37:57] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.520 second response time [01:38:41] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:38:51] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.90, 11.85, 8.93 [01:39:34] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:39:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.33, 7.04, 7.54 [01:40:00] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:40:05] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:41:25] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.63, 12.87, 9.18 [01:41:31] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw142 mw133 [01:41:31] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.511 second response time [01:41:56] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.570 second response time [01:42:03] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.519 second response time [01:42:32] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.08, 10.54, 8.58 [01:43:20] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.20, 11.12, 8.95 [01:43:31] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:44:25] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:44:30] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.13, 8.87, 8.22 [01:44:37] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [01:44:51] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.20, 10.14, 9.57 [01:45:16] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.12, 8.63, 8.29 [01:45:30] PROBLEM - mw131 MediaWiki Rendering on mw131 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [01:45:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.06, 8.02, 7.76 [01:46:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.43, 7.34 [01:46:36] PROBLEM - ping6 on mem131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.03 ms [01:47:31] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.682 second response time [01:47:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.82, 2.68, 3.70 [01:47:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.75, 7.69, 7.68 [01:47:53] !log [universalomega@mwtask141] starting deploy of {'files': '1.40/includes/user/UserGroupManager.php', 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:48:04] !log [universalomega@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.wikitide.net [01:48:11] !log [universalomega@mwtask141] starting deploy of {'force': True, 'files': '1.40/includes/user/UserGroupManager.php', 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [01:48:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.90, 7.14, 7.24 [01:48:22] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.969 second response time [01:48:25] !log [universalomega@mwtask141] finished deploy of {'force': True, 'files': '1.40/includes/user/UserGroupManager.php', 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 13s [01:49:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.34, 1.96, 3.32 [01:50:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.82, 7.61, 7.39 [01:52:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.96, 7.75, 7.47 [01:52:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.51, 7.94 [01:53:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.43, 7.15, 7.85 [01:55:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fcdd044b7de3 [01:55:44] [02miraheze/puppet] 07Universal-Omega 03fcdd044 - Add mw(task)?18* Varnish backends [01:55:47] [02puppet] 07Universal-Omega created branch 03backends - 13https://github.com/miraheze/puppet [01:55:50] [02puppet] 07Universal-Omega opened pull request 03#3617: Add mw(task)?18* Varnish backends - 13https://github.com/miraheze/puppet/pull/3617 [01:56:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 8.18, 8.09 [01:56:58] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [01:56:59] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 0s [01:58:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.62, 7.89 [01:59:02] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [01:59:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03backends [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fcdd044b7de3...6f342e61c758 [01:59:10] [02miraheze/puppet] 07Universal-Omega 036f342e6 - Update ACLs [01:59:12] [02puppet] 07Universal-Omega synchronize pull request 03#3617: Add mw(task)?18* Varnish backends - 13https://github.com/miraheze/puppet/pull/3617 [02:01:06] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:01:12] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.85 ms [02:02:04] [02puppet] 07paladox closed pull request 03#3617: Add mw(task)?18* Varnish backends - 13https://github.com/miraheze/puppet/pull/3617 [02:02:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/aa6e91ac3e8a...fd12a7a17741 [02:02:08] [02miraheze/puppet] 07Universal-Omega 03fd12a7a - Add mw(task)?18* Varnish backends (#3617) [02:02:10] [02miraheze/puppet] 07paladox deleted branch 03backends [02:02:13] [02puppet] 07paladox deleted branch 03backends - 13https://github.com/miraheze/puppet [02:02:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 7.99, 7.99 [02:02:50] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [02:03:10] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.38 ms [02:03:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.14, 7.34, 7.52 [02:04:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.57, 7.84 [02:05:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.28, 7.20, 7.08 [02:06:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.60, 8.13, 7.60 [02:06:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.81, 8.55, 8.19 [02:08:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.64, 7.81, 7.54 [02:09:25] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw181 mw182 [02:09:31] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw181 mw182 [02:09:41] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.70 ms [02:09:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.83, 7.95, 7.43 [02:10:50] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw181 mw182 [02:11:40] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [02:12:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.89, 7.68, 7.49 [02:13:14] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw181 mw182 [02:13:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.19, 7.83, 7.47 [02:14:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 13.02, 8.49, 6.79 [02:14:37] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.83 ms [02:15:15] PROBLEM - cp34 Varnish Backends on cp34 is WARNING: No backends detected. If this is an error, see readme.txt [02:17:00] PROBLEM - ping6 on mem131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.00 ms [02:17:20] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw181 mw182 [02:17:51] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.66, 10.39, 8.28 [02:18:16] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:18:39] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.59, 10.70, 8.44 [02:18:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.74, 3.32, 2.74 [02:19:00] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 10.40, 7.89 [02:19:40] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 9.99, 6.58, 3.99 [02:19:48] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.45, 10.49, 8.59 [02:20:36] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.58, 10.00, 8.47 [02:20:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.73, 2.96 [02:20:55] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.13, 10.33, 8.17 [02:21:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-wt [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/64d992597ebf [02:21:12] [02miraheze/puppet] 07Universal-Omega 0364d9925 - varnish: support wikitide.net for backends [02:21:15] [02puppet] 07Universal-Omega created branch 03varnish-wt - 13https://github.com/miraheze/puppet [02:21:17] [02puppet] 07Universal-Omega opened pull request 03#3618: varnish: support wikitide.net for backends - 13https://github.com/miraheze/puppet/pull/3618 [02:22:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.83, 10.18, 8.37 [02:23:41] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.13, 11.23, 9.27 [02:24:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-wt [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/64d992597ebf...401853732006 [02:24:42] [02miraheze/puppet] 07Universal-Omega 034018537 - Update [02:24:45] [02puppet] 07Universal-Omega synchronize pull request 03#3618: varnish: support wikitide.net for backends - 13https://github.com/miraheze/puppet/pull/3618 [02:25:09] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 18 backends are healthy [02:25:31] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:39] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.18, 10.11, 9.10 [02:26:14] [02puppet] 07Universal-Omega closed pull request 03#3618: varnish: support wikitide.net for backends - 13https://github.com/miraheze/puppet/pull/3618 [02:26:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/fd12a7a17741...b5b6a93010c0 [02:26:18] [02miraheze/puppet] 07Universal-Omega 03b5b6a93 - varnish: support wikitide.net for backends (#3618) [02:26:20] [02miraheze/puppet] 07Universal-Omega deleted branch 03varnish-wt [02:26:22] [02puppet] 07Universal-Omega deleted branch 03varnish-wt - 13https://github.com/miraheze/puppet [02:27:03] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 18 backends are healthy [02:27:28] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.687 second response time [02:27:51] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.69 ms [02:27:57] PROBLEM - ping6 on os141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.79 ms [02:28:33] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.10, 11.09, 9.19 [02:28:35] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 18 backends are healthy [02:28:59] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.49 ms [02:29:39] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.83, 11.93, 10.08 [02:29:43] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.92, 11.48, 9.43 [02:29:53] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.60 ms [02:29:58] PROBLEM - ping6 on os141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [02:30:27] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.65, 9.86, 8.98 [02:30:44] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:31:02] PROBLEM - ping6 on db131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.83 ms [02:31:37] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.45, 10.56, 9.34 [02:31:39] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.47, 10.46, 9.76 [02:33:32] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.33, 9.90, 9.24 [02:33:39] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.25, 9.24, 9.41 [02:33:54] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:37] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:33] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 4.53, 7.12, 6.90 [02:35:43] PROBLEM - ping6 on bast141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.05 ms [02:36:33] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.81 ms [02:37:32] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.59, 6.62, 6.74 [02:37:45] PROBLEM - ping6 on bast141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.06 ms [02:38:03] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.933 second response time [02:38:18] PROBLEM - ping6 on swiftobject122 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.96 ms [02:38:20] PROBLEM - swiftobject111 PowerDNS Recursor on swiftobject111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:38:34] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.28, 3.80, 3.83 [02:38:37] PROBLEM - ping6 on graylog131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.82 ms [02:38:53] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.311 second response time [02:38:53] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 18 backends are healthy [02:40:20] PROBLEM - ping6 on swiftobject122 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.63 ms [02:42:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b5b6a93010c0...f4c6e38a15dc [02:42:04] [02miraheze/puppet] 07Universal-Omega 03f4c6e38 - Use mwtask181 in tests [02:42:37] RECOVERY - swiftobject111 PowerDNS Recursor on swiftobject111 is OK: DNS OK: 8.371 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:43:06] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 11.35, 9.93 [02:43:34] miraheze/puppet - Universal-Omega the build passed. [02:44:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f4c6e38a15dc...c1dd09307548 [02:44:26] [02miraheze/puppet] 07Universal-Omega 03c1dd093 - Update mwupgradetool [02:45:01] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.95, 10.78, 9.89 [02:45:16] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 10.80, 9.77 [02:45:40] miraheze/puppet - Universal-Omega the build passed. [02:46:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ccbfc48717b1 [02:46:22] [02miraheze/puppet] 07Universal-Omega 03ccbfc48 - Remove mwtask141 [02:46:24] [02puppet] 07Universal-Omega created branch 03remove-mwtask141 - 13https://github.com/miraheze/puppet [02:46:26] [02puppet] 07Universal-Omega opened pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [02:46:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/ccbfc48717b1...940ead47233a [02:47:00] [02miraheze/puppet] 07Universal-Omega 03940ead4 - Delete hieradata/hosts/mwtask141.yaml [02:47:01] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [02:47:14] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.52, 10.25, 9.67 [02:48:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/940ead47233a...384478b71388 [02:48:09] [02miraheze/puppet] 07Universal-Omega 03384478b - Remove [02:48:12] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [02:49:52] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.19, 11.06, 8.13 [02:50:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/384478b71388...8a62a59778b1 [02:50:07] [02miraheze/puppet] 07Universal-Omega 038a62a59 - Remove [02:50:09] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [02:50:55] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.62, 9.69, 9.82 [02:51:10] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.53, 8.54, 9.17 [02:51:24] miraheze/puppet - Universal-Omega the build passed. [02:51:39] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.88 ms [02:51:43] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.10 ms [02:51:46] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.78, 9.72, 7.98 [02:53:21] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.77 ms [02:53:41] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.76 ms [02:53:43] PROBLEM - ping6 on ns2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.13 ms [02:54:12] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/updateSpecialPages.php --wiki=blizzardwiki (END - exit=0) [02:54:55] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.69, 10.89, 10.21 [02:55:23] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.38 ms [02:56:55] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.21, 9.78, 9.88 [02:57:16] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.49, 10.15, 9.62 [03:00:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.69, 6.99, 7.98 [03:00:34] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.64, 3.04, 3.32 [03:01:09] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.69, 9.32, 9.43 [03:02:30] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.002027481794 secs [03:02:44] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [03:02:57] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [03:04:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.14, 7.32, 7.86 [03:04:21] PROBLEM - ping6 on bast141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.11 ms [03:06:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.85, 6.95, 7.67 [03:06:24] PROBLEM - ping6 on bast141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.93 ms [03:09:19] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [03:11:04] MacFan4000: FYI I fixed it so mw181/182 will pass debug checks now. [03:11:19] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:14:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.39, 7.21, 7.43 [03:17:27] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=shelbyvillewikiwiki --no-updates --username-prefix=w /home/universalomega/ImportDump/shelbyvillewikiwiki-20240109235116.xml (START) [03:18:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.22, 7.13, 7.37 [03:21:00] PROBLEM - ping6 on mw132 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.30 ms [03:21:19] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.22 ms [03:22:59] PROBLEM - ping6 on mw132 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [03:23:20] PROBLEM - ping6 on mem131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.13 ms [03:23:46] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.97 ms [03:25:19] !log test [03:25:49] PROBLEM - ping6 on graylog131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.80 ms [03:26:04] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.35 ms [03:26:41] !log test [03:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:06] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.75 ms [03:28:24] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.31, 9.88, 8.97 [03:29:20] PROBLEM - mon181 IRC Log Bot on mon181 is CRITICAL: PROCS CRITICAL: 0 processes with args 'adminlogbot.py' [03:29:49] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [03:30:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.43, 8.98, 8.73 [03:30:47] !log test (IRC) [03:30:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:57] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:31:14] !log test (Discord) [03:31:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:20] RECOVERY - mon181 IRC Log Bot on mon181 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [03:32:16] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.49, 6.54, 6.79 [03:32:44] PROBLEM - ping6 on db101 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.05 ms [03:32:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:32:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c1dd09307548...be07563cee3a [03:33:00] [02miraheze/puppet] 07Universal-Omega 03be07563 - adminlogbot: fix logging from IRC [03:33:19] @pixldev https://github.com/miraheze/puppet/commit/be07563cee3a0927baf62d02dae4521791273e59 that fixed it [03:34:31] miraheze/puppet - Universal-Omega the build passed. [03:34:43] PROBLEM - ping6 on db101 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.00 ms [03:49:09] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.60 ms [03:49:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.42, 6.99, 6.68 [03:50:47] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=shelbyvillewikiwiki --no-updates --username-prefix=w /home/universalomega/ImportDump/shelbyvillewikiwiki-20240109235116.xml (END - exit=0) [03:50:48] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=shelbyvillewikiwiki (START) [03:50:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:11] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.62 ms [03:51:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.25, 7.67, 6.95 [03:52:49] PROBLEM - ping6 on os141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.77 ms [03:53:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.95, 7.35, 6.92 [03:54:50] PROBLEM - ping6 on os141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.48 ms [03:59:09] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.02, 6.36, 6.70 [04:01:20] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:03:08] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.70, 6.80, 6.81 [04:04:44] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [04:08:10] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.04 ms [04:10:12] PROBLEM - ping6 on prometheus131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.64 ms [04:10:33] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.78 ms [04:11:07] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.78, 6.30, 6.62 [04:12:35] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.91 ms [04:24:37] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.85 ms [04:25:07] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.28, 7.14, 6.80 [04:26:37] PROBLEM - ping6 on mw141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.98 ms [04:27:06] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.31, 6.72, 6.68 [04:32:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:32:57] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [04:39:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.06, 6.66, 6.57 [04:40:29] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.96 ms [04:41:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.25, 7.29, 6.81 [04:42:31] PROBLEM - ping6 on cloud11 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.06 ms [04:43:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.32, 6.87, 6.72 [04:45:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.58, 7.67, 7.01 [04:47:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.90, 7.92, 7.20 [04:50:25] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:52:22] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.626 second response time [04:53:36] PROBLEM - ping6 on swiftproxy131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.62 ms [04:55:37] PROBLEM - ping6 on swiftproxy131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.59 ms [04:57:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.56, 6.25, 6.70 [05:00:57] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:11:19] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.13, 9.19, 7.88 [05:11:21] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.73 ms [05:13:19] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.72, 8.88, 7.95 [05:13:23] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.80 ms [05:18:34] PROBLEM - ping6 on cloud13 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.73 ms [05:20:37] PROBLEM - ping6 on cloud13 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [05:22:54] PROBLEM - ping6 on matomo121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.44 ms [05:24:55] PROBLEM - ping6 on matomo121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.83 ms [05:28:58] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.48, 1.88, 1.16 [05:29:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.54, 7.62, 7.98 [05:30:58] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.97, 1.57, 1.13 [05:31:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.41, 7.96, 8.06 [05:33:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.88, 7.83, 7.99 [05:34:44] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [05:35:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.79, 7.10, 7.87 [05:37:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.63, 7.83, 7.93 [05:39:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.16, 7.44, 7.81 [05:39:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.70, 7.62, 7.83 [05:41:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.15, 8.59, 8.17 [05:42:29] PROBLEM - ping6 on bast121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.04 ms [05:44:32] PROBLEM - ping6 on bast121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.04 ms [05:45:13] PROBLEM - ping6 on matomo121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.76 ms [05:45:14] PROBLEM - ping6 on cloud13 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.71 ms [05:47:15] PROBLEM - ping6 on matomo121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.84 ms [05:47:17] PROBLEM - ping6 on cloud13 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [05:49:05] PROBLEM - mon181 Puppet on mon181 is UNKNOWN: NRPE: Unable to read output [05:49:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.39, 7.91, 8.00 [05:51:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.29, 8.51, 8.21 [05:52:58] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.75, 1.29, 1.08 [05:54:58] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.82, 1.11, 1.05 [05:57:26] PROBLEM - ping6 on matomo121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.81 ms [05:57:40] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [05:57:42] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.84 ms [05:59:27] PROBLEM - ping6 on matomo121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.77 ms [05:59:43] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.83 ms [06:00:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [06:02:57] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [06:10:14] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.74 ms [06:10:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=shelbyvillewikiwiki (END - exit=0) [06:10:52] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=shelbyvillewikiwiki --active --update (END - exit=0) [06:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:16] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.70 ms [06:16:27] PROBLEM - ping6 on swiftproxy131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.53 ms [06:17:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:18:28] PROBLEM - ping6 on swiftproxy131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.57 ms [06:19:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.76, 7.40, 7.86 [06:21:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.59, 7.63, 8.00 [06:25:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.90, 8.28, 8.17 [06:25:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.95, 8.26, 8.04 [06:27:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.91, 7.43, 7.91 [06:27:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.69, 7.87, 7.92 [06:29:12] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:29:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.73, 8.20, 8.03 [06:29:59] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=4chanintwiki --no-updates /home/reception/ImportDump/4chanintwiki-20240109215707.xml (START) [06:30:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:30:57] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:31:51] PROBLEM - ping6 on swiftobject121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.12 ms [06:33:17] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [06:33:17] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.00 ms [06:33:17] PROBLEM - ping6 on mem141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.22 ms [06:33:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.08, 7.20, 7.75 [06:33:55] PROBLEM - ping6 on swiftobject121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.05 ms [06:33:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.84, 8.33, 8.10 [06:35:05] PROBLEM - ping6 on prometheus131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.12 ms [06:35:10] PROBLEM - ping6 on mem141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.04 ms [06:35:31] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/migrateRevisionCommentTemp.php --wiki=projectstardustwiki (END - exit=0) [06:35:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:35:54] PROBLEM - ping6 on mw143 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.87 ms [06:37:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.65, 7.70, 7.78 [06:37:51] PROBLEM - ping6 on mw143 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.92 ms [06:39:39] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=4chanintwiki --no-updates /home/reception/ImportDump/4chanintwiki-20240109215707.xml (END - exit=0) [06:39:40] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=4chanintwiki (START) [06:39:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:39:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:39:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.68, 7.56, 7.71 [06:41:25] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=4chanintwiki (END - exit=0) [06:41:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=4chanintwiki --active --update (END - exit=0) [06:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:41:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.84, 7.84, 7.77 [06:43:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.66, 7.55, 7.56 [06:45:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.30, 7.47, 7.53 [06:48:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.45, 6.88, 7.69 [06:49:55] PROBLEM - ping6 on os131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.11 ms [06:51:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.39, 7.69, 7.88 [06:51:55] PROBLEM - ping6 on os131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.65 ms [06:53:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.09, 7.44, 7.61 [06:54:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.30, 7.94 [06:55:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.28, 7.79, 7.65 [06:56:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.73, 7.97, 8.09 [06:57:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.38, 7.58, 7.70 [06:57:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.41, 7.60, 7.64 [06:58:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.43, 7.88 [06:59:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.59, 7.86, 7.72 [07:00:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:01:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.10, 7.56, 7.68 [07:01:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.90, 8.04, 7.82 [07:03:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.17, 7.26, 7.55 [07:03:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.94, 8.11, 7.80 [07:04:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.79, 8.02, 7.98 [07:05:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.31, 7.75, 7.72 [07:07:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.59, 7.68, 7.70 [07:07:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.49, 7.43, 7.51 [07:09:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 7.27, 7.43 [07:10:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.70, 7.95 [07:12:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.80, 8.89, 8.37 [07:14:30] PROBLEM - ping6 on swiftobject112 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.91 ms [07:15:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.47, 7.77, 7.71 [07:16:31] PROBLEM - ping6 on swiftobject112 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.99 ms [07:19:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.69, 7.80, 7.75 [07:21:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.10, 7.96, 7.82 [07:21:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.10, 7.34, 7.40 [07:25:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.05, 6.57, 5.95 [07:27:39] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.44, 6.45, 5.99 [07:35:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.21, 7.05, 7.98 [07:35:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.32, 6.95, 7.81 [07:39:18] PROBLEM - ping6 on cloud10 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.76 ms [07:39:28] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.54, 2.30, 1.33 [07:39:31] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.99 ms [07:39:42] PROBLEM - ping6 on ldap141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.06 ms [07:41:17] PROBLEM - ping6 on cloud10 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.73 ms [07:41:27] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.25, 2.04, 1.37 [07:41:32] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.77 ms [07:41:43] PROBLEM - ping6 on ldap141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.12 ms [07:41:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.55, 7.35, 7.75 [07:43:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.75, 7.43, 7.73 [07:43:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.20, 7.70, 7.78 [07:45:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.67, 8.01, 7.92 [07:45:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.94, 7.76, 7.79 [07:45:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.83, 7.31, 7.88 [07:47:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.09, 7.81, 7.87 [07:57:49] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [07:57:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.43, 8.01, 7.96 [07:58:31] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1012709141 secs [07:58:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.81, 6.70, 7.69 [07:59:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.82, 7.89, 7.92 [08:02:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 7.36, 7.66 [08:03:31] PROBLEM - ping6 on swiftac111 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [08:05:31] PROBLEM - ping6 on swiftac111 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [08:05:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.56, 8.29, 7.96 [08:06:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.37, 7.61 [08:08:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.72, 7.70 [08:10:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.21, 7.51 [08:11:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.50, 7.80, 7.93 [08:12:16] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.37 ms [08:12:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.30, 7.60, 7.93 [08:14:16] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.14 ms [08:14:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.22, 7.79, 7.96 [08:15:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.34, 8.14, 7.57 [08:16:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.62, 7.33, 7.76 [08:18:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.64, 8.43, 7.90 [08:19:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.65, 7.88, 7.61 [08:19:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.20, 7.61, 7.19 [08:21:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.18, 7.46, 7.19 [08:22:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.56, 7.75, 7.79 [08:23:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.32, 7.93, 7.68 [08:23:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.53, 7.24 [08:23:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.64, 7.68, 7.66 [08:25:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.34, 7.36, 7.20 [08:27:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.23, 7.68, 7.67 [08:29:21] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:29:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.64, 7.31, 7.61 [08:32:17] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.11 ms [08:34:17] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.15 ms [08:34:39] PROBLEM - ping6 on db121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.16 ms [08:34:44] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [08:35:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.72, 7.34, 7.38 [08:35:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 13.44, 9.29, 8.25 [08:36:40] PROBLEM - ping6 on db121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.13 ms [08:37:24] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [08:37:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.66, 6.93, 7.22 [08:39:21] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.694 second response time [08:40:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.55, 7.94 [08:41:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.10, 6.31, 6.71 [08:41:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.85, 7.84, 7.98 [08:43:39] PROBLEM - ping6 on prometheus131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.04 ms [08:43:44] !log DELETED AND DROPPED https://phabricator.miraheze.org/P495 [08:43:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.28, 7.89, 7.97 [08:44:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:44:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.10, 8.02, 8.01 [08:45:12] PROBLEM - ping6 on mwtask141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.87 ms [08:45:41] PROBLEM - ping6 on prometheus131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.07 ms [08:45:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.00, 8.03, 7.56 [08:45:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.84, 7.36, 7.06 [08:45:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.67, 7.72, 7.89 [08:47:11] PROBLEM - ping6 on mwtask141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [08:47:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.71, 7.72, 7.52 [08:47:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.45, 7.45, 7.11 [08:47:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.37, 8.01, 7.98 [08:49:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.63, 7.49, 7.16 [08:53:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.55, 7.81, 7.98 [08:54:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.67, 7.97 [08:57:15] RECOVERY - db142 Disk Space on db142 is OK: DISK OK - free space: / 37486 MB (11% inode=97%); [08:58:04] PROBLEM - ping6 on ldap141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.09 ms [08:58:19] PROBLEM - ping6 on cloud14 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.76 ms [08:59:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.28, 6.32, 6.80 [09:00:07] PROBLEM - ping6 on ldap141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.11 ms [09:00:19] PROBLEM - ping6 on cloud14 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [09:05:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.74, 7.69, 7.69 [09:07:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.50, 7.18, 7.01 [09:08:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.07, 7.34 [09:09:25] PROBLEM - ping6 on swiftac111 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.04 ms [09:10:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 6.98, 7.27 [09:10:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.32, 7.33, 7.87 [09:11:24] PROBLEM - ping6 on swiftac111 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.91 ms [09:12:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.72, 7.50 [09:14:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.33, 7.38 [09:14:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 7.77, 7.93 [09:16:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.10, 7.47, 7.80 [09:17:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.34, 7.65, 7.95 [09:19:05] PROBLEM - mon181 Puppet on mon181 is UNKNOWN: NRPE: Unable to read output [09:20:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.44, 7.34 [09:22:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.94, 7.87, 7.82 [09:24:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.43, 7.40 [09:24:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.68, 7.58, 7.71 [09:27:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.80, 6.38, 6.70 [09:28:26] PROBLEM - ping6 on swiftproxy131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.70 ms [09:29:50] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.36, 6.53, 6.79 [09:30:27] PROBLEM - ping6 on swiftproxy131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.47 ms [09:30:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.59, 7.15, 7.15 [09:30:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [09:32:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 6.89, 7.06 [09:32:57] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [09:33:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.55, 6.72, 6.83 [09:33:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.78, 6.83, 6.78 [09:34:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.73, 7.17, 7.11 [09:35:50] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.15, 6.53, 6.75 [09:36:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.86, 7.74, 7.60 [09:38:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.34, 7.19 [09:38:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.51, 7.42, 7.31 [09:39:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.92, 6.70, 6.78 [09:39:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.67, 7.54, 7.60 [09:42:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 8.00, 7.43, 7.32 [09:43:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.19, 7.47, 7.05 [09:47:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:47:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.08, 7.58, 7.20 [09:48:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.60, 7.25, 6.97 [09:52:20] PROBLEM - cloud11 Puppet on cloud11 is UNKNOWN: NRPE: Unable to read output [09:54:42] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.43, 6.40, 6.79 [09:57:58] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [09:58:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.44, 7.41, 7.06 [09:59:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.13, 7.33, 7.33 [10:00:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.02, 6.95 [10:00:57] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [10:01:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.13, 7.34, 7.34 [10:04:14] PROBLEM - ping6 on swiftobject112 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.09 ms [10:04:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.42, 6.74 [10:04:32] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.43, 6.43, 6.74 [10:05:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.45, 7.24, 7.02 [10:06:16] PROBLEM - ping6 on swiftobject112 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.09 ms [10:07:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.46, 8.25, 7.73 [10:09:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.66, 6.97, 6.84 [10:09:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.74, 7.05, 7.00 [10:10:01] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.64, 7.98, 7.69 [10:12:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.82, 6.85, 6.77 [10:13:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.95, 7.46, 7.06 [10:13:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.35, 7.51, 7.18 [10:13:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.27, 8.34, 7.84 [10:14:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.47, 7.70, 7.10 [10:15:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.23, 6.92, 6.81 [10:17:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.21, 6.79, 6.78 [10:17:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.60, 7.90, 7.42 [10:18:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 7.76, 7.28 [10:19:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.05, 8.00, 7.57 [10:20:20] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [10:20:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.56, 8.41, 7.60 [10:21:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.78, 7.72, 7.14 [10:23:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.33, 7.24, 7.04 [10:24:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.65, 7.87, 7.87 [10:26:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 7.56, 7.51 [10:28:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 8.04, 7.70 [10:29:29] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:29:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.41, 7.79, 7.58 [10:30:25] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.95 ms [10:30:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.82, 7.66 [10:32:27] PROBLEM - ping6 on cloud11 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.92 ms [10:33:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.14, 8.06, 7.86 [10:34:44] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [10:35:02] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.41, 6.47, 6.76 [10:36:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 7.63, 7.54 [10:39:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.29, 7.33, 7.09 [10:41:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.72, 7.95, 7.97 [10:42:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.61, 7.56 [10:44:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.53, 8.39, 8.12 [10:45:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.50, 7.47, 7.89 [10:45:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.54, 7.64, 7.27 [10:45:54] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.82 ms [10:46:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.79, 7.62 [10:47:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.88, 7.35, 7.21 [10:47:56] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [10:47:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.31, 7.72, 7.90 [10:48:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 7.54, 7.55 [10:50:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.88, 7.67 [10:50:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.47, 7.53, 7.20 [10:52:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.80, 7.37, 7.51 [10:52:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.68, 7.08, 7.06 [10:53:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.42, 7.85, 7.89 [10:54:24] !log RENAMED orbitronwiki luckbychancewiki [10:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 7.93, 7.71 [10:55:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.77, 7.48, 7.75 [10:55:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.03, 7.63, 7.74 [10:57:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.53, 7.97, 7.91 [10:57:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.50, 7.42, 7.64 [10:58:07] PROBLEM - ping6 on mw142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.98 ms [10:58:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.55, 7.61 [10:59:02] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [10:59:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.81, 8.17, 7.91 [11:00:09] PROBLEM - ping6 on mw142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.71 ms [11:00:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.21, 8.52, 7.96 [11:00:36] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.50, 7.16, 7.04 [11:01:00] !log [salt-user@mwtask141] Renamed thatgaypigeonwiki to pigeonwiki using renamewiki.py [11:01:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:01:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.60, 8.05, 7.53 [11:02:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.14, 7.14, 7.05 [11:02:57] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [11:03:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.56, 7.99, 7.91 [11:04:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.80, 7.47, 7.70 [11:05:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.37, 8.67, 8.15 [11:06:18] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.29 ms [11:06:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.26, 8.00, 7.38 [11:06:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 13.94, 10.00, 8.60 [11:08:18] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.22 ms [11:11:19] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [11:12:19] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.20 ms [11:13:16] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.520 second response time [11:13:19] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.13, 9.00, 6.06 [11:14:19] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.25 ms [11:15:19] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.10, 6.96, 5.67 [11:16:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.73, 7.24, 6.35 [11:22:57] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.25, 6.79, 6.46 [11:32:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:32:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.89, 7.09, 6.65 [11:34:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.27, 7.59, 6.89 [11:36:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.13, 7.55, 6.97 [11:38:26] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [11:38:57] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.08, 6.27, 6.57 [11:39:40] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.12, 3.85, 2.19 [11:40:24] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.618 second response time [11:41:01] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.22, 9.99, 7.60 [11:41:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.96, 3.22, 2.18 [11:42:20] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.18 ms [11:43:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.73, 8.72, 7.43 [11:44:20] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.31 ms [11:49:05] PROBLEM - mon181 Puppet on mon181 is UNKNOWN: NRPE: Unable to read output [11:51:48] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.91 ms [11:53:50] PROBLEM - ping6 on cloud11 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.93 ms [11:54:02] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [11:55:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.99, 6.30, 7.74 [11:56:31] PROBLEM - ping6 on swiftobject111 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.08 ms [11:58:30] PROBLEM - ping6 on swiftobject111 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.49 ms [11:59:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.30, 6.37, 7.71 [12:00:06] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:00:57] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:02:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.53, 6.37, 7.95 [12:07:43] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:08:18] PROBLEM - ping6 on mem141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [12:09:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.01, 6.97, 7.92 [12:09:41] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.574 second response time [12:10:20] PROBLEM - ping6 on mem141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.17 ms [12:13:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.38, 7.30, 7.21 [12:17:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 7.18, 7.59 [12:17:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.11, 7.88, 7.52 [12:17:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.81, 6.36, 6.78 [12:19:04] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.57, 7.35, 7.61 [12:20:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.52, 7.42 [12:21:00] PROBLEM - ping6 on ldap141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.00 ms [12:21:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.47, 7.15, 7.02 [12:22:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.47, 7.40 [12:23:01] PROBLEM - ping6 on ldap141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.04 ms [12:23:59] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.82, 7.38, 7.89 [12:27:53] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.85, 6.49, 6.78 [12:29:39] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [12:31:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.07, 7.38, 7.45 [12:33:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.96, 7.28, 7.41 [12:33:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.14, 7.58, 7.74 [12:34:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.63, 7.86, 7.61 [12:36:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.01, 7.47, 7.49 [12:38:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 7.81, 7.61 [12:39:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.10, 7.50, 7.20 [12:40:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.47, 7.59, 7.54 [12:41:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.50, 7.21, 7.13 [12:43:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.71, 7.77, 7.34 [12:44:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 7.25, 7.07 [12:45:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.23, 7.73, 7.56 [12:45:33] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [12:45:50] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.55, 7.02, 6.84 [12:46:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.01, 7.46, 7.49 [12:47:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.20, 7.75, 7.50 [12:47:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.58, 7.29, 6.94 [12:48:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.90, 7.89, 7.63 [12:49:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.62, 7.40, 7.03 [12:51:09] PROBLEM - ping6 on mw134 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [12:52:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.70, 7.67, 7.59 [12:53:07] PROBLEM - ping6 on mw134 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.90 ms [12:54:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.86, 7.80, 7.62 [12:55:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.55, 7.53, 7.39 [12:56:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.86, 7.70, 7.60 [12:57:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.16, 7.37, 7.35 [12:58:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.01, 7.62, 7.57 [12:59:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.29, 7.84, 7.84 [12:59:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.83, 7.82, 7.98 [13:00:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.31, 7.52, 7.55 [13:01:07] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:01:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.19, 7.98, 7.57 [13:01:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.26, 7.20, 6.98 [13:01:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 8.17, 7.94 [13:01:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.26, 8.00, 8.03 [13:02:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.74, 8.07, 7.77 [13:03:03] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [13:03:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.32, 7.45, 7.44 [13:03:42] 👍 [13:03:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.00, 7.47, 7.72 [13:03:43] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.57, 6.94, 6.92 [13:04:36] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [13:04:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.96, 7.32, 7.54 [13:05:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.71, 7.78 [13:07:24] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.60, 8.00, 7.66 [13:08:23] PROBLEM - ping6 on mwtask141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.70 ms [13:08:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.15, 8.04, 7.75 [13:09:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.23, 7.48, 7.12 [13:10:26] PROBLEM - ping6 on mwtask141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.68 ms [13:10:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.93, 7.61, 7.63 [13:11:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.79, 7.99, 7.73 [13:11:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.24, 7.07, 7.02 [13:13:18] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.57, 8.13, 7.80 [13:15:03] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] [13:15:05] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 1s [13:15:06] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] [13:15:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:15:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:15:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.99, 7.73, 7.98 [13:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:16:07] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 63s [13:16:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.80, 7.06, 7.81 [13:19:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.33, 7.43, 7.66 [13:23:29] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] [13:23:30] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 1s [13:23:31] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] [13:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:29] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 60s [13:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:46] !log [paladox@mwtask141] starting deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [13:24:49] !log [paladox@mwtask141] finished deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 3s [13:24:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.30, 7.13, 7.39 [13:24:50] !log [paladox@mwtask141] starting deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [13:24:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:24:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.22, 7.91, 7.60 [13:24:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:25:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:26:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.94, 7.28 [13:27:16] PROBLEM - itadminwiki.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for itadminwiki.pl could not be found [13:27:51] !log [paladox@mwtask141] finished deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 185s [13:27:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 8.00, 7.77 [13:29:00] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.84 ms [13:29:10] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.74, 6.07, 6.77 [13:30:25] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:30:59] PROBLEM - ping6 on db131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.88 ms [13:31:03] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:31:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.72, 6.36, 6.69 [13:33:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.82, 6.53, 6.81 [13:33:12] PROBLEM - db112 Puppet on db112 is UNKNOWN: NRPE: Unable to read output [13:34:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.26, 7.87, 7.76 [13:35:09] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.59, 6.47, 6.75 [13:35:10] PROBLEM - ping6 on mwtask141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.37 ms [13:35:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.69, 7.73, 7.48 [13:36:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.98, 7.96, 7.81 [13:37:11] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 6 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/home/paladox],File[/home/reception],File[/home/universalomega],File[/home/void] [13:37:13] PROBLEM - ping6 on mwtask141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.78 ms [13:37:58] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:38:10] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [13:38:25] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:38:29] PROBLEM - bast121 Current Load on bast121 is CRITICAL: LOAD CRITICAL - total load average: 2.07, 1.21, 0.51 [13:38:44] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 1.313 second response time [13:38:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.77, 7.44 [13:38:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.63, 3.37, 1.97 [13:38:54] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:38:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.58, 8.05, 7.88 [13:38:56] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.461 second response time [13:39:07] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.84, 4.34, 2.82 [13:39:10] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:39:11] PROBLEM - mail121 Current Load on mail121 is WARNING: LOAD WARNING - total load average: 1.77, 1.17, 0.58 [13:39:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.63, 6.92, 6.84 [13:39:12] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [13:39:22] PROBLEM - ns2 Current Load on ns2 is CRITICAL: LOAD CRITICAL - total load average: 2.19, 1.25, 0.57 [13:39:39] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [13:39:39] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw132 mw141 mw142 mw134 mw143 [13:39:41] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:39:42] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [13:39:44] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.55, 1.65, 0.85 [13:39:52] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.520 second response time [13:40:01] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:08] PROBLEM - mem141 memcached on mem141 is CRITICAL: connect to address 2a10:6740::6:510 and port 11211: Connection refused [13:40:10] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 12 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 Nimbus],Exec[git_checkout_MediaWiki-REL1_41 StandardDialogs] [13:40:12] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.51, 11.32, 7.79 [13:40:15] PROBLEM - reports121 Current Load on reports121 is CRITICAL: CRITICAL - load average: 2.75, 1.37, 0.59 [13:40:25] PROBLEM - swiftobject112 Puppet on swiftobject112 is UNKNOWN: NRPE: Unable to read output [13:40:25] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 2.04, 1.32, 0.59 [13:40:30] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.69, 7.73, 7.13 [13:40:34] PROBLEM - ldap141 Current Load on ldap141 is CRITICAL: LOAD CRITICAL - total load average: 2.25, 1.45, 0.64 [13:40:43] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.743 second response time [13:40:56] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [13:41:09] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.32, 1.59, 0.80 [13:41:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.35, 7.52, 7.08 [13:41:15] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.33, 12.99, 8.00 [13:41:20] RECOVERY - ns2 Current Load on ns2 is OK: LOAD OK - total load average: 1.58, 1.43, 0.74 [13:41:28] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:41:37] PROBLEM - phab121 Puppet on phab121 is UNKNOWN: NRPE: Unable to read output [13:41:48] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 5.839 second response time [13:42:00] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:05] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:42:07] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.99, 10.86, 8.03 [13:42:07] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw134 [13:42:07] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw134 [13:42:08] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.158 second response time [13:42:26] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 1.97, 1.58, 0.78 [13:42:34] PROBLEM - ldap141 Current Load on ldap141 is WARNING: LOAD WARNING - total load average: 1.77, 1.60, 0.79 [13:42:35] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.494 second response time [13:42:39] PROBLEM - bast121 Puppet on bast121 is UNKNOWN: NRPE: Unable to read output [13:43:01] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.681 second response time [13:43:06] PROBLEM - swiftobject111 SSH on swiftobject111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:08] PROBLEM - swiftobject101 Puppet on swiftobject101 is UNKNOWN: NRPE: Unable to read output [13:43:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.27, 7.39, 7.09 [13:43:14] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.43, 11.29, 7.97 [13:43:24] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.328 second response time [13:43:29] PROBLEM - swiftobject111 PowerDNS Recursor on swiftobject111 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:43:31] PROBLEM - swiftobject113 Puppet on swiftobject113 is UNKNOWN: NRPE: Unable to read output [13:43:39] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [13:43:40] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.613 second response time [13:43:44] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.96, 1.61, 1.05 [13:43:47] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.544 second response time [13:44:04] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.930 second response time [13:44:06] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.051 second response time [13:44:07] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.01, 12.34, 8.89 [13:44:07] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 18 backends are healthy [13:44:08] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 18 backends are healthy [13:44:08] RECOVERY - mem141 memcached on mem141 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:510 port 11211 [13:44:10] RECOVERY - reports121 Current Load on reports121 is OK: OK - load average: 0.53, 1.27, 0.76 [13:44:13] PROBLEM - mail121 Puppet on mail121 is UNKNOWN: NRPE: Unable to read output [13:44:25] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.81, 1.25, 0.76 [13:44:26] RECOVERY - bast121 Current Load on bast121 is OK: LOAD OK - total load average: 0.55, 1.49, 0.95 [13:44:32] RECOVERY - ldap141 Current Load on ldap141 is OK: LOAD OK - total load average: 0.91, 1.43, 0.84 [13:44:39] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [13:44:44] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 28 minutes ago with 0 failures [13:44:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 7.40, 7.42 [13:44:59] PROBLEM - swiftobject121 Puppet on swiftobject121 is UNKNOWN: NRPE: Unable to read output [13:45:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.02, 6.67, 5.81 [13:45:07] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 1.70, 3.60, 3.10 [13:45:12] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 18 backends are healthy [13:45:14] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.87, 10.15, 7.96 [13:45:22] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [13:45:40] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 18 backends are healthy [13:46:59] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.45, 6.64, 5.91 [13:47:07] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.47, 2.85, 2.88 [13:47:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.96, 7.81, 7.29 [13:47:46] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [13:48:07] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.93, 10.04, 8.90 [13:48:09] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:49:10] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[puppet-agent] [13:49:13] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 17 minutes ago with 0 failures [13:50:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.26, 7.70, 7.52 [13:50:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.35, 7.34 [13:52:08] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [13:52:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.44, 7.91, 7.61 [13:52:42] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.23, 3.54, 3.46 [13:52:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.49, 7.40 [13:53:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.47, 7.98, 7.57 [13:53:11] PROBLEM - ping6 on bast121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.10 ms [13:53:16] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 503 Service Unavailable [13:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.85, 7.55 [13:54:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.27, 7.57, 7.96 [13:55:10] PROBLEM - ping6 on bast121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.05 ms [13:55:14] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.899 second response time [13:55:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0475a28c205b [13:55:36] [02miraheze/puppet] 07paladox 030475a28 - opensearch: support ruby3 [13:55:39] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [13:55:42] [02puppet] 07paladox opened pull request 03#3620: opensearch: support ruby3 - 13https://github.com/miraheze/puppet/pull/3620 [13:56:09] [02puppet] 07paladox closed pull request 03#3620: opensearch: support ruby3 - 13https://github.com/miraheze/puppet/pull/3620 [13:56:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/be07563cee3a...7140c173083a [13:56:12] [02miraheze/puppet] 07paladox 037140c17 - opensearch: support ruby3 (#3620) [13:56:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [13:56:15] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [13:56:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 13.08, 9.02, 8.40 [13:57:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.05, 8.10, 7.66 [13:58:39] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.25, 3.14, 3.38 [13:58:55] RECOVERY - swiftobject111 PowerDNS Recursor on swiftobject111 is OK: DNS OK: 4.861 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [13:59:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.05, 7.70, 7.57 [13:59:37] RECOVERY - swiftobject111 SSH on swiftobject111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [13:59:39] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [14:00:34] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 47 minutes ago with 0 failures [14:01:09] PROBLEM - mail121 Current Load on mail121 is WARNING: LOAD WARNING - total load average: 1.23, 1.83, 1.72 [14:02:34] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.08 ms [14:03:06] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [14:03:06] PROBLEM - ping6 on db131 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:03:06] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:03:12] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [14:03:15] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.24, 1.25, 1.52 [14:03:28] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 42 minutes ago with 0 failures [14:03:42] PROBLEM - swiftobject111 Puppet on swiftobject111 is UNKNOWN: NRPE: Unable to read output [14:04:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 4.86, 7.32, 7.80 [14:04:36] PROBLEM - ping6 on ns2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [14:04:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.93, 8.00 [14:04:51] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.549 second response time [14:04:55] PROBLEM - ping6 on db131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.97 ms [14:04:55] PROBLEM - ping6 on mw141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.48 ms [14:09:38] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [14:10:24] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:10:38] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:11:15] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [14:12:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.87, 7.30, 7.42 [14:12:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.76, 9.33, 8.33 [14:13:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.78, 7.89, 7.41 [14:13:30] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [14:14:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.73, 7.01, 7.29 [14:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.07, 7.68, 7.89 [14:19:49] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:21:49] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.606 second response time [14:22:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.59, 7.24, 7.95 [14:23:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.31, 7.52, 7.54 [14:24:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.70, 8.14, 7.95 [14:26:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.36, 7.46, 7.85 [14:27:23] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:28:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.95, 7.32, 7.68 [14:29:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/437d9dbbdd34 [14:29:30] [02miraheze/puppet] 07paladox 03437d9db - Set openserarch Hiera values in os131/os141 [14:29:31] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [14:29:33] [02puppet] 07paladox opened pull request 03#3621: Set openserarch Hiera values in os131/os141 - 13https://github.com/miraheze/puppet/pull/3621 [14:29:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/437d9dbbdd34...674e380c8522 [14:30:00] [02miraheze/puppet] 07paladox 03674e380 - Update os141.yaml [14:30:01] [02puppet] 07paladox synchronize pull request 03#3621: Set openserarch Hiera values in os131/os141 - 13https://github.com/miraheze/puppet/pull/3621 [14:30:04] [02puppet] 07paladox closed pull request 03#3621: Set openserarch Hiera values in os131/os141 - 13https://github.com/miraheze/puppet/pull/3621 [14:30:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [14:30:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/7140c173083a...dfa5e41a417a [14:30:12] [02miraheze/puppet] 07paladox 03dfa5e41 - Set openserarch Hiera values in os131/os141 (#3621) [14:30:13] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [14:30:40] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.99, 9.69, 7.98 [14:30:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.16, 7.93, 7.99 [14:31:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/dfa5e41a417a...a8ff91e2091b [14:31:42] [02miraheze/puppet] 07paladox 03a8ff91e - Revert "Set openserarch Hiera values in os131/os141 (#3621)" [14:32:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.25, 9.35, 8.08 [14:32:41] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.41, 10.52, 9.15 [14:33:35] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 209.32 ms [14:33:41] PROBLEM - ping6 on swiftobject113 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.76 ms [14:34:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a8ff91e2091b...f46ec4ff393f [14:34:21] [02miraheze/puppet] 07paladox 03f46ec4f - Temporarily hack opensearch_rest [14:34:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.28, 7.45, 7.54 [14:34:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.48, 8.11, 8.04 [14:35:09] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5035399795 secs [14:35:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.62, 7.54, 7.35 [14:35:23] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.77, 9.99, 8.78 [14:35:35] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.90 ms [14:35:40] PROBLEM - ping6 on swiftobject113 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.06 ms [14:36:11] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.09, 7.34, 7.07 [14:36:31] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.00, 10.00, 9.31 [14:36:32] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.01, 10.67, 8.90 [14:36:36] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:36:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 7.25, 7.47 [14:37:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.92, 7.37, 7.32 [14:37:22] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.64, 9.28, 8.67 [14:38:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.21, 6.84, 6.91 [14:38:30] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.45, 9.82, 8.78 [14:38:43] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:39:08] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 208.33 ms [14:40:09] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.58, 6.38, 6.73 [14:40:30] PROBLEM - ping6 on mail121 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 211.35 ms [14:40:33] PROBLEM - ping6 on db112 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 215.08 ms [14:41:11] PROBLEM - ping6 on cloud11 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.92 ms [14:42:33] PROBLEM - ping6 on db112 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [14:42:34] PROBLEM - ping6 on mail121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.00 ms [14:42:41] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [14:42:47] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 211.18 ms [14:42:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.58, 7.49 [14:42:50] PROBLEM - ping6 on mw143 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 219.62 ms [14:42:51] PROBLEM - ping6 on cp25 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 217.41 ms [14:42:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.95, 7.50, 7.86 [14:44:46] PROBLEM - ping6 on mem131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.99 ms [14:44:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.61, 7.51 [14:44:50] PROBLEM - ping6 on cp25 is WARNING: PING WARNING - Packet loss = 0%, RTA = 145.21 ms [14:44:51] PROBLEM - ping6 on mw143 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.88 ms [14:45:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.60, 7.24, 7.19 [14:46:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 8.85, 8.02 [14:46:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.81, 8.26, 8.03 [14:48:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.48, 7.65, 7.15 [14:49:10] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.39, 7.96, 7.54 [14:50:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.76, 7.60, 7.19 [14:51:22] PROBLEM - ping6 on mwtask141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.05 ms [14:52:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 7.73, 7.86 [14:52:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.42, 7.80, 7.97 [14:53:01] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [14:53:25] PROBLEM - ping6 on mwtask141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.70 ms [14:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 8.05, 7.97 [14:54:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.40, 8.27, 8.11 [14:55:00] PROBLEM - ping6 on reports121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.26 ms [14:55:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f46ec4ff393f...095b6f8b3b77 [14:55:49] [02miraheze/puppet] 07paladox 03095b6f8 - cpufrequtils: replace deprecated ensure_packages [14:56:59] PROBLEM - ping6 on reports121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.22 ms [14:57:03] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [14:58:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.66, 7.77, 7.31 [15:00:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.17, 7.12, 7.13 [15:00:48] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.63, 2.98, 2.55 [15:01:24] PROBLEM - ping6 on os131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.99 ms [15:02:48] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.73, 2.81, 2.53 [15:02:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.73, 7.54, 7.92 [15:03:25] PROBLEM - ping6 on os131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.20 ms [15:04:01] PROBLEM - ping6 on cloud14 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.84 ms [15:06:01] PROBLEM - ping6 on cloud14 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [15:06:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.52, 7.50, 7.93 [15:06:57] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.10, 7.35, 7.99 [15:09:10] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.50, 6.13, 6.68 [15:10:01] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.70, 6.34, 6.78 [15:10:41] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:12:02] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:13:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.27, 6.99, 6.88 [15:13:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.73, 6.86, 6.91 [15:16:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.06, 7.49, 7.67 [15:17:00] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:17:07] PROBLEM - ping6 on mw133 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 203.66 ms [15:17:10] PROBLEM - ping6 on bast121 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 214.76 ms [15:17:10] PROBLEM - ping6 on cloud12 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 216.22 ms [15:17:11] PROBLEM - ping6 on swiftobject122 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 212.14 ms [15:18:51] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.78, 7.13, 7.51 [15:19:07] PROBLEM - ping6 on mw133 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [15:19:10] PROBLEM - ping6 on bast121 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.01 ms [15:19:10] PROBLEM - ping6 on cloud12 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.99 ms [15:19:11] PROBLEM - ping6 on swiftobject122 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.73 ms [15:20:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.94, 7.60, 7.16 [15:22:00] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.38, 7.53, 7.19 [15:23:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.48, 7.47, 7.44 [15:27:59] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.23, 6.12, 6.66 [15:28:50] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.38, 6.76 [15:29:36] PROBLEM - ping6 on db142 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.75 ms [15:30:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.13, 7.52, 7.42 [15:31:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:31:35] PROBLEM - ping6 on db142 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [15:32:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.06, 7.29, 7.35 [15:35:10] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.93, 6.29, 6.71 [15:36:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.83, 8.15, 7.68 [15:36:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.12, 6.33, 6.79 [15:38:42] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.36 ms [15:39:49] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.41, 9.76, 8.76 [15:39:51] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:39:56] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.conworlds.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:40:02] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:40:41] PROBLEM - ping6 on mw141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.27 ms [15:41:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 6.92, 6.71 [15:41:49] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.09, 9.83, 8.86 [15:42:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.76, 7.83, 7.76 [15:42:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 6.77, 6.82 [15:43:49] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.40, 9.57, 8.91 [15:46:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.23, 7.86, 7.19 [15:48:42] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.89 ms [15:49:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 7.86, 7.18 [15:49:34] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [15:50:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.33, 7.91, 7.38 [15:50:41] PROBLEM - ping6 on mw141 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [15:50:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.68, 7.64, 7.69 [15:50:55] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.59, 7.88, 7.06 [15:50:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.22, 7.81, 7.28 [15:51:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 7.47, 7.12 [15:52:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.95, 7.75, 7.73 [15:52:55] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.75, 7.30, 6.94 [15:54:54] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.11, 6.69, 6.75 [15:56:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.74, 7.84, 7.45 [15:56:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.31, 7.74, 7.47 [15:57:30] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.91 ms [15:57:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.92, 7.22, 7.28 [15:58:09] PROBLEM - ping6 on os131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [15:59:33] PROBLEM - ping6 on graylog131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.78 ms [15:59:49] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.61, 10.23, 9.48 [16:00:11] PROBLEM - ping6 on os131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.98 ms [16:01:49] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.71, 11.11, 9.89 [16:02:42] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.15, 10.39, 8.97 [16:02:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.50, 6.76 [16:03:04] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [16:03:38] PROBLEM - ping6 on graylog131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.85 ms [16:03:49] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.50, 6.24, 6.76 [16:03:49] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.57, 11.01, 9.99 [16:03:59] PROBLEM - ping6 on cloud14 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.83 ms [16:04:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.28, 10.91, 9.32 [16:04:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.07, 8.28, 7.86 [16:05:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:05:40] PROBLEM - ping6 on graylog131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.88 ms [16:05:59] PROBLEM - ping6 on cloud14 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.72 ms [16:06:40] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.92, 10.63, 9.44 [16:06:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.79, 6.82 [16:07:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.22, 6.79, 6.86 [16:08:40] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.00, 10.11, 9.40 [16:08:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.43, 6.72, 6.79 [16:08:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.09, 7.22, 7.26 [16:09:46] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.36, 7.05, 6.93 [16:11:18] RECOVERY - db181 MariaDB on db181 is OK: Uptime: 85139 Threads: 1 Questions: 125 Slow queries: 0 Opens: 32 Open tables: 26 Queries per second avg: 0.001 [16:11:42] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:11:44] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.81, 7.31, 7.04 [16:11:58] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.49, 10.68, 9.40 [16:12:00] PROBLEM - ping6 on cloud14 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.72 ms [16:12:06] PROBLEM - ping6 on test131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.02 ms [16:12:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.87, 7.69, 7.88 [16:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 7.73, 7.20 [16:12:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.99, 7.47, 7.37 [16:13:41] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.03, 7.77, 7.23 [16:13:42] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [16:13:49] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.03, 11.27, 10.54 [16:14:00] PROBLEM - ping6 on cloud14 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.87 ms [16:14:05] PROBLEM - ping6 on test131 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.29 ms [16:14:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.59, 11.19, 10.08 [16:14:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 7.02, 6.99 [16:15:49] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.58, 11.65, 10.78 [16:15:58] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.11, 10.16, 9.49 [16:16:07] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=sketchianmultiversewiki --no-updates --username-prefix=wikia:sketchian-multiverse /home/agent/ImportDump/sketchianmultiversewiki-20240110021758.xml (START) [16:16:08] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=sketchianmultiversewiki --no-updates --username-prefix=wikia:sketchian-multiverse /home/agent/ImportDump/sketchianmultiversewiki-20240110021758.xml (END - exit=256) [16:16:09] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=sketchianmultiversewiki (START) [16:16:10] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=sketchianmultiversewiki (END - exit=0) [16:16:11] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=sketchianmultiversewiki --active --update (END - exit=0) [16:16:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:24] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/agent/ImportDump/warriorclancatswiki-20240110151420.xml (START) [16:16:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:25] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/agent/ImportDump/warriorclancatswiki-20240110151420.xml (END - exit=256) [16:16:26] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (START) [16:16:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.18, 7.43, 7.69 [16:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 7.39, 7.11 [16:16:49] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/agent/ImportDump/warriorclancatswiki-20240110151420.xml (START) [16:16:50] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/agent/ImportDump/warriorclancatswiki-20240110151420.xml (END - exit=256) [16:16:52] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (START) [16:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:59] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (END - exit=2) [16:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:12] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=warriorclancatswiki --active --update (END - exit=0) [16:17:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:34] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:17:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.69, 7.96, 7.48 [16:17:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:17:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b0f5780b60c7 [16:18:22] [02miraheze/puppet] 07paladox 03b0f5780 - monitoring: fix ping6 [16:18:24] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [16:18:27] [02puppet] 07paladox opened pull request 03#3622: monitoring: fix ping6 - 13https://github.com/miraheze/puppet/pull/3622 [16:18:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.68, 7.19, 7.57 [16:19:10] [02puppet] 07paladox closed pull request 03#3622: monitoring: fix ping6 - 13https://github.com/miraheze/puppet/pull/3622 [16:19:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/095b6f8b3b77...6806aa6cfabc [16:19:14] [02miraheze/puppet] 07paladox 036806aa6 - monitoring: fix ping6 (#3622) [16:19:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [16:19:18] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [16:19:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.52, 7.30, 6.84 [16:20:14] RECOVERY - ping6 on mw132 is OK: PING OK - Packet loss = 0%, RTA = 128.94 ms [16:20:20] RECOVERY - ping6 on mw141 is OK: PING OK - Packet loss = 0%, RTA = 128.93 ms [16:20:23] RECOVERY - ping6 on prometheus131 is OK: PING OK - Packet loss = 0%, RTA = 129.01 ms [16:20:24] RECOVERY - ping6 on cp25 is OK: PING OK - Packet loss = 0%, RTA = 145.59 ms [16:20:26] RECOVERY - ping6 on matomo121 is OK: PING OK - Packet loss = 0%, RTA = 128.94 ms [16:20:28] RECOVERY - ping6 on swiftobject121 is OK: PING OK - Packet loss = 0%, RTA = 129.54 ms [16:20:30] RECOVERY - ping6 on os131 is OK: PING OK - Packet loss = 0%, RTA = 128.96 ms [16:20:30] RECOVERY - ping6 on bast121 is OK: PING OK - Packet loss = 0%, RTA = 129.12 ms [16:20:30] RECOVERY - ping6 on db112 is OK: PING OK - Packet loss = 0%, RTA = 129.13 ms [16:20:33] RECOVERY - ping6 on mw133 is OK: PING OK - Packet loss = 0%, RTA = 128.72 ms [16:20:34] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 128.99 ms [16:20:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.96, 8.24, 7.68 [16:20:36] RECOVERY - ping6 on swiftproxy111 is OK: PING OK - Packet loss = 0%, RTA = 128.78 ms [16:20:39] RECOVERY - ping6 on db142 is OK: PING OK - Packet loss = 0%, RTA = 128.82 ms [16:20:41] RECOVERY - ping6 on db131 is OK: PING OK - Packet loss = 0%, RTA = 129.10 ms [16:20:42] RECOVERY - ping6 on swiftobject122 is OK: PING OK - Packet loss = 0%, RTA = 128.74 ms [16:20:45] RECOVERY - ping6 on mw131 is OK: PING OK - Packet loss = 0%, RTA = 128.82 ms [16:20:48] RECOVERY - ping6 on mwtask141 is OK: PING OK - Packet loss = 0%, RTA = 128.70 ms [16:20:49] RECOVERY - ping6 on bast141 is OK: PING OK - Packet loss = 0%, RTA = 129.22 ms [16:20:49] RECOVERY - ping6 on swiftproxy131 is OK: PING OK - Packet loss = 0%, RTA = 128.72 ms [16:20:51] RECOVERY - ping6 on mw143 is OK: PING OK - Packet loss = 0%, RTA = 128.90 ms [16:20:51] RECOVERY - ping6 on swiftac111 is OK: PING OK - Packet loss = 0%, RTA = 129.04 ms [16:20:52] RECOVERY - ping6 on reports121 is OK: PING OK - Packet loss = 0%, RTA = 129.20 ms [16:20:53] RECOVERY - ping6 on mem131 is OK: PING OK - Packet loss = 0%, RTA = 129.10 ms [16:20:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.74, 7.82, 7.75 [16:20:55] RECOVERY - ping6 on ldap141 is OK: PING OK - Packet loss = 0%, RTA = 129.05 ms [16:20:56] RECOVERY - ping6 on mail121 is OK: PING OK - Packet loss = 0%, RTA = 129.08 ms [16:21:01] RECOVERY - ping6 on swiftobject113 is OK: PING OK - Packet loss = 0%, RTA = 129.14 ms [16:21:09] RECOVERY - ping6 on test131 is OK: PING OK - Packet loss = 0%, RTA = 129.53 ms [16:21:13] RECOVERY - ping6 on cp24 is OK: PING OK - Packet loss = 0%, RTA = 144.69 ms [16:21:16] RECOVERY - ping6 on phab121 is OK: PING OK - Packet loss = 0%, RTA = 129.08 ms [16:21:17] RECOVERY - ping6 on mw134 is OK: PING OK - Packet loss = 0%, RTA = 129.26 ms [16:21:17] RECOVERY - ping6 on mem141 is OK: PING OK - Packet loss = 0%, RTA = 129.00 ms [16:21:31] RECOVERY - ping6 on os141 is OK: PING OK - Packet loss = 0%, RTA = 128.85 ms [16:21:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.39, 7.65, 7.40 [16:21:34] RECOVERY - ping6 on db121 is OK: PING OK - Packet loss = 0%, RTA = 129.61 ms [16:21:44] RECOVERY - ping6 on swiftobject112 is OK: PING OK - Packet loss = 0%, RTA = 129.03 ms [16:21:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.45, 7.46, 6.97 [16:21:45] RECOVERY - ping6 on swiftobject111 is OK: PING OK - Packet loss = 0%, RTA = 129.10 ms [16:21:45] RECOVERY - ping6 on jobchron121 is OK: PING OK - Packet loss = 0%, RTA = 128.97 ms [16:21:57] RECOVERY - ping6 on graylog131 is OK: PING OK - Packet loss = 0%, RTA = 129.71 ms [16:21:59] RECOVERY - ping6 on mw142 is OK: PING OK - Packet loss = 0%, RTA = 128.67 ms [16:22:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.38, 7.92, 7.63 [16:22:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6806aa6cfabc...9a43fa2eeb99 [16:22:41] [02miraheze/puppet] 07paladox 039a43fa2 - monitoring: Apply ping6 changes to cloud* [16:22:57] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (END - exit=0) [16:22:57] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.21, 10.94, 9.98 [16:22:58] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=warriorclancatswiki --active --update (END - exit=0) [16:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:29] RECOVERY - ping6 on cloud14 is OK: PING OK - Packet loss = 0%, RTA = 128.74 ms [16:23:45] RECOVERY - ping6 on cloud12 is OK: PING OK - Packet loss = 0%, RTA = 129.15 ms [16:23:48] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 129.11 ms [16:23:55] PROBLEM - ping6 on cloud10 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.84 ms [16:23:57] RECOVERY - ping6 on cloud13 is OK: PING OK - Packet loss = 0%, RTA = 128.68 ms [16:24:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9a43fa2eeb99...6a2dac451b76 [16:24:27] [02miraheze/puppet] 07paladox 036a2dac4 - Fix [16:24:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.15, 7.98, 7.87 [16:25:19] PROBLEM - ping6 on cloud10 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.74 ms [16:25:22] RECOVERY - ping6 on db101 is OK: PING OK - Packet loss = 0%, RTA = 128.96 ms [16:25:32] RECOVERY - ping6 on swiftobject101 is OK: PING OK - Packet loss = 0%, RTA = 128.90 ms [16:26:11] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.66, 8.99, 10.00 [16:26:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.90, 7.57, 7.46 [16:26:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.51, 7.76, 7.59 [16:26:45] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.39, 9.73, 10.12 [16:29:24] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.00, 6.47, 6.74 [16:30:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.07, 7.80, 7.58 [16:30:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.24, 8.29, 7.99 [16:31:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.95, 7.85 [16:32:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.27, 7.69, 7.58 [16:32:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.17, 7.52, 7.75 [16:32:49] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [16:33:14] [02mw-config] 07redbluegreenhat opened pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432 [16:33:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.88, 6.75, 6.79 [16:33:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.67, 8.15, 7.96 [16:34:15] miraheze/mw-config - redbluegreenhat the build has errored. [16:34:29] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.42, 9.85, 10.05 [16:35:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.86, 7.89 [16:36:23] [02mw-config] 07redbluegreenhat synchronize pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432 [16:37:23] miraheze/mw-config - redbluegreenhat the build has errored. [16:37:24] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.16, 6.55, 6.71 [16:38:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.59, 7.58, 7.92 [16:38:30] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.73, 7.75, 7.69 [16:38:52] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php --wiki=gengbaikewiki /srv/mediawiki/1.40/extensions/GrowthExperiments/sql/mysql/growthexperiments_mentor_mentee.sql (END - exit=256) [16:38:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:40:41] [02mw-config] 07redbluegreenhat synchronize pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432 [16:41:46] miraheze/mw-config - redbluegreenhat the build passed. [16:41:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.34, 7.74, 7.74 [16:41:50] finally [16:42:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.39, 7.88, 7.95 [16:42:11] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.68, 7.47, 7.32 [16:43:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.14, 7.75, 7.76 [16:44:05] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.75, 9.75, 9.44 [16:44:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.66, 7.20, 6.95 [16:45:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.68, 8.13, 7.90 [16:46:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 7.67, 7.89 [16:46:05] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 10.04, 9.87, 9.53 [16:47:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.36, 7.63 [16:48:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.46, 7.61, 7.47 [16:49:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.54, 7.66 [16:50:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.90, 8.31, 7.38 [16:51:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.86, 7.80 [16:51:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.48, 7.40, 7.64 [16:52:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.43, 7.41, 7.86 [16:52:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.56, 7.92, 7.35 [16:53:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.60, 7.15, 7.52 [16:55:05] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.41, 10.14, 9.64 [16:56:38] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/cleanupEmptyCategories.php --wiki=trollpastawiki (END - exit=0) [16:56:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:57] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/refreshLinks.php --wiki=trollpastawiki (START) [16:57:05] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 10.20, 10.11, 9.69 [16:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:05] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.29, 5.81, 6.70 [17:00:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.03, 7.04, 7.42 [17:00:29] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [17:02:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.99, 7.21, 7.45 [17:03:24] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [17:06:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 11.36, 8.57, 7.90 [17:06:08] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 2.56, 3.49, 3.09 [17:08:04] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.94, 2.94, 2.93 [17:14:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.71, 7.35, 7.71 [17:14:25] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.49, 6.64, 6.80 [17:15:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.31, 6.64, 6.91 [17:17:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.68, 6.52, 6.83 [17:19:24] Orange_Star: I would have the if statement on the outside of the hook [17:19:35] So it's only registered if the variable is true [17:19:40] Not only runs if true [17:19:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.11, 6.30, 6.71 [17:20:15] But yes that's what I was thinking of [17:21:23] [02mw-config] 07OAuthority commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885280594 [17:22:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.70, 7.08, 6.94 [17:23:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.29, 7.05 [17:23:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.35, 6.83, 6.81 [17:25:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 7.35, 7.10 [17:25:53] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.60, 10.95, 9.63 [17:26:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.17, 7.30, 7.33 [17:26:25] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.44, 6.40, 6.71 [17:26:55] [02mw-config] 07Universal-Omega commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885291189 [17:27:52] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.16, 10.60, 9.66 [17:28:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.65, 7.31, 7.32 [17:31:24] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [17:31:50] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.38, 10.06, 9.74 [17:36:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.16, 7.87, 7.48 [17:36:14] [02miraheze/mediawiki-repos] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/2da6c29453e0...aa2c93e44c75 [17:36:16] [02miraheze/mediawiki-repos] 07OAuthority 03aa2c93e - Add Screenplay (T10902) and SaneCase (T11276) [17:37:25] [02miraheze/mediawiki-repos] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/aa2c93e44c75...85f78e7ba1d0 [17:37:28] [02miraheze/mediawiki-repos] 07OAuthority 0385f78e7 - fix SaneCase [17:38:45] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/85f78e7ba1d0...749a3ef7bd5b [17:38:46] [02miraheze/mediawiki-repos] 07Universal-Omega 03749a3ef - Add commit to README [17:39:16] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/749a3ef7bd5b...5565eeddc7ba [17:39:19] [02miraheze/mediawiki-repos] 07Universal-Omega 035565eed - -newline [17:39:54] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/5565eeddc7ba...2155a3803a25 [17:39:57] [02miraheze/mediawiki-repos] 07Universal-Omega 032155a38 - Fix _branch_ display [17:40:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.67, 7.60, 7.48 [17:41:50] !log [agent@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/populateGroupPermissionsWithDefaults.php --wiki=mandaepediawiki (END - exit=0) [17:41:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.12, 7.06, 7.02 [17:43:31] [02MirahezeMagic] 07Universal-Omega closed pull request 03#450: Bump Python CI - 13https://github.com/miraheze/MirahezeMagic/pull/450 [17:43:34] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/05df3cb9d291...6ade774e0603 [17:43:35] [02miraheze/MirahezeMagic] 07Universal-Omega 036ade774 - Bump Python CI (#450) [17:43:36] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03python [17:43:39] [02MirahezeMagic] 07Universal-Omega deleted branch 03python - 13https://github.com/miraheze/MirahezeMagic [17:43:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.90, 6.90, 6.96 [17:45:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.51, 6.16, 6.70 [17:47:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4ff63e19ca12...42dfc6c55acc [17:47:26] [02miraheze/mw-config] 07Universal-Omega 0342dfc6c - CI: remove PHP 7.4 [17:48:42] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/5b7ec1620afc...b484e0a8d238 [17:48:45] [02miraheze/phabricator-extensions] 07Universal-Omega 03b484e0a - CI: use PHP 8.2 [17:48:54] [02phabricator-extensions] 07Universal-Omega closed pull request 03#40: Bump styfle/cancel-workflow-action from 0.11.0 to 0.12.0 - 13https://github.com/miraheze/phabricator-extensions/pull/40 [17:48:55] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/b484e0a8d238...07ee6a46bc59 [17:48:56] [02miraheze/phabricator-extensions] 07dependabot[bot] 0307ee6a4 - Bump styfle/cancel-workflow-action from 0.11.0 to 0.12.0 (#40) [17:48:57] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.0 [17:49:00] [02phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.0 - 13https://github.com/miraheze/phabricator-extensions [17:49:01] [02phabricator-extensions] 07dependabot[bot] edited pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38 [17:49:04] [02phabricator-extensions] 07dependabot[bot] edited pull request 03#40: Bump styfle/cancel-workflow-action from 0.11.0 to 0.12.0 - 13https://github.com/miraheze/phabricator-extensions/pull/40 [17:49:12] [02phabricator-extensions] 07Universal-Omega closed pull request 03#39: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/miraheze/phabricator-extensions/pull/39 [17:49:15] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/07ee6a46bc59...d5f1e7f26e26 [17:49:17] [02miraheze/phabricator-extensions] 07dependabot[bot] 03d5f1e7f - Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 (#39) [17:49:19] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 [17:49:21] [02phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 - 13https://github.com/miraheze/phabricator-extensions [17:49:25] miraheze/phabricator-extensions - Universal-Omega the build passed. [17:49:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.84, 7.04, 6.93 [17:49:37] [02phabricator-extensions] 07Universal-Omega commented on pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38#issuecomment-1885337844 [17:49:40] miraheze/phabricator-extensions - Universal-Omega the build passed. [17:49:41] [02phabricator-extensions] 07dependabot[bot] edited pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38 [17:49:55] [02phabricator-extensions] 07dependabot[bot] edited pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38 [17:49:56] miraheze/phabricator-extensions - Universal-Omega the build passed. [17:49:57] [02miraheze/phabricator-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/checkout-4 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/95f41ba866cc...cf0cb8809d95 [17:49:59] [02miraheze/phabricator-extensions] 07dependabot[bot] 03cf0cb88 - Bump actions/checkout from 3 to 4 [17:50:00] [02phabricator-extensions] 07dependabot[bot] synchronize pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38 [17:50:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.74, 7.43, 6.99 [17:50:47] miraheze/phabricator-extensions - dependabot[bot] the build passed. [17:52:02] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [17:52:24] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:52:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 7.14, 6.95 [17:53:13] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03checkout [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/2784bd349617 [17:53:16] [02miraheze/phabricator-extensions] 07Universal-Omega 032784bd3 - Bump actions/checkout from 3 to 4 [17:53:19] [02phabricator-extensions] 07Universal-Omega created branch 03checkout - 13https://github.com/miraheze/phabricator-extensions [17:53:21] [02phabricator-extensions] 07Universal-Omega opened pull request 03#41: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/41 [17:53:33] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03checkout [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/2784bd349617...fde4013f7470 [17:53:35] [02miraheze/phabricator-extensions] 07github-actions 03fde4013 - CI: lint code to MediaWiki standards [17:53:37] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#41: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/41 [17:53:49] [02mw-config] 07redbluegreenhat commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885343895 [17:54:01] miraheze/phabricator-extensions - Universal-Omega the build passed. [17:54:38] [02phabricator-extensions] 07Universal-Omega closed pull request 03#41: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/41 [17:54:41] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03checkout [17:54:42] [02phabricator-extensions] 07Universal-Omega deleted branch 03checkout - 13https://github.com/miraheze/phabricator-extensions [17:55:00] [02phabricator-extensions] 07Universal-Omega closed pull request 03#38: Bump actions/checkout from 3 to 4 - 13https://github.com/miraheze/phabricator-extensions/pull/38 [17:55:01] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/d5f1e7f26e26...559085191e39 [17:55:03] [02miraheze/phabricator-extensions] 07dependabot[bot] 035590851 - Bump actions/checkout from 3 to 4 (#38) [17:55:06] [02miraheze/phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-4 [17:55:07] [02phabricator-extensions] 07Universal-Omega deleted branch 03dependabot/github_actions/actions/checkout-4 - 13https://github.com/miraheze/phabricator-extensions [17:55:42] miraheze/phabricator-extensions - Universal-Omega the build passed. [17:56:23] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.16, 6.63, 6.77 [17:56:46] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [17:56:47] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [17:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:06] !log [@test131] starting deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] [17:57:07] !log [@test131] finished deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] - SUCCESS in 0s [17:57:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:34] RhinosF1: I see you were pinging me, didn't notice sorry [17:58:05] I recently changed IRC clients to catgirl, something funky is going on with mentions and messages mentioning me aren't properly shown [17:59:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.42, 7.76, 7.23 [18:00:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.24, 7.15, 7.11 [18:01:19] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/4e6131877449 [18:01:20] [02miraheze/phabricator-extensions] 07Universal-Omega 034e61318 - Cleanup [18:01:22] [02phabricator-extensions] 07Universal-Omega created branch 03cleanup - 13https://github.com/miraheze/phabricator-extensions [18:01:24] [02phabricator-extensions] 07Universal-Omega opened pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:01:40] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/4e6131877449...6602f84c598f [18:01:43] [02miraheze/phabricator-extensions] 07github-actions 036602f84 - CI: lint code to MediaWiki standards [18:01:44] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:01:56] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/6602f84c598f...8fe77f145640 [18:01:57] [02miraheze/phabricator-extensions] 07Universal-Omega 038fe77f1 - Remove ignore [18:02:00] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:02:01] !log [@mwtask181] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [18:02:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:10] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:02:29] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [18:02:44] miraheze/phabricator-extensions - Universal-Omega the build has errored. [18:02:56] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.11, 10.78, 8.79 [18:03:07] !log [@mwtask181] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 66s [18:03:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:23] !log [@mwtask181] starting deploy of {'folders': '1.40/extensions/MirahezeMagic', 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [18:03:25] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.12, 10.49, 8.41 [18:03:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:36] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.01, 11.77, 10.13 [18:03:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.49, 7.65, 7.35 [18:04:02] !log [@mwtask181] finished deploy of {'folders': '1.40/extensions/MirahezeMagic', 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 39s [18:04:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:30] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+1/-1/±0] 13https://github.com/miraheze/phabricator-extensions/compare/8fe77f145640...682597d1fe15 [18:04:32] [02miraheze/phabricator-extensions] 07Universal-Omega 03682597d - Rename [18:04:34] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:04:56] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.95, 10.16, 8.78 [18:05:19] miraheze/phabricator-extensions - Universal-Omega the build has errored. [18:05:25] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.49, 10.01, 8.49 [18:06:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.53, 7.83, 7.47 [18:07:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.50, 7.14 [18:08:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.60, 8.69, 7.82 [18:08:14] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/682597d1fe15...a8e865ecb571 [18:08:16] [02miraheze/phabricator-extensions] 07Universal-Omega 03a8e865e - - [18:08:19] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:08:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.99, 6.70, 6.56 [18:08:32] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03cleanup [+0/-0/±2] 13https://github.com/miraheze/phabricator-extensions/compare/a8e865ecb571...bb3608c947de [18:08:33] [02miraheze/phabricator-extensions] 07github-actions 03bb3608c - CI: lint code to MediaWiki standards [18:08:34] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:09:03] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:09:14] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [18:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:34] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.55, 11.24, 10.45 [18:09:34] !log [@mwtask141] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 19s [18:09:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:46] !log [@mwtask141] starting deploy of {'folders': '1.40/extensions/MirahezeMagic', 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [18:09:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.02, 7.42, 7.30 [18:10:06] !log [@mwtask141] finished deploy of {'folders': '1.40/extensions/MirahezeMagic', 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 20s [18:10:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.04, 7.98, 7.73 [18:12:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.00, 6.37, 6.51 [18:13:32] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 10.00, 9.77, 9.98 [18:15:00] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/bb3608c947de...9214df7ba70d [18:15:02] [02miraheze/phabricator-extensions] 07Universal-Omega 039214df7 - Cleanup [18:15:03] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:15:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.57, 7.35 [18:15:55] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:16:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.57, 8.07, 7.81 [18:17:28] [02miraheze/phabricator-extensions] 07Universal-Omega pushed 031 commit to 03cleanup [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/9214df7ba70d...ab3c09e6d6a1 [18:17:29] [02miraheze/phabricator-extensions] 07Universal-Omega 03ab3c09e - Update CI commit message [18:17:32] [02phabricator-extensions] 07Universal-Omega synchronize pull request 03#42: Cleanup - 13https://github.com/miraheze/phabricator-extensions/pull/42 [18:17:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.05, 6.71, 6.05 [18:18:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.83, 7.83, 7.74 [18:18:14] miraheze/phabricator-extensions - Universal-Omega the build passed. [18:18:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.48, 6.93 [18:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.75, 8.15, 7.61 [18:19:37] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.39, 5.93, 5.85 [18:21:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.90, 7.59 [18:21:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.15, 7.44, 7.53 [18:22:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.49, 7.33, 7.03 [18:24:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.10, 7.58, 7.15 [18:25:14] [02mw-config] 07Universal-Omega commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885393458 [18:26:11] [02mw-config] 07Universal-Omega commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885395130 [18:26:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.83, 7.63, 7.23 [18:28:05] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.51, 12.59, 10.46 [18:28:56] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.54, 10.43, 9.17 [18:29:25] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.65, 11.02, 9.78 [18:30:00] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.69, 11.57, 10.34 [18:30:29] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:30:56] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.24, 9.58, 9.01 [18:31:00] PROBLEM - wiki.gab.pt.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gab.pt.eu.org' expires in 15 day(s) (Fri 26 Jan 2024 06:21:28 PM GMT +0000). [18:31:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3866641b4d2f...42b9f9a06855 [18:31:20] [02miraheze/ssl] 07MirahezeSSLBot 0342b9f9a - Bot: Update SSL cert for wiki.gab.pt.eu.org [18:31:25] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.91, 9.62, 9.41 [18:31:55] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.15, 10.06, 9.93 [18:32:23] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.08, 6.47, 6.79 [18:36:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.26, 7.86, 7.41 [18:36:22] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 6.84, 6.88 [18:37:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.44, 7.38 [18:38:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.30, 6.63, 6.79 [18:39:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.50, 7.29, 7.33 [18:40:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.97, 7.69, 7.47 [18:41:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.68, 7.65, 7.44 [18:41:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.65, 8.01, 7.49 [18:43:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 7.46, 7.39 [18:47:12] [02mw-config] 07redbluegreenhat synchronize pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432 [18:47:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.41, 7.38, 7.36 [18:48:17] miraheze/mw-config - redbluegreenhat the build has errored. [18:49:27] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [18:51:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.25, 7.24 [18:51:37] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.90, 2.92, 2.01 [18:52:21] [02mw-config] 07redbluegreenhat synchronize pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432 [18:53:02] [02mw-config] 07redbluegreenhat commented on pull request 03#5432: T11040: Create $wmgMirahezeContactPageFooter - 13https://github.com/miraheze/mw-config/pull/5432#issuecomment-1885434894 [18:53:05] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:53:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.20, 7.22 [18:53:13] miraheze/mw-config - redbluegreenhat the build passed. [18:53:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.44, 2.22, 1.85 [18:54:21] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [18:57:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.21, 7.20 [19:00:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.09, 8.26, 7.61 [19:00:39] RECOVERY - wiki.gab.pt.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gab.pt.eu.org' will expire on Tue 09 Apr 2024 05:31:08 PM GMT +0000. [19:01:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.77, 7.45 [19:02:29] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [19:03:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.90, 6.87, 6.72 [19:03:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03mwtask181-mathoid [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8a7d525b0d79 [19:03:30] [02miraheze/mw-config] 07Universal-Omega 038a7d525 - Use mwtask181 for Mathoid [19:03:33] [02mw-config] 07Universal-Omega created branch 03mwtask181-mathoid - 13https://github.com/miraheze/mw-config [19:03:36] [02mw-config] 07Universal-Omega opened pull request 03#5433: Use mwtask181 for Mathoid - 13https://github.com/miraheze/mw-config/pull/5433 [19:03:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.36, 6.56, 6.80 [19:04:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.91, 7.97, 7.63 [19:04:59] miraheze/mw-config - Universal-Omega the build passed. [19:05:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.64, 7.45 [19:05:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.14, 7.10, 6.81 [19:06:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 13.58, 9.39, 8.15 [19:07:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.98, 7.04, 6.83 [19:07:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.21, 7.12, 6.98 [19:08:56] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [19:08:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03add-mwtask181 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1dc69d1cc53b [19:09:00] [02miraheze/mw-config] 07Universal-Omega 031dc69d1 - Also add mwtask181 [19:09:03] [02mw-config] 07Universal-Omega created branch 03add-mwtask181 - 13https://github.com/miraheze/mw-config [19:09:06] [02mw-config] 07Universal-Omega opened pull request 03#5434: Also add mwtask181 - 13https://github.com/miraheze/mw-config/pull/5434 [19:09:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.91, 6.91, 7.26 [19:09:20] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.17, 6.35, 6.60 [19:09:56] [02mw-config] 07Universal-Omega closed pull request 03#5434: Also add mwtask181 - 13https://github.com/miraheze/mw-config/pull/5434 [19:09:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/42dfc6c55acc...111c1f7f8046 [19:10:01] [02miraheze/mw-config] 07Universal-Omega 03111c1f7 - Also add mwtask181 (#5434) [19:10:02] miraheze/mw-config - Universal-Omega the build passed. [19:10:03] [02miraheze/mw-config] 07Universal-Omega deleted branch 03add-mwtask181 [19:10:06] [02mw-config] 07Universal-Omega deleted branch 03add-mwtask181 - 13https://github.com/miraheze/mw-config [19:10:20] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:10:44] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [19:10:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:51] miraheze/mw-config - Universal-Omega the build passed. [19:10:54] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.561 second response time [19:11:09] !log [@mwtask141] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 25s [19:11:09] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.02, 12.99, 8.67 [19:11:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.01, 7.45, 7.39 [19:11:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:17] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.49, 8.88, 6.68 [19:11:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.84, 7.45, 7.10 [19:13:05] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 5.95, 10.28, 8.18 [19:13:17] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 6.20, 7.84, 6.58 [19:13:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/111c1f7f8046...3bb708200af9 [19:13:42] [02miraheze/mw-config] 07paladox 033bb7082 - Add mw181, mw182 and mwtask181 to wgLinterSubmitterWhitelist [19:13:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.22, 7.28, 7.08 [19:14:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.12, 7.74, 7.84 [19:14:20] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:14:33] miraheze/mw-config - paladox the build passed. [19:14:55] [02mw-config] 07paladox closed pull request 03#5433: Use mwtask181 for Mathoid - 13https://github.com/miraheze/mw-config/pull/5433 [19:14:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3bb708200af9...680e21e37c18 [19:14:59] [02miraheze/mw-config] 07Universal-Omega 03680e21e - Use mwtask181 for Mathoid (#5433) [19:15:01] [02miraheze/mw-config] 07paladox deleted branch 03mwtask181-mathoid [19:15:01] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.28, 8.33, 7.72 [19:15:04] [02mw-config] 07paladox deleted branch 03mwtask181-mathoid - 13https://github.com/miraheze/mw-config [19:15:53] miraheze/mw-config - paladox the build passed. [19:16:18] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 6.84, 6.06 [19:17:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 7.57, 7.53 [19:17:27] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/680e21e37c18...0f7e9c4c8b4e [19:17:50] [02miraheze/mw-config] 07Universal-Omega 030f7e9c4 - mwtask141 -> mwtask181 [19:17:53] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [19:17:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.60, 6.44, 6.79 [19:17:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:13] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 20s [19:18:17] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.90, 6.37, 5.98 [19:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:19] !log [paladox@mwtask141] starting deploy of {'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [19:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:38] !log [paladox@mwtask141] finished deploy of {'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 18s [19:18:42] miraheze/mw-config - Universal-Omega the build passed. [19:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:21:00] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [19:21:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 8.13, 7.72 [19:22:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.50, 6.93, 6.73 [19:22:57] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.673 second response time [19:23:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.69, 7.60 [19:23:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.74, 6.91, 6.86 [19:24:18] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.55, 6.43, 6.57 [19:27:23] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [19:27:24] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 0s [19:27:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.41, 8.21, 7.34 [19:28:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.80, 7.49, 7.36 [19:29:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.45, 8.35, 7.77 [19:29:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.85, 7.96, 7.35 [19:30:01] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [19:30:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.77, 7.03, 6.77 [19:30:29] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:31:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/8a62a59778b1...8e5071cdc34e [19:31:56] [02miraheze/puppet] 07Universal-Omega 038e5071c - Re-add [19:31:57] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [19:31:58] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.562 second response time [19:32:09] !log [@mwtask181] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [19:32:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:38] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 - 13https://github.com/miraheze/puppet/pull/3619 [19:32:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e5071cdc34e...a651671e48ae [19:32:41] [02miraheze/puppet] 07Universal-Omega 03a651671 - Re-add [19:32:52] !log [@mwtask181] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] - SUCCESS in 43s [19:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.04, 7.72, 7.65 [19:33:16] [02puppet] 07Universal-Omega edited pull request 03#3619: Remove mwtask141 from deployment, jobrunner, mathoid, and varnish - 13https://github.com/miraheze/puppet/pull/3619 [19:33:20] miraheze/puppet - Universal-Omega the build passed. [19:33:24] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [19:33:59] miraheze/puppet - Universal-Omega the build passed. [19:34:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.39, 7.73, 7.54 [19:34:49] [02puppet] 07Universal-Omega synchronize pull request 03#3619: Remove mwtask141 from deployment, jobrunner, mathoid, and varnish - 13https://github.com/miraheze/puppet/pull/3619 [19:34:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mwtask141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a651671e48ae...53685191ddb5 [19:34:54] [02miraheze/puppet] 07Universal-Omega 035368519 - Fix port [19:36:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.15, 8.14, 7.71 [19:36:16] miraheze/puppet - Universal-Omega the build passed. [19:36:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.23, 7.75, 7.09 [19:37:14] [02puppet] 07Universal-Omega closed pull request 03#3619: Remove mwtask141 from deployment, jobrunner, mathoid, and varnish - 13https://github.com/miraheze/puppet/pull/3619 [19:37:16] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-mwtask141 [19:37:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/6a2dac451b76...37ce690be07b [19:37:20] [02miraheze/puppet] 07Universal-Omega 0337ce690 - Remove mwtask141 from deployment, jobrunner, mathoid, and varnish (#3619) [19:37:22] [02puppet] 07Universal-Omega deleted branch 03remove-mwtask141 - 13https://github.com/miraheze/puppet [19:38:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.52, 6.78, 6.81 [19:38:44] miraheze/puppet - Universal-Omega the build passed. [19:39:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 7.47, 7.34 [19:40:02] !log [paladox@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/extensions/RottenLinks/maintenance/updateExternalLinks.php [19:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.40, 6.70, 6.77 [19:41:43] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/80c0f926cc84...9dee5147a137 [19:41:46] [02miraheze/MirahezeDebug] 07Universal-Omega 039dee514 - Remove mwtask141 [19:41:54] !log [paladox@mwtask181] cancel [19:42:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:18] PROBLEM - cp35 Nginx Backend for mwtask141 on cp35 is CRITICAL: connect to address localhost and port 8150: Connection refused [19:43:02] PROBLEM - mwtask141 mathoid on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 8.33, 7.85 [19:44:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.07, 6.87, 6.82 [19:45:44] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.81, 7.01, 7.34 [19:46:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.58, 6.34, 6.63 [19:47:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0f7e9c4c8b4e...e461263342ce [19:47:17] [02miraheze/mw-config] 07Universal-Omega 03e461263 - Remove mwtask141 [19:47:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.47, 7.47, 7.68 [19:48:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.48, 7.79, 7.92 [19:48:09] miraheze/mw-config - Universal-Omega the build passed. [19:49:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.66, 8.08, 8.01 [19:50:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.77, 7.26, 6.90 [19:51:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 8.48, 8.05 [19:51:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.75, 7.84, 7.95 [19:52:10] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [19:52:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.23, 7.14, 6.93 [19:53:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.08, 7.17, 7.61 [19:54:18] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.75, 6.32, 6.65 [19:55:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.20, 7.76, 7.58 [19:57:24] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [19:57:25] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 0s [19:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.08, 7.48, 7.50 [19:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:09] PROBLEM - cp34 Nginx Backend for mwtask141 on cp34 is CRITICAL: connect to address localhost and port 8150: Connection refused [20:00:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.89, 6.59, 6.62 [20:00:32] PROBLEM - cp25 Nginx Backend for mwtask141 on cp25 is CRITICAL: connect to address localhost and port 8150: Connection refused [20:02:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.96, 6.32, 6.51 [20:02:22] !log [@mwtask181] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [20:02:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:40] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:03:04] !log [@mwtask181] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 41s [20:03:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:05:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.19, 7.53, 7.45 [20:09:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.37, 7.75, 7.58 [20:10:39] PROBLEM - cp24 Nginx Backend for mwtask141 on cp24 is CRITICAL: connect to address localhost and port 8150: Connection refused [20:11:47] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [20:12:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cb95758c4895...62967e0e0cb8 [20:12:53] [02miraheze/dns] 07Universal-Omega 0362967e0 - Remove mwtask141 CNAMEs [20:13:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.15, 6.11, 6.62 [20:13:19] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/reception/ImportDump/warriorclancatswiki-20240110151420.xml (START) [20:13:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.28, 7.70, 7.35 [20:13:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:15:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.90, 7.51, 7.31 [20:18:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.43, 6.97, 6.75 [20:19:40] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7967 MB (5% inode=99%); [20:20:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=warriorclancatswiki --no-updates --username-prefix=wikia:warriorclancats /home/reception/ImportDump/warriorclancatswiki-20240110151420.xml (END - exit=0) [20:20:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (START) [20:20:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.56, 7.56, 6.98 [20:23:14] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:24:31] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:24:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 7.38, 7.04 [20:26:30] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=warriorclancatswiki (END - exit=0) [20:26:31] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=warriorclancatswiki --active --update (END - exit=0) [20:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:27:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.03, 7.47, 7.35 [20:27:52] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.95, 6.44, 6.67 [20:29:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.73, 6.91, 7.17 [20:30:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.13, 7.19, 7.02 [20:31:15] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.04, 7.07, 7.08 [20:32:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.18, 7.04 [20:33:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.22, 7.51, 7.34 [20:34:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.70, 2.89, 2.34 [20:35:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.98, 7.00, 7.17 [20:35:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.76, 7.66, 7.34 [20:36:27] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.27, 2.69, 2.34 [20:37:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.65, 7.94, 7.48 [20:38:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.77, 8.28, 7.52 [20:38:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.12, 7.95, 7.49 [20:39:55] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.35, 4.55, 2.70 [20:40:55] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:40:56] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:41:08] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:41:12] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:41:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.26, 7.55, 7.47 [20:41:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.09, 7.11, 6.82 [20:42:53] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.73, 9.79, 6.77 [20:42:55] PROBLEM - mwtask141 HTTPS on mwtask141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.769 second response time [20:42:57] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.618 second response time [20:43:05] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.658 second response time [20:43:10] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.548 second response time [20:43:16] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.45, 10.70, 7.77 [20:44:34] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.94, 9.43, 7.83 [20:44:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.94, 8.29, 6.58 [20:46:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.74, 7.41, 7.51 [20:46:34] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.17, 7.77, 7.41 [20:46:48] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 7.97, 7.75 [20:47:12] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.65, 9.29, 7.88 [20:48:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.95, 7.70 [20:49:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.30, 7.38, 7.02 [20:51:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.95, 7.42, 7.09 [20:51:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [20:52:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.90, 7.84 [20:53:27] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:53:29] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.530 second response time [20:55:15] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.53, 8.12, 7.70 [20:55:25] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.604 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [20:55:32] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.38, 6.59, 6.80 [20:56:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.90, 7.83 [20:58:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 7.55, 7.72 [20:58:54] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:58:58] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 17.16, 12.73, 10.16 [20:59:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [20:59:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.54, 7.37, 7.50 [20:59:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:00:54] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.508 second response time [21:00:55] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.62, 11.36, 10.03 [21:00:59] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.974 second response time [21:01:28] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.43, 10.98, 9.25 [21:01:29] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.692 second response time [21:02:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.03, 7.43, 7.53 [21:02:53] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.56, 9.86, 9.61 [21:03:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.08, 7.88, 7.64 [21:03:52] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [21:04:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 7.39, 7.50 [21:05:14] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:05:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03expand_path [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f8e570f36275 [21:05:23] [02miraheze/puppet] 07Universal-Omega 03f8e570f - Fix expand_path for Ruby3 [21:05:24] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.77, 12.74, 10.25 [21:05:25] [02puppet] 07Universal-Omega created branch 03expand_path - 13https://github.com/miraheze/puppet [21:05:28] [02puppet] 07Universal-Omega opened pull request 03#3623: Fix expand_path for Ruby3 - 13https://github.com/miraheze/puppet/pull/3623 [21:07:12] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.633 second response time [21:07:22] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.48, 11.91, 10.30 [21:07:25] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.38, 10.46, 9.06 [21:07:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03expand_path [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f8e570f36275...937ffe5da932 [21:07:30] [02miraheze/puppet] 07Universal-Omega 03937ffe5 - Use expand_path again [21:07:31] [02puppet] 07Universal-Omega synchronize pull request 03#3623: Fix expand_path for Ruby3 - 13https://github.com/miraheze/puppet/pull/3623 [21:07:49] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.39, 10.52, 9.22 [21:09:25] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.16, 9.25, 8.78 [21:09:44] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.20, 9.49, 8.98 [21:10:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.40, 7.29 [21:11:18] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.39, 9.15, 9.54 [21:12:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.41, 7.32 [21:13:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.46, 7.99, 7.92 [21:19:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.68, 7.43 [21:20:50] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.12, 7.38, 6.92 [21:21:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 7.66, 7.46 [21:22:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.93, 6.90, 6.80 [21:23:06] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [21:23:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.41, 7.33, 7.43 [21:24:43] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.63, 6.63, 6.73 [21:25:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 7.96, 7.56 [21:27:07] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [21:27:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.12, 7.45, 7.47 [21:27:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 7.80, 7.55 [21:29:07] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.599 second response time [21:31:20] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [21:31:31] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [21:33:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.66, 7.73, 7.57 [21:33:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.01, 6.51, 6.75 [21:34:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.67, 6.94, 6.79 [21:35:24] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:37:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.84, 7.51, 7.12 [21:38:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.43, 7.61, 7.08 [21:39:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [21:40:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.94, 7.31, 7.04 [21:41:29] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.664 second response time [21:42:11] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.05, 7.94, 7.30 [21:43:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.44, 7.78, 7.46 [21:44:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.85, 7.64, 7.27 [21:45:24] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.70, 7.92, 7.99 [21:45:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.35, 7.34 [21:47:46] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.365 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [21:48:22] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [21:48:27] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 4s [21:48:28] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [21:48:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:48:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.40, 7.93, 7.97 [21:49:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.94, 8.00, 7.54 [21:51:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.53, 7.45, 7.79 [21:51:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.37, 7.90, 7.42 [21:52:35] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:53:45] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.55, 3.18, 3.99 [21:53:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.33, 7.61, 7.37 [21:54:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/37ce690be07b...d2c039e6ac08 [21:54:16] [02miraheze/puppet] 07paladox 03d2c039e - jobrunner-hi: increase parsoid runners to 8 [21:54:21] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:57:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.69, 7.67 [21:57:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [21:57:44] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.42, 3.91 [21:58:54] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03T11467 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/commit/8e465fc5e9ac [21:58:55] [02miraheze/RottenLinks] 07Universal-Omega 038e465fc - T11467: use getFullURL instead of getLocalURL [21:58:57] [02RottenLinks] 07Universal-Omega created branch 03T11467 - 13https://github.com/miraheze/RottenLinks [21:58:59] [02RottenLinks] 07Universal-Omega opened pull request 03#75: T11467: use getFullURL instead of getLocalURL - 13https://github.com/miraheze/RottenLinks/pull/75 [21:59:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.19, 8.11, 7.81 [21:59:29] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.509 second response time [21:59:44] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.91, 3.45, 3.87 [22:00:46] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.62, 11.22, 8.80 [22:01:23] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.74, 12.73, 9.73 [22:01:31] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:01:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.48, 7.83, 7.40 [22:02:44] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.06, 11.03, 9.04 [22:02:52] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [22:03:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.73, 7.73 [22:03:21] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.99, 11.43, 9.60 [22:03:29] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.34, 7.57, 7.50 [22:04:10] [02RottenLinks] 07Universal-Omega closed pull request 03#75: T11467: use getFullURL instead of getLocalURL - 13https://github.com/miraheze/RottenLinks/pull/75 [22:04:12] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/395ffbd83259...07283174091e [22:04:14] [02miraheze/RottenLinks] 07Universal-Omega 030728317 - T11467: use getFullURL instead of getLocalURL (#75) [22:04:16] miraheze/RottenLinks - Universal-Omega the build has errored. [22:04:17] [02miraheze/RottenLinks] 07Universal-Omega deleted branch 03T11467 [22:04:20] [02RottenLinks] 07Universal-Omega deleted branch 03T11467 - 13https://github.com/miraheze/RottenLinks [22:04:41] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.02, 10.01, 8.92 [22:05:10] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:05:12] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [22:05:18] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:05:19] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.80, 9.94, 9.26 [22:05:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:05:44] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.56, 4.20, 3.98 [22:06:37] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 21.95, 15.30, 10.31 [22:07:04] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.634 second response time [22:07:12] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.749 second response time [22:07:19] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.629 second response time [22:07:29] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.503 second response time [22:07:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 7.96, 7.63 [22:08:01] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [22:08:04] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:09:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.20, 7.37, 7.24 [22:09:58] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.611 second response time [22:09:58] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.649 second response time [22:10:25] miraheze/RottenLinks - Universal-Omega the build passed. [22:10:45] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.40'], 'upgrade_extensions': ['RottenLinks']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [22:10:48] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.40'], 'upgrade_extensions': ['RottenLinks']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 3s [22:10:50] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.40'], 'upgrade_extensions': ['RottenLinks']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [22:11:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.79, 7.66 [22:11:21] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [22:11:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.86, 7.47, 7.29 [22:11:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:12:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:13:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:13:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 7.34, 7.50 [22:13:24] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.869 second response time [22:13:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.62, 7.70, 7.94 [22:13:44] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [22:13:47] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [22:13:52] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [22:14:57] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.40'], 'upgrade_extensions': ['RottenLinks']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 252s [22:14:59] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.514 second response time [22:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:42] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 17 backends are healthy [22:15:42] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 17 backends are healthy [22:15:50] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 17 backends are healthy [22:20:27] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:20:35] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [22:23:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [22:23:39] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [22:24:59] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.553 second response time [22:25:01] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.52, 11.20, 9.69 [22:25:35] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.81, 7.70, 7.67 [22:25:48] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:25:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.92, 7.64, 7.38 [22:26:56] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.06, 9.74, 9.33 [22:27:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 7.84, 7.74 [22:27:43] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.396 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [22:27:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.38, 7.20, 7.26 [22:28:55] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [22:30:22] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.74, 9.79, 11.78 [22:30:52] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:30:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.37, 7.29 [22:31:31] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [22:32:20] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.35, 12.27, 12.50 [22:32:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.50, 7.34 [22:33:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.19, 7.93, 7.73 [22:34:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.15, 8.01, 7.54 [22:35:10] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [22:35:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.37, 7.54, 7.21 [22:36:16] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.76, 10.62, 11.81 [22:37:04] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.025 second response time [22:38:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.52, 7.62, 7.52 [22:39:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [22:39:30] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [22:39:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.20, 7.24, 7.16 [22:40:16] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.59, 13.85, 12.80 [22:41:30] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.622 second response time [22:43:05] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.562 second response time [22:44:16] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.68, 10.75, 11.82 [22:44:50] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.92, 11.68, 10.06 [22:44:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.69, 7.52 [22:45:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.48, 7.71, 7.93 [22:45:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.15, 2.64, 3.68 [22:46:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.64, 9.90, 9.60 [22:46:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.86, 7.62 [22:47:28] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.60, 6.82, 6.05 [22:48:00] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.02, 3.66 [22:49:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.50, 6.65, 6.08 [22:49:52] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.25, 9.57, 8.39 [22:50:16] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.09, 7.90, 10.09 [22:51:47] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.66, 9.63, 8.59 [22:52:00] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.16, 3.53, 3.77 [22:55:32] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.33, 6.51, 6.75 [22:57:03] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 4120s [22:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.59, 2.07, 3.06 [22:58:55] [02puppet] 07paladox closed pull request 03#3623: Fix expand_path for Ruby3 - 13https://github.com/miraheze/puppet/pull/3623 [22:58:56] [02miraheze/puppet] 07paladox deleted branch 03expand_path [22:58:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/d2c039e6ac08...da59d73570b0 [22:58:59] [02miraheze/puppet] 07Universal-Omega 03da59d73 - Fix expand_path for Ruby3 (#3623) [22:59:01] [02puppet] 07paladox deleted branch 03expand_path - 13https://github.com/miraheze/puppet [22:59:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.45, 7.43, 7.48 [23:00:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.40, 7.73, 7.28 [23:01:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.13, 8.34, 7.71 [23:02:06] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:02:25] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.94, 7.14, 6.91 [23:03:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.78, 7.94, 7.65 [23:04:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 7.48, 7.29 [23:09:32] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.40, 6.20, 6.64 [23:09:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.90, 7.46, 7.65 [23:12:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.42, 7.25 [23:13:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.24, 7.63, 7.66 [23:14:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.10, 7.14 [23:15:37] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.92, 7.75, 7.70 [23:15:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.35, 7.75, 7.60 [23:17:37] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.15, 7.79, 7.72 [23:18:58] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [23:19:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.97, 7.21, 7.43 [23:21:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.77, 7.00, 6.85 [23:21:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.98, 7.64, 7.71 [23:22:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.60, 7.74, 7.40 [23:24:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.26, 6.97, 7.16 [23:25:13] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.53, 6.39, 6.64 [23:26:10] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:28:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.01, 7.10 [23:30:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 6.76, 7.00 [23:31:31] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:05] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:08] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:52] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [23:33:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.56, 7.58, 7.55 [23:35:56] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.48, 6.81, 6.78 [23:37:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.27, 8.48, 7.72 [23:38:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.92, 7.46 [23:39:50] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.52, 6.44, 6.65 [23:40:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.59, 7.39 [23:42:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.72, 8.25, 7.64 [23:43:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.45, 7.28, 7.66 [23:43:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.47, 7.75, 7.64 [23:44:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.25, 7.80, 7.54 [23:48:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.71, 7.54 [23:50:15] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:50:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.05, 7.29, 7.42 [23:53:48] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.20, 7.81, 7.60 [23:53:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.82, 8.18, 7.80 [23:57:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.21, 7.52, 7.55 [23:57:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.84, 7.86, 7.81 [23:59:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.52, 8.17, 7.80