[00:01:35] PROBLEM - puppet181 SSH on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:42] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 6.20, 3.52, 2.06 [00:02:55] PROBLEM - db112 Puppet on db112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [00:03:10] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.14, 7.77, 3.61 [00:03:31] RECOVERY - puppet181 SSH on puppet181 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:04:42] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.58, 3.09, 2.08 [00:05:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/546400244eb5...f21f56b7461a [00:05:26] [02miraheze/puppet] 07paladox 03f21f56b - Update graylog131.yaml [00:05:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f21f56b7461a...3739b0587033 [00:05:36] [02miraheze/puppet] 07paladox 033739b05 - Update jobchron121.yaml [00:05:56] PROBLEM - db121 Puppet on db121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [00:06:22] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:07:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 3.55, 2.23 [00:07:46] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:07:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/3739b0587033...ce527af6c81b [00:07:57] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:07:59] [02miraheze/puppet] 07paladox 03ce527af - Delete hieradata/hosts/ldap141.yaml [00:08:00] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:08:08] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10000 milliseconds [00:08:14] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:08:22] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.96, 3.41, 2.94 [00:08:24] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 hour ago with 0 failures [00:08:44] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:26:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3013e2372062...f451970b39a0 [00:26:44] [02miraheze/puppet] 07paladox 03f451970 - varnish: fix firewall [00:27:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f451970b39a0...b46b44e4a4e7 [00:27:21] [02miraheze/puppet] 07paladox 03b46b44e - Update mon141.yaml [00:27:27] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 79s [00:28:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b46b44e4a4e7...60aa572a2d41 [00:28:39] [02miraheze/puppet] 07paladox 0360aa572 - Update mw131.yaml [00:28:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/60aa572a2d41...c51f8b46925a [00:28:48] [02miraheze/puppet] 07paladox 03c51f8b4 - Update mw132.yaml [00:28:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c51f8b46925a...c64d41640953 [00:28:56] [02miraheze/puppet] 07paladox 03c64d416 - Update mw133.yaml [00:28:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.79, 8.31, 8.05 [00:29:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c64d41640953...2b05c1e68ce6 [00:29:04] [02miraheze/puppet] 07paladox 032b05c1e - Update mw134.yaml [00:29:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2b05c1e68ce6...f0378d43e721 [00:29:08] [02miraheze/puppet] 07paladox 03f0378d4 - Update mw141.yaml [00:29:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f0378d43e721...5c80b5972371 [00:29:15] [02miraheze/puppet] 07paladox 035c80b59 - Update mw142.yaml [00:29:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c80b5972371...a0b314e80e06 [00:29:23] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:29:23] [02miraheze/puppet] 07paladox 03a0b314e - Update mw143.yaml [00:29:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a0b314e80e06...e0e992974276 [00:29:31] [02miraheze/puppet] 07paladox 03e0e9929 - Update mwtask141.yaml [00:30:32] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:30:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.58, 7.60, 7.82 [00:32:01] PROBLEM - swiftobject113 Puppet on swiftobject113 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 hour ago with 0 failures [00:33:09] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:33:09] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:33:39] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:34:36] PROBLEM - mw132 Puppet on mw132 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [00:34:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 8.13, 7.98 [00:35:21] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.57, 3.59, 3.17 [00:35:45] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 hour ago with 0 failures [00:35:47] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.13, 7.76, 7.93 [00:36:09] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:37:07] PROBLEM - mw133 Puppet on mw133 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:37:10] PROBLEM - mw134 Puppet on mw134 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:37:21] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.54, 3.14, 3.05 [00:37:28] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:39:41] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:39:47] PROBLEM - mw141 Puppet on mw141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [00:39:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e0e992974276...b5c8da2ddc94 [00:39:50] [02miraheze/puppet] 07paladox 03b5c8da2 - Update ns1.yaml [00:39:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b5c8da2ddc94...86b6683b12a4 [00:39:57] [02miraheze/puppet] 07paladox 0386b6683 - Update ns2.yaml [00:40:09] PROBLEM - mw142 Puppet on mw142 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:40:35] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:41:23] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.90, 3.57, 3.23 [00:41:28] PROBLEM - cp35 Puppet on cp35 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:43:23] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.89, 3.29, 3.17 [00:44:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/86b6683b12a4...aa084a31327c [00:44:19] [02miraheze/puppet] 07paladox 03aa084a3 - Update os131.yaml [00:44:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa084a31327c...acc0fd20832b [00:44:27] [02miraheze/puppet] 07paladox 03acc0fd2 - Update os141.yaml [00:44:35] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:44:58] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:45:38] PROBLEM - mw143 Puppet on mw143 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:45:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/acc0fd20832b...58d6d489d2db [00:45:44] [02miraheze/puppet] 07paladox 0358d6d48 - Update phab121.yaml [00:46:29] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:48:53] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:49:06] [1/3] I'll help the first 20 people interested in how to earn $100k within 72hours but you will reimburse me 10% of your dividend when you collect it. Note: only interested people should send a friend request or  send me a dm! asking  me HOW! Or contact via WhatsApp, [00:49:07] [2/3] 👇 [00:49:07] [3/3] +1 (508) 214‑8755 [00:49:11] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:49:21] PROBLEM - os131 Puppet on os131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:49:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/58d6d489d2db...c00151d78c3e [00:49:32] [02miraheze/puppet] 07paladox 03c00151d - Update swiftac111.yaml [00:49:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c00151d78c3e...80313c332170 [00:49:41] [02miraheze/puppet] 07paladox 0380313c3 - Update swiftobject101.yaml [00:49:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80313c332170...ca0faad4e7b3 [00:49:46] [02miraheze/puppet] 07paladox 03ca0faad - Update swiftobject111.yaml [00:49:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca0faad4e7b3...49c3616f0596 [00:49:54] [02miraheze/puppet] 07paladox 0349c3616 - Update swiftobject112.yaml [00:49:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.34, 7.97, 7.89 [00:49:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/49c3616f0596...daf1e6ac1e7a [00:49:59] [02miraheze/puppet] 07paladox 03daf1e6a - Update swiftobject113.yaml [00:50:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/daf1e6ac1e7a...19097ff69ec7 [00:50:07] [02miraheze/puppet] 07paladox 0319097ff - Update swiftobject121.yaml [00:50:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/19097ff69ec7...c428a1b0d544 [00:50:14] [02miraheze/puppet] 07paladox 03c428a1b - Update swiftobject122.yaml [00:50:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c428a1b0d544...0e461f08cf32 [00:50:18] [02miraheze/puppet] 07paladox 030e461f0 - Update swiftproxy111.yaml [00:50:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e461f08cf32...1911099e5c88 [00:50:29] [02miraheze/puppet] 07paladox 031911099 - Update swiftproxy131.yaml [00:50:35] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [00:51:16] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.95, 1.75, 1.45 [00:51:35] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:51:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1911099e5c88...708f3a18b1af [00:51:54] [02miraheze/puppet] 07paladox 03708f3a1 - reports: fix firewall [00:51:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.41, 7.76, 7.82 [00:52:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/708f3a18b1af...87fa68b906ea [00:52:19] [02miraheze/puppet] 07paladox 0387fa68b - db: fix firewall [00:52:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/87fa68b906ea...e4a6b6a86cf0 [00:52:52] PROBLEM - phab121 Puppet on phab121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:52:55] [02miraheze/puppet] 07paladox 03e4a6b6a - irc: fix firewall [00:53:10] PROBLEM - reports121 Puppet on reports121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [00:53:15] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.55, 1.60, 1.42 [00:53:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e4a6b6a86cf0...d4ed71dbf307 [00:53:20] [02miraheze/puppet] 07paladox 03d4ed71d - mathoid: fix firewall [00:53:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d4ed71dbf307...a6449168ea54 [00:53:38] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.79, 3.18 [00:53:39] [02miraheze/puppet] 07paladox 03a644916 - matomo: fix firewall [00:53:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a6449168ea54...130325dab74f [00:54:01] [02miraheze/puppet] 07paladox 03130325d - memcached: fix firewall [00:54:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/130325dab74f...4adbac3c9cf7 [00:54:40] [02miraheze/puppet] 07paladox 034adbac3 - openldap: fix firewall [00:55:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4adbac3c9cf7...a869a9e555d3 [00:55:10] [02miraheze/puppet] 07paladox 03a869a9e - opensearch: fix firewall [00:55:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a869a9e555d3...129c8dc8d487 [00:55:28] [02miraheze/puppet] 07paladox 03129c8dc - Phabricator: fix firewall [00:55:35] PROBLEM - swiftac111 Puppet on swiftac111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:55:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/129c8dc8d487...d3381e31e831 [00:55:46] [02miraheze/puppet] 07paladox 03d3381e3 - poolcounter: fix firewall [00:56:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d3381e31e831...484c79ebb17f [00:56:03] [02miraheze/puppet] 07paladox 03484c79e - PostgreSQL: fix firewall [00:56:22] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.84, 3.97, 3.30 [00:56:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/484c79ebb17f...2771f5f81f48 [00:56:27] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:56:28] [02miraheze/puppet] 07paladox 032771f5f - puppetdb: fix firewall [00:56:35] PROBLEM - os131 Puppet on os131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:56:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2771f5f81f48...159c978e1633 [00:56:49] [02miraheze/puppet] 07paladox 03159c978 - redis: fix firewall [00:56:55] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.56, 11.41, 8.71 [00:56:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.69, 7.71, 7.96 [00:57:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/159c978e1633...9c923dcebc22 [00:57:09] [02miraheze/puppet] 07paladox 039c923dc - roundcubemail: fix firewall [00:57:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9c923dcebc22...ff8d152c1cc6 [00:57:28] [02miraheze/puppet] 07paladox 03ff8d152 - ssl: fix firewall [00:57:51] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.37, 10.78, 8.72 [00:57:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff8d152c1cc6...8850f112dc49 [00:57:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.85, 7.92, 7.83 [00:57:57] [02miraheze/puppet] 07paladox 038850f11 - swift: fix firewall [00:58:38] PROBLEM - mem131 Puppet on mem131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 41 minutes ago with 0 failures [00:58:40] PROBLEM - os131 Puppet on os131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [00:59:03] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.70, 11.14, 9.21 [00:59:10] RECOVERY - puppet181 SSH on puppet181 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:59:38] RECOVERY - puppet181 ferm_active on puppet181 is OK: OK ferm input default policy is set [00:59:43] RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.198 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [00:59:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.35, 7.73, 7.78 [00:59:59] RECOVERY - puppet181 Current Load on puppet181 is OK: OK - load average: 0.16, 0.09, 0.12 [01:00:08] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [01:00:12] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 25062 MB (82% inode=93%); [01:00:13] RECOVERY - puppet181 NTP time on puppet181 is OK: NTP OK: Offset 0.02287870646 secs [01:00:15] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:00:30] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:00:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.05, 11.71, 9.47 [01:01:02] RECOVERY - puppet181 conntrack_table_size on puppet181 is OK: OK: nf_conntrack is 0 % full [01:01:02] RECOVERY - puppet181 puppetdb on puppet181 is OK: TCP OK - 0.129 second response time on 2602:294:0:b12::101 port 8081 [01:01:06] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:01:51] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.46, 12.84, 9.94 [01:02:00] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:02:42] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.37, 12.54, 10.02 [01:02:47] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:02:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.06, 8.66, 6.63 [01:03:03] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.70, 11.55, 9.88 [01:03:15] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:03:50] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.36, 11.11, 9.67 [01:03:56] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.182 second response time [01:04:37] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.13, 10.59, 9.62 [01:04:38] PROBLEM - swiftobject112 Puppet on swiftobject112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [01:04:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.71, 7.29, 6.36 [01:04:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.73, 7.78, 7.80 [01:05:04] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [01:05:13] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:05:35] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:05:50] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.65, 10.00, 9.44 [01:06:32] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.60, 9.71, 9.42 [01:06:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.83, 7.65, 7.75 [01:07:03] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.06, 9.34, 9.37 [01:07:16] PROBLEM - prometheus131 Puppet on prometheus131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:07:36] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:07:52] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.85, 9.97, 7.57 [01:08:22] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.25, 3.87, 3.85 [01:08:30] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:08:33] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 4.54, 3.74 [01:08:50] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.21, 10.07, 7.74 [01:08:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.75, 7.97, 7.84 [01:09:35] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:09:37] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.10, 3.69, 4.00 [01:09:48] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.71, 8.64, 7.39 [01:10:29] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.232 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:10:39] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:10:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.96, 8.55, 7.47 [01:10:52] PROBLEM - swiftobject111 Puppet on swiftobject111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [01:11:07] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.31, 8.02, 7.88 [01:12:32] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.76, 4.00, 3.69 [01:12:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8850f112dc49...52e6dbc4e235 [01:12:43] [02miraheze/puppet] 07paladox 0352e6dbc - Update test131.yaml [01:12:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.42, 7.63, 7.74 [01:13:01] PROBLEM - swiftobject113 Puppet on swiftobject113 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:13:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.64, 7.61, 7.75 [01:14:02] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:14:31] PROBLEM - swiftobject121 Puppet on swiftobject121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:14:37] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.28, 4.02, 3.73 [01:15:08] PROBLEM - swiftobject122 Puppet on swiftobject122 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [01:15:48] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_checkout_MediaWiki-REL1_41 WikiSEO] [01:16:30] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [01:16:39] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.17, 3.55, 3.59 [01:16:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.67, 7.68 [01:17:28] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.46, 7.87, 7.81 [01:19:23] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.35, 2.57, 3.24 [01:19:44] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [01:19:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.74, 7.69, 7.74 [01:20:13] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:20:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/52e6dbc4e235...4515efda5152 [01:20:53] [02miraheze/puppet] 07paladox 034515efd - salt: fix getting hosts [01:22:22] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.82, 2.70, 3.23 [01:22:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c71d22342f0a [01:22:39] [02miraheze/puppet] 07paladox 03c71d223 - prometheus: fix firewall [01:22:42] [02puppet] 07paladox created branch 03paladox-patch-16 - 13https://github.com/miraheze/puppet [01:22:43] [02puppet] 07paladox opened pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:22:43] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.55, 3.60, 3.55 [01:23:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c71d22342f0a...650661127760 [01:23:03] [02miraheze/puppet] 07paladox 036506611 - Update cadvisor.pp [01:23:04] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:23:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/650661127760...756abba2c327 [01:23:17] [02miraheze/puppet] 07paladox 03756abba - Update elasticsearch.pp [01:23:20] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:23:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/756abba2c327...9d1f3a80ad01 [01:23:30] [02miraheze/puppet] 07paladox 039d1f3a8 - Update fpm.pp [01:23:31] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:23:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d1f3a80ad01...a82de872fa19 [01:23:50] [02miraheze/puppet] 07paladox 03a82de87 - Update jmx.pp [01:23:51] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:23:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.53, 8.07, 7.84 [01:24:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a82de872fa19...a76cd51ee08d [01:24:02] [02miraheze/puppet] 07paladox 03a76cd51 - Update mariadb.pp [01:24:05] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:24:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a76cd51ee08d...aff2ef167196 [01:24:15] [02miraheze/puppet] 07paladox 03aff2ef1 - Update memcached.pp [01:24:15] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.publictestwiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:24:18] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:24:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aff2ef167196...6c1c62b560c6 [01:24:22] [02miraheze/puppet] 07paladox 036c1c62b - Update nginx.pp [01:24:24] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:24:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6c1c62b560c6...2c3afa62c8f7 [01:24:38] [02miraheze/puppet] 07paladox 032c3afa6 - Update openldap.pp [01:24:39] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:24:45] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.54, 3.66, 3.59 [01:24:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2c3afa62c8f7...dc1d995ad365 [01:24:46] [02miraheze/puppet] 07paladox 03dc1d995 - Update postfix.pp [01:24:47] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:24:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc1d995ad365...b6bb490d294d [01:24:56] [02miraheze/puppet] 07paladox 03b6bb490 - Update redis.pp [01:24:59] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:25:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b6bb490d294d...6cf9d67ef708 [01:25:09] [02miraheze/puppet] 07paladox 036cf9d67 - Update statsd_exporter.pp [01:25:11] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:25:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-16 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6cf9d67ef708...2794b3b9e241 [01:25:19] [02miraheze/puppet] 07paladox 032794b3b - Update varnish.pp [01:25:21] [02puppet] 07paladox synchronize pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:25:32] [02puppet] 07paladox closed pull request 03#3598: prometheus: fix firewall - 13https://github.com/miraheze/puppet/pull/3598 [01:25:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-16 [01:25:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±13] 13https://github.com/miraheze/puppet/compare/4515efda5152...176984edafd9 [01:25:38] [02miraheze/puppet] 07paladox 03176984e - prometheus: fix firewall (#3598) [01:25:39] [02puppet] 07paladox deleted branch 03paladox-patch-16 - 13https://github.com/miraheze/puppet [01:25:40] [02puppet] 07paladox closed pull request 03#3596: ssl-renew: switch to using puppet181 - 13https://github.com/miraheze/puppet/pull/3596 [01:25:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [01:25:45] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [01:25:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.42, 7.78, 7.76 [01:27:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/176984edafd9...06f46203203d [01:27:33] [02miraheze/puppet] 07paladox 0306f4620 - varnish: switch to puppet181 [01:27:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.53, 8.13, 7.90 [01:28:32] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:28:44] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.99, 3.18, 3.40 [01:28:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.44, 8.00, 7.94 [01:29:28] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [01:29:34] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:29:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.42, 7.32, 7.65 [01:30:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.69, 8.31, 8.02 [01:31:12] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [01:31:17] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:31:58] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.73, 8.05, 7.88 [01:32:09] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:32:17] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:33:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/06f46203203d...2d7f44d9cb76 [01:33:14] [02miraheze/puppet] 07paladox 032d7f44d - bastion: fix firewall [01:33:18] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:33:48] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:33:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.75, 7.70, 7.78 [01:34:16] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:35:17] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [01:35:29] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.publictestwiki.com could not be found [01:35:56] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:35:57] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.23, 8.28, 8.01 [01:36:31] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:36:49] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:36:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.63, 3.49, 3.46 [01:37:41] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:38:44] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:49] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.57, 7.47, 7.74 [01:40:08] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:40:23] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:40:41] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:40:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.85, 7.66, 7.92 [01:41:31] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [01:42:06] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.154 second response time [01:42:27] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:43:50] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:44:12] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:45:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.95, 8.32, 8.07 [01:45:12] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.48, 3.46 [01:45:20] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:45:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 10.62, 7.88, 7.69 [01:46:07] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.45 ms [01:46:32] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:47:09] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.78, 3.62, 3.52 [01:47:28] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:48:51] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [01:48:55] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:49:07] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.97, 2.98, 3.29 [01:49:27] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:50:09] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:51:58] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:52:22] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:53:26] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for publictestwiki.com could not be found [01:53:55] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:55:03] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:55:11] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:56:16] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:56:44] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:56:55] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:57:14] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:57:28] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:57:37] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.97, 7.90, 7.91 [01:59:30] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:59:55] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:01:18] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:01:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.01, 7.77, 7.82 [02:02:42] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.26, 10.15, 7.88 [02:02:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 4.14, 3.57 [02:04:41] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.34, 8.66, 7.61 [02:04:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.80, 3.55, 3.41 [02:05:27] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:06:47] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.72, 3.33, 3.35 [02:06:58] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:09:17] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.59, 7.52, 7.78 [02:10:42] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.71, 4.33, 3.75 [02:12:41] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.35, 3.60, 3.55 [02:13:04] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:13:54] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:13] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [02:14:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.59, 4.60, 3.91 [02:14:42] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [02:15:18] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:12] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.164 second response time [02:17:41] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:17:51] !log [paladox@test131] starting deploy of {'config': True, 'versions': ['1.41']} to [test131] [02:17:52] !log [paladox@test131] finished deploy of {'config': True, 'versions': ['1.41']} to [test131] - SUCCESS in 1s [02:18:09] !log [paladox@test131] starting deploy of {'config': True, 'versions': ['1.41']} to [test131] [02:18:10] !log [paladox@test131] finished deploy of {'config': True, 'versions': ['1.41']} to [test131] - SUCCESS in 0s [02:18:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.48, 3.75, 3.76 [02:19:20] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config],Exec[git_checkout_MediaWiki-REL1_41 WikiSEO] [02:22:37] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 4.01, 3.83 [02:22:48] RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'publictestwiki.com' will expire on Fri 01 Mar 2024 00:25:08 GMT +0000. [02:23:00] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:23:11] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.33, 1.53, 1.18 [02:23:12] RECOVERY - publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - publictestwiki.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [02:24:56] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.162 second response time [02:25:11] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.06, 1.34, 1.15 [02:25:16] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:25:23] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 4 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/home/salt-user],File[/etc/logrotate.d/swift],File[/etc/rsyslog.d/40-swift.conf],File[/etc/sysctl.d] [02:26:32] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.17, 3.38, 3.72 [02:28:29] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.49, 2.38, 3.31 [02:30:47] !log [paladox@test131] starting deploy of {'versions': ['1.41'], 'upgrade_extensions': ['WikiSEO']} to [test131] [02:30:48] !log [paladox@test131] finished deploy of {'versions': ['1.41'], 'upgrade_extensions': ['WikiSEO']} to [test131] - SUCCESS in 0s [02:30:49] !log [paladox@test131] starting deploy of {'versions': ['1.41'], 'upgrade_extensions': ['WikiSEO']} to [test131] [02:30:50] !log [paladox@test131] finished deploy of {'versions': ['1.41'], 'upgrade_extensions': ['WikiSEO']} to [test131] - SUCCESS in 3s [02:31:06] !log [paladox@test131] starting deploy of {'world': True, 'versions': ['1.41']} to [test131] [02:31:07] !log [paladox@test131] finished deploy of {'world': True, 'versions': ['1.41']} to [test131] - SUCCESS in 0s [02:31:08] !log [paladox@test131] starting deploy of {'world': True, 'versions': ['1.41']} to [test131] [02:31:27] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:16] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.mahdiruiz.line.pm and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [02:32:18] !log [paladox@test131] finished deploy of {'world': True, 'versions': ['1.41']} to [test131] - SUCCESS in 71s [02:35:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.63, 7.54, 7.44 [02:37:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.11, 7.40, 7.41 [02:40:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.46, 7.79, 7.99 [02:41:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.83, 7.36, 7.82 [02:44:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.00, 7.99, 8.03 [02:46:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.88, 7.83, 7.96 [02:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.57, 6.88, 7.82 [02:47:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.54, 7.22, 7.21 [02:47:22] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.84 ms [02:47:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.65, 7.32, 7.59 [02:48:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.28, 8.29, 8.10 [02:48:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.72, 7.72, 8.04 [02:49:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.62, 7.43, 7.30 [02:49:25] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.70 ms [02:49:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.74, 7.40, 7.58 [02:50:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.38, 7.83, 7.95 [02:50:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.17, 7.08, 7.76 [02:51:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.12, 7.98, 7.77 [02:52:13] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:53:16] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:53:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.56, 7.45, 7.60 [02:53:41] RECOVERY - barengreza.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'barengreza.my.id' will expire on Sun 17 Mar 2024 17:33:19 GMT +0000. [02:54:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.80, 7.81, 7.89 [02:55:05] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:55:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.52, 7.86, 7.73 [02:56:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.85, 8.12, 8.00 [02:57:20] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.40, 7.64, 7.66 [02:57:51] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [02:58:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.90, 7.88, 7.91 [03:00:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.37, 7.95, 7.93 [03:01:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.63, 7.55, 7.56 [03:01:58] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Mon 01 Apr 2024 17:11:49 GMT +0000. [03:03:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.54, 7.44, 7.52 [03:04:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.85, 7.70, 7.88 [03:05:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.33, 7.78, 7.63 [03:06:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.01, 7.99, 7.95 [03:07:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.42, 7.60, 7.59 [03:09:59] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [03:11:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.07, 8.03, 7.68 [03:13:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.26, 7.37 [03:15:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.43, 7.19, 7.44 [03:15:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.05, 6.78, 7.18 [03:15:12] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [03:18:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.52, 7.68, 7.96 [03:19:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.38, 7.54, 7.41 [03:21:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.18, 7.76, 7.64 [03:22:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.60, 7.62, 7.98 [03:23:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.75, 8.00, 7.65 [03:23:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.93, 7.51, 7.57 [03:24:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.59, 7.02, 7.92 [03:24:45] PROBLEM - barengreza.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address barengreza.my.id and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:25:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.31, 7.74, 7.64 [03:25:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.71, 7.97, 7.66 [03:26:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.21, 8.04, 8.05 [03:26:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.62, 7.60, 8.02 [03:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.38, 8.64, 8.16 [03:28:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 7.33, 7.87 [03:29:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.87, 7.67, 7.65 [03:29:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.09, 7.66, 7.67 [03:29:51] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:30:58] RECOVERY - www.publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - www.publictestwiki.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [03:31:04] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.61 ms [03:33:05] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.25 ms [03:33:31] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.012 second response time [03:35:30] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.140 second response time [03:36:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.85, 7.90, 7.89 [03:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.56, 7.55, 7.96 [03:38:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.30, 7.66, 7.80 [03:38:37] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp24.miraheze.org - CNAME OK [03:41:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.86, 8.11, 7.71 [03:42:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.68, 7.99, 7.98 [03:43:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.79, 7.89, 7.67 [03:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 7.58, 7.83 [03:45:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.71, 8.00, 7.73 [03:46:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.20, 8.38, 8.08 [03:47:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.64, 7.53, 7.59 [03:49:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.55, 7.73, 7.47 [03:50:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.83, 7.57, 7.67 [03:51:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.85, 8.34, 7.87 [03:51:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.75, 7.67, 7.48 [03:53:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.20, 7.52, 7.63 [03:54:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.92, 6.94, 7.92 [03:54:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.50, 7.92, 7.98 [03:56:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.77, 7.20, 7.87 [03:56:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.31, 7.49, 7.59 [03:57:28] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.06 ms [03:58:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.76, 6.90, 7.67 [03:59:30] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.24 ms [04:00:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.81, 7.58, 7.82 [04:01:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.38, 6.12, 6.79 [04:04:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.91, 7.76, 7.84 [04:05:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.31, 3.25, 1.99 [04:06:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.90, 8.68, 8.15 [04:07:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.75, 6.73, 6.84 [04:07:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.31, 7.45, 7.28 [04:07:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.50, 2.52, 1.87 [04:09:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.26, 7.46, 7.31 [04:10:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.76, 7.96, 7.68 [04:11:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.62, 7.26, 7.00 [04:12:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.92, 7.83, 7.98 [04:13:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.42, 7.26, 7.03 [04:13:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.84, 8.03, 7.56 [04:14:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.27, 8.42, 8.18 [04:14:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.31, 7.50, 7.49 [04:18:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.48, 7.84, 7.65 [04:19:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.49, 7.42, 7.53 [04:21:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.29, 7.86, 7.68 [04:22:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.27, 7.58, 7.81 [04:23:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.06, 7.64, 7.62 [04:24:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.22, 7.59, 7.95 [04:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.86, 7.89, 7.89 [04:26:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 7.71, 7.84, 8.01 [04:27:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.42, 7.62, 7.60 [04:28:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.08, 7.56, 7.76 [04:29:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.70, 7.61, 7.59 [04:31:11] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:32:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.77, 7.85, 7.99 [04:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.04, 7.56, 7.69 [04:33:10] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.393 second response time [04:34:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.17, 7.29, 7.57 [04:36:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.17, 8.45, 8.14 [04:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.77, 8.56, 7.98 [04:37:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.36, 7.39, 7.14 [04:37:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.35, 6.86, 7.09 [04:39:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.98, 7.15, 7.08 [04:39:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.14, 7.09, 7.16 [04:40:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.42, 7.82, 7.81 [04:42:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.45, 7.78, 7.98 [04:44:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.45, 7.74, 7.57 [04:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.22, 8.15, 7.91 [04:46:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.05, 7.88, 7.96 [04:47:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.15, 7.26, 7.10 [04:48:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.51, 7.20, 7.64 [04:49:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.27, 7.41, 7.18 [04:50:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.95, 7.39, 7.74 [04:50:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.21, 7.64, 7.64 [04:57:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.72, 7.81, 7.45 [04:58:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.29, 7.54, 7.50 [04:59:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.62, 7.16, 7.24 [05:01:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.33, 6.12, 6.72 [05:02:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.42, 7.58, 7.55 [05:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.46, 7.95, 7.60 [05:06:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.92, 8.18, 7.74 [05:10:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.90, 7.87, 7.71 [05:12:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.33, 7.85, 7.73 [05:14:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.59, 7.37, 7.16 [05:15:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.34, 6.27, 6.69 [05:16:23] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.79 ms [05:17:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.36, 6.80, 6.70 [05:18:26] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [05:18:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.58, 9.02, 8.03 [05:19:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.13, 7.22, 6.97 [05:21:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.20, 7.22, 7.00 [05:22:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.15, 7.59, 7.52 [05:22:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.67, 7.82, 7.78 [05:23:46] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.10, 6.52, 6.61 [05:29:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.25, 6.53, 6.78 [05:34:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.21, 7.42, 7.37 [05:34:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.09, 7.93, 7.66 [05:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.18, 7.65, 7.59 [05:40:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.94, 8.00, 7.70 [05:42:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.76, 7.54, 7.58 [05:42:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.63, 7.75, 7.64 [05:42:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.03, 7.49, 7.95 [05:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.64, 7.94 [05:46:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.50, 7.93, 7.32 [05:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.34, 7.71, 7.61 [05:47:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.49, 6.83, 6.77 [05:48:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.39, 7.70, 7.31 [05:48:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.70, 7.38, 7.48 [05:48:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.77, 7.93 [05:49:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.32, 6.94 [05:50:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.66, 7.62, 7.53 [05:50:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 8.14, 8.06 [05:52:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.71, 7.33, 7.44 [05:52:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.73, 7.92 [05:53:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.69, 7.69, 7.18 [05:56:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.47, 6.09, 6.74 [05:57:51] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [05:58:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.34, 7.65, 7.34 [06:00:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.79, 7.06, 7.10 [06:00:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.14, 7.74, 7.72 [06:01:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.02, 7.02, 6.95 [06:02:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.47, 7.28, 7.18 [06:02:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.03, 7.19, 7.20 [06:02:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.43, 7.62, 7.20 [06:02:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.46, 7.62 [06:03:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.95, 6.88, 6.90 [06:04:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.04, 7.09, 7.06 [06:05:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.78, 6.62, 6.80 [06:06:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.47, 8.46, 7.63 [06:06:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.30, 8.20, 7.48 [06:06:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 8.73, 8.12 [06:08:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.37, 7.73, 7.41 [06:09:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.47, 7.10, 6.99 [06:10:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.93, 7.90, 7.92 [06:11:07] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:12:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.38, 7.75, 7.60 [06:14:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.61, 8.16, 8.00 [06:16:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.90, 7.91, 7.44 [06:17:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.04, 6.55, 6.79 [06:18:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.60, 7.61, 7.38 [06:18:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.68, 7.27, 7.20 [06:20:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.35, 7.76, 7.46 [06:21:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.21, 8.24, 7.40 [06:22:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.69, 7.42, 7.37 [06:23:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.91, 7.82, 7.34 [06:26:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.13, 6.94, 6.63 [06:28:17] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.63, 6.77, 6.60 [06:28:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.37, 7.62, 7.51 [06:28:38] PROBLEM - cp34 Puppet on cp34 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/varnishlog_logs] [06:29:51] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:30:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.53, 7.95 [06:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.89, 8.04, 7.71 [06:34:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 11.62, 8.27, 7.55 [06:34:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 13.38, 8.82, 7.69 [06:34:20] [02MirahezeMagic] 07Reception123 closed pull request 03#452: Use object-threads for swift deletion - 13https://github.com/miraheze/MirahezeMagic/pull/452 [06:34:21] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/81aff3441ea4...05df3cb9d291 [06:34:22] [02miraheze/MirahezeMagic] 07paladox 0305df3cb - Use object-threads for swift deletion (#452) [06:34:24] [02miraheze/MirahezeMagic] 07Reception123 deleted branch 03paladox-patch-7 [06:34:25] [02MirahezeMagic] 07Reception123 deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/MirahezeMagic [06:34:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.32, 9.54, 8.62 [06:35:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.42, 7.85, 7.38 [06:36:50] !log [reception@mwtask141] starting deploy of {'pull': 'world', 'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [06:36:51] !log [reception@mwtask141] finished deploy of {'pull': 'world', 'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 0s [06:36:52] !log [reception@mwtask141] starting deploy of {'pull': 'world', 'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [06:37:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.57, 7.65, 6.96 [06:38:33] !log [reception@mwtask141] finished deploy of {'pull': 'world', 'world': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 102s [06:39:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.53, 7.13, 6.85 [06:39:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.14, 7.82, 7.50 [06:39:07] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:40:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.77, 7.71, 7.56 [06:43:02] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.24, 6.73, 6.76 [06:43:24] miraheze/MirahezeMagic - Reception123 the build passed. [06:44:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.28, 8.06, 7.67 [06:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.43, 7.78, 7.99 [06:46:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.78, 7.92, 7.97 [06:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.72, 7.91, 8.02 [06:49:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.57, 7.04, 6.83 [06:49:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.37, 7.33, 7.29 [06:50:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.17, 7.66, 7.90 [06:51:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.15, 7.27, 6.93 [06:52:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.21, 8.41, 8.11 [06:52:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.50, 8.80, 8.29 [06:53:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.20, 7.09, 6.90 [06:53:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.96, 7.93, 7.57 [06:54:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.87, 7.70, 7.89 [06:55:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.92, 6.62, 6.75 [06:56:38] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:21] !log [@test131] starting deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] [06:57:22] !log [@test131] finished deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] - SUCCESS in 0s [06:58:01] !log RENAMED marioandluigiwiki to mariorpgwiki [06:58:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:00:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.19, 8.00, 7.95 [07:02:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.27, 7.85, 7.91 [07:04:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.42, 8.28, 8.05 [07:05:54] [02mw-config] 07Universal-Omega closed pull request 03#5428: Fix chameleon layout files for multiversion - 13https://github.com/miraheze/mw-config/pull/5428 [07:05:55] [02miraheze/mw-config] 07Universal-Omega deleted branch 03chameleon [07:05:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+5/-0/±2] 13https://github.com/miraheze/mw-config/compare/8cf30db9564d...3ac51f21ea6d [07:05:57] [02miraheze/mw-config] 07Universal-Omega 033ac51f2 - Fix chameleon layout files for multiversion (#5428) [07:05:58] [02mw-config] 07Universal-Omega deleted branch 03chameleon - 13https://github.com/miraheze/mw-config [07:06:41] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/removeSettings.php egChameleonLayoutFile (START) [07:06:55] miraheze/mw-config - Universal-Omega the build passed. [07:09:16] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:09:31] !log [@mwtask141] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 14s [07:11:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [07:12:33] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [07:13:28] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.166 second response time [07:13:46] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [07:14:32] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.144 second response time [07:15:44] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [07:18:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.29, 7.20, 6.89 [07:19:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.32, 7.50, 7.32 [07:20:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.40, 7.74, 7.96 [07:20:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.58, 7.09, 7.82 [07:20:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.18, 7.63, 7.07 [07:22:41] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.24, 7.36, 7.03 [07:24:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.02, 7.92, 7.98 [07:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.83, 8.27, 8.09 [07:26:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.90, 7.42, 7.79 [07:26:42] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [07:26:43] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 0s [07:28:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.82, 7.37, 7.92 [07:29:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.26, 7.12, 7.38 [07:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.33, 7.67, 7.96 [07:34:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.13, 7.81, 7.65 [07:34:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.00, 8.10, 8.06 [07:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.20, 7.71, 7.93 [07:39:07] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.41, 9.08, 7.03 [07:40:17] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.09, 6.38, 6.70 [07:41:05] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.22, 7.86, 6.82 [07:42:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.14, 7.55, 7.57 [07:42:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.51, 8.04, 7.89 [07:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.84, 7.91, 7.86 [07:47:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.23, 7.55, 7.32 [07:48:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.43, 7.67, 7.71 [07:50:04] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.64, 6.89, 6.72 [07:53:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.35, 7.79, 7.54 [07:54:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.08, 7.73, 7.95 [07:55:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.17, 8.37, 7.78 [07:56:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.62, 7.70, 7.60 [07:59:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.41, 7.62, 7.63 [07:59:47] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.07, 6.35, 6.67 [08:00:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.33, 7.65, 7.61 [08:04:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 7.19, 7.97 [08:06:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.47, 7.95, 7.74 [08:06:19] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.18, 6.29, 6.80 [08:06:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.57, 7.92, 8.13 [08:08:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.17, 7.82, 7.63 [08:10:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.68, 7.64, 7.70 [08:12:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.01, 7.57, 7.58 [08:12:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.75, 7.99 [08:16:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.03, 7.73, 7.16 [08:17:00] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:17:58] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/removeSettings.php egChameleonLayoutFile (END - exit=0) [08:18:01] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.21, 7.51, 7.15 [08:18:32] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [08:18:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 8.05, 8.00 [08:18:51] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 19s [08:18:56] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.238 second response time [08:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.14, 7.19, 7.18 [08:24:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.70, 7.95 [08:25:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.92, 6.41, 6.72 [08:25:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.30, 7.18, 7.04 [08:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.73, 7.01, 7.11 [08:26:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.58, 8.10, 8.06 [08:27:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.17, 7.26, 7.09 [08:28:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.11, 7.36, 7.23 [08:29:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.07, 6.89, 6.87 [08:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.65, 6.89, 7.09 [08:31:01] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 10.13, 7.17 [08:31:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.95, 6.42, 6.69 [08:31:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.44, 7.85, 7.36 [08:32:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.10, 7.44, 7.14 [08:33:01] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.00, 9.23, 7.23 [08:33:24] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.88, 7.47, 7.28 [08:34:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.11, 7.16, 7.07 [08:36:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.60, 7.97, 7.37 [08:38:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.04, 8.10, 7.47 [08:39:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.38, 7.52, 7.26 [08:40:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.79, 7.78, 7.45 [08:41:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.80, 7.29, 7.21 [08:44:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.39, 7.77, 7.61 [08:45:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.27, 7.55, 7.31 [08:45:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.78, 6.93, 6.83 [08:46:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.26, 7.92, 7.68 [08:46:55] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.99, 7.82, 7.45 [08:48:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.65, 7.86, 7.69 [08:48:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.27, 7.96, 7.56 [08:50:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.07, 7.64, 7.49 [08:53:43] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.00, 6.58, 6.76 [08:54:37] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.30, 7.60, 7.45 [08:58:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.39, 6.98, 7.15 [08:58:29] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.96, 7.81, 7.56 [08:59:20] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [09:00:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.50, 6.75, 7.05 [09:02:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.20, 7.89, 7.65 [09:04:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.05, 6.86, 7.30 [09:06:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.59, 7.46, 7.22 [09:08:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.87, 7.62, 7.47 [09:10:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.62, 6.72, 6.54 [09:10:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.88, 7.32, 7.38 [09:12:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.40, 7.77, 7.54 [09:12:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.74, 8.00, 7.48 [09:14:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.23, 7.63, 7.49 [09:16:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.28, 8.07, 7.71 [09:16:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.00, 7.46, 7.36 [09:18:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.46, 7.80, 7.66 [09:18:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.11, 7.91, 7.70 [09:19:19] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.04, 6.92, 6.49 [09:22:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.57, 7.73, 7.45 [09:23:14] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.04, 6.57, 6.45 [09:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.93, 7.44, 7.38 [09:25:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.46, 7.26, 7.04 [09:26:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.13, 7.38, 7.43 [09:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 7.69, 7.48 [09:27:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.44, 7.30, 7.08 [09:27:58] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp25.miraheze.org - CNAME OK [09:28:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.28, 7.32, 7.40 [09:29:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.60, 7.72, 7.26 [09:30:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.16, 7.50, 7.45 [09:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.74, 7.47, 7.46 [09:31:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.66, 7.03, 7.07 [09:32:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.41, 7.47, 7.44 [09:34:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.47, 7.35, 7.25 [09:38:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.57, 7.50, 7.42 [09:40:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 7.40, 7.38 [09:42:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.28, 7.88, 7.62 [09:43:07] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:43:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.47, 7.41, 7.13 [09:44:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.40, 7.49, 7.37 [09:45:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.18, 7.22, 7.09 [09:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.31, 7.75, 7.45 [09:48:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.31, 6.83, 6.60 [09:49:24] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.56, 8.24, 7.49 [09:49:28] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:50:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.77, 7.27, 7.44 [09:50:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.64, 7.83, 7.59 [09:50:47] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.37, 6.75, 6.60 [09:51:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 7.98, 7.49 [09:53:28] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:54:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.04, 7.66, 7.52 [09:56:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.74, 7.39, 7.45 [10:00:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.38, 7.67, 7.52 [10:00:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.97, 7.33, 7.31 [10:00:34] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.32 ms [10:02:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.07, 7.11, 7.20 [10:02:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.82, 7.04, 7.32 [10:02:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.63, 7.49, 7.38 [10:02:36] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.16 ms [10:04:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.90, 6.87, 7.09 [10:05:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.06, 6.30, 6.79 [10:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.15, 7.87, 7.42 [10:06:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.24, 8.77, 7.88 [10:09:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.00, 7.03, 7.00 [10:09:07] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [10:10:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.19, 7.88, 7.73 [10:12:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.93, 7.90, 7.71 [10:14:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.39, 7.71, 7.34 [10:18:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.93, 7.38, 7.35 [10:19:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.71, 6.70, 6.80 [10:21:05] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [10:23:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.75, 6.89, 6.88 [10:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.85, 7.77, 7.44 [10:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.20, 6.98, 7.19 [10:27:51] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [10:30:07] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.47, 6.35, 6.78 [10:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.49, 7.01, 7.12 [10:30:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.48, 7.07, 7.97 [10:31:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.71, 6.36, 6.72 [10:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.42, 6.75, 7.01 [10:34:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.20, 6.23, 6.73 [10:35:28] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:38:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.23, 7.48, 7.15 [10:40:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.66, 7.96, 7.85 [10:42:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.82, 6.89, 6.73 [10:42:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.84, 7.82 [10:43:55] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.70, 6.90, 6.84 [10:44:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.22, 7.32, 7.13 [10:44:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.56, 7.38, 6.93 [10:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.97, 8.16, 7.43 [10:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.84, 7.81 [10:46:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.32, 6.99, 7.03 [10:46:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.71, 7.60, 7.07 [10:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.66, 7.51, 7.28 [10:46:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 7.45, 7.67 [10:47:49] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.84, 6.68, 6.79 [10:49:48] !log DELETED and DROPPED krshprjwiki (reset) [10:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:51:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.96, 6.73, 6.81 [10:52:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.09, 7.41, 7.26 [10:53:05] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:53:40] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.75, 7.48, 7.05 [10:54:07] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.67, 6.31, 6.74 [10:54:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.21, 8.77, 7.99 [10:55:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.09, 6.92, 6.62 [10:55:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.16, 7.18, 6.99 [10:57:20] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.16, 6.73, 6.59 [11:00:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.16, 7.82, 7.70 [11:02:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.01, 7.35, 7.72 [11:04:19] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.08, 6.58, 6.78 [11:04:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.56, 8.00, 7.72 [11:04:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.41, 7.26, 6.92 [11:08:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.79, 7.24, 6.96 [11:08:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.66, 7.36, 7.49 [11:08:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.49, 7.86, 7.78 [11:10:07] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.18, 6.65, 6.79 [11:10:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.55, 7.67 [11:11:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.44, 6.62, 6.75 [11:16:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.06, 6.94, 6.85 [11:16:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.77, 8.44, 7.35 [11:20:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.06, 7.37, 7.19 [11:20:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 7.83, 7.60 [11:21:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.23, 7.21, 6.93 [11:24:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.90, 8.00, 7.71 [11:26:07] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.43, 6.65, 6.79 [11:26:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.30, 8.61, 7.96 [11:29:51] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:34:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.68, 7.62, 7.10 [11:36:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.94, 6.29, 6.67 [11:40:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.02, 7.76, 7.38 [11:40:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.88, 6.08, 6.73 [11:40:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.54, 7.85 [11:41:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.22, 6.38, 6.72 [11:42:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.84, 6.62, 6.69 [11:44:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.31, 7.19, 6.89 [11:48:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.58, 6.87, 6.88 [11:50:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.75, 6.54, 6.76 [11:51:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [11:53:28] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.212 second response time [11:55:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.27, 6.90, 6.83 [11:56:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.74, 8.10, 7.91 [12:15:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.20, 7.52, 7.09 [12:17:40] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.87, 7.57, 7.16 [12:19:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 6.69, 6.53 [12:20:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.20, 7.53, 7.44 [12:21:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.14, 7.52, 7.20 [12:22:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.49, 7.54, 7.46 [12:22:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.82, 7.57, 7.06 [12:23:28] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 7.20, 7.11 [12:25:04] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.66, 6.76, 6.66 [12:26:38] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.17, 7.53, 7.12 [12:28:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 7.12, 7.01 [12:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.29, 7.89, 7.28 [12:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.13, 7.61, 7.27 [12:34:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2d7f44d9cb76...f1a056f83dc7 [12:34:19] [02miraheze/puppet] 07paladox 03f1a056f - ssl: fix firewall [12:35:01] RECOVERY - puppet181 MirahezeRenewSsl on puppet181 is OK: TCP OK - 0.129 second response time on 2602:294:0:b12::101 port 5000 [12:36:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.72, 7.95, 7.44 [12:36:31] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f1a103ab1aae...6e108f59c8c2 [12:36:32] [02miraheze/dns] 07paladox 036e108f5 - Add acme challenges to mirabeta [12:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.68, 8.32, 7.50 [12:36:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 11.43, 8.24, 7.47 [12:37:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c0825fda90d1...a3eba2f7526a [12:37:40] [02miraheze/ssl] 07MirahezeSSLBot 03a3eba2f - Bot: Update SSL cert for mirabeta.org [12:37:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.91, 7.99, 7.15 [12:38:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.61, 7.65, 7.36 [12:39:03] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.84, 7.16, 6.55 [12:39:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.01, 7.56, 7.09 [12:41:07] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:42:46] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6e108f59c8c2...c47f7fdfeee8 [12:42:47] [02miraheze/dns] 07paladox 03c47f7fd - Add acme challenges to m.miraheze.org [12:43:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.47, 6.74, 6.51 [12:43:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a3eba2f7526a...c46dcb0c2a86 [12:43:56] [02miraheze/ssl] 07MirahezeSSLBot 03c46dcb0 - Bot: Update SSL cert for m.miraheze.org [12:44:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.35, 7.94, 7.94 [12:44:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.85, 7.60, 7.70 [12:45:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.27, 7.32, 7.09 [12:45:44] RECOVERY - mirabeta.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Mon 08 Apr 2024 11:37:28 GMT +0000. [12:45:58] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Mon 08 Apr 2024 11:43:44 GMT +0000. [12:46:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 7.51, 8.06, 7.99 [12:46:33] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [12:47:36] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 7.08, 7.03 [12:48:04] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:48:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.23, 7.82, 7.92 [12:48:32] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.159 second response time [12:49:59] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.740 second response time [12:55:24] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.42, 6.24, 6.71 [12:58:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.33, 6.96, 6.99 [12:59:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.17, 6.74, 6.81 [13:00:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.55, 7.03, 7.00 [13:02:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.37, 7.54, 7.36 [13:04:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.70, 6.06, 6.64 [13:04:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.54, 7.47, 7.35 [13:05:08] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.55, 6.61, 6.80 [13:06:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.24, 7.51, 7.36 [13:08:42] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.98, 6.86, 6.64 [13:10:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.60, 7.88, 7.59 [13:11:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.97, 7.58, 7.11 [13:13:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.88, 7.50, 7.12 [13:13:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.36, 6.78, 6.70 [13:14:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.32, 7.09, 7.08 [13:15:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.86, 7.82, 7.09 [13:16:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.84, 7.40, 7.20 [13:16:33] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.71, 7.51, 6.96 [13:18:31] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.09, 7.12, 6.87 [13:19:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.82, 7.69, 7.24 [13:21:05] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [13:21:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:22:24] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.73, 6.79, 6.79 [13:22:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.63, 7.41, 7.20 [13:22:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.25, 7.51, 7.23 [13:23:28] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.149 second response time [13:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.92, 7.87, 7.42 [13:24:33] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [13:24:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 7.41, 7.93 [13:24:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.10, 6.99, 7.07 [13:25:00] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [13:26:04] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [13:26:32] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.179 second response time [13:26:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.32, 7.69, 7.96 [13:26:55] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.48, 9.71, 7.08 [13:26:56] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.175 second response time [13:27:41] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [13:27:59] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.666 second response time [13:28:18] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.12, 11.63, 9.01 [13:28:19] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.32, 7.06, 6.89 [13:28:55] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.06, 8.40, 6.90 [13:30:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.78, 7.21, 7.07 [13:30:16] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.30, 6.50, 6.72 [13:30:35] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.38, 7.77, 7.37 [13:32:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.37, 6.95, 7.00 [13:32:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.66, 10.13, 9.01 [13:32:32] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.98, 7.45, 7.30 [13:33:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.01, 7.46, 7.24 [13:34:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.57, 7.57, 7.22 [13:34:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.73, 7.94 [13:35:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.72, 6.77, 7.01 [13:38:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.66, 7.93, 7.47 [13:38:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.57, 8.02, 7.99 [13:40:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.27, 7.72, 7.88 [13:44:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.11, 7.19, 7.04 [13:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 8.15, 8.02 [13:46:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.54, 7.63, 7.21 [13:46:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 7.91, 7.95 [13:48:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.81, 7.95, 7.43 [13:48:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.45, 7.42, 7.18 [13:50:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 8.43, 8.12 [13:51:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.67, 7.40, 7.10 [13:53:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.24, 7.29, 7.09 [13:55:40] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:56:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.76, 6.35, 6.76 [13:59:19] PROBLEM - wiki.nmc.games - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.40C7DB0B44150000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303 [14:00:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.03, 6.84, 6.88 [14:01:26] PROBLEM - wiki.nmc.games - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nmc.games reverse DNS resolves to any-in-2001-4860-4802-36--15.1e100.net [14:04:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.02, 7.50, 7.21 [14:06:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.71, 8.05, 7.39 [14:08:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.32, 7.31, 7.19 [14:08:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.19, 7.32, 7.83 [14:10:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.21, 7.51, 7.42 [14:10:11] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:11:03] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.25, 6.53, 6.78 [14:11:22] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:11:40] PROBLEM - puppet181 SSH on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:13] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:13:16] PROBLEM - puppet181 puppetserver on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:25] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.79 ms [14:13:34] RECOVERY - puppet181 SSH on puppet181 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [14:14:16] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:14:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.33, 7.27, 7.03 [14:14:21] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.72 ms [14:14:28] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:04] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:16] RECOVERY - puppet181 puppetserver on puppet181 is OK: TCP OK - 0.129 second response time on 2602:294:0:b12::101 port 8140 [14:15:31] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:38] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:40] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:16:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.85, 7.56, 7.18 [14:16:33] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 3 minutes ago with 24 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [14:17:39] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 44 failures. Last run 3 minutes ago with 44 failures. Failed resources (up to 3 shown): File[/etc/ssl/private],File[/etc/ImageMagick-6/policy.xml],File[/usr/local/bin/mediawiki-firejail-convert],File[/etc/firejail/mediawiki.local] [14:20:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 8.08, 7.83 [14:21:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.03, 6.86 [14:23:05] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:23:38] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.80 ms [14:24:09] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.16, 6.48, 6.22 [14:24:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.15, 8.10, 7.47 [14:25:41] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.73 ms [14:26:05] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.61, 6.44, 6.23 [14:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.95, 7.84, 7.44 [14:28:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.32, 7.10, 7.05 [14:28:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.75, 7.47, 7.17 [14:28:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.59, 8.02, 7.55 [14:28:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.83, 7.94 [14:30:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.75, 6.95, 7.00 [14:30:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.87, 7.01, 7.02 [14:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.78, 7.38, 7.36 [14:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.17, 7.62, 7.44 [14:35:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:35:42] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [14:36:01] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.72, 2.60, 1.99 [14:36:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.11, 7.87, 7.32 [14:36:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.54, 7.72 [14:37:31] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.143 second response time [14:37:39] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.354 second response time [14:37:58] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.02, 2.21 [14:38:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.06, 7.23, 7.02 [14:39:38] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [14:39:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.59, 3.26, 2.40 [14:40:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.72, 7.96, 7.49 [14:40:08] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [14:40:16] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:40:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 7.16, 7.03 [14:40:37] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [14:41:15] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:41:41] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:41:54] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.00, 2.63, 2.27 [14:42:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.45, 7.45, 7.58 [14:43:28] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:43:38] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [14:44:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [14:44:27] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.92, 7.60, 7.94 [14:45:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.00, 11.41, 9.09 [14:45:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.58, 2.73 [14:46:19] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:46:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.56, 7.70, 7.26 [14:46:32] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:48:16] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c47f7fdfeee8...7f5d60180289 [14:48:17] [02miraheze/dns] 07paladox 037f5d601 - Update wikitide.net [14:48:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.17, 7.39, 7.19 [14:48:30] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.182 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [14:49:07] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.83, 9.30, 8.77 [14:49:42] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.10, 3.02, 2.78 [14:50:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.65, 8.69, 8.20 [14:51:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.01, 7.54, 7.21 [14:51:35] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.37, 6.89, 6.48 [14:53:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.02, 7.36, 7.18 [14:54:33] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [14:54:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.08, 7.33, 7.77 [14:54:50] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [14:55:34] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [14:55:50] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 20.26, 12.14, 8.52 [14:56:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.27, 7.34, 7.19 [14:56:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.07, 8.36, 8.08 [14:56:47] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.187 second response time [14:56:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 17.71, 13.39, 10.43 [14:57:25] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.92, 6.64, 6.50 [14:57:28] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.139 second response time [14:57:30] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.39, 3.49, 2.97 [14:57:49] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.38, 11.00, 8.57 [14:57:55] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.04, 10.69, 8.72 [14:58:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.35, 7.25, 7.12 [14:58:41] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.158 second response time [14:58:41] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 10.86, 8.78 [14:58:58] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.57, 11.21, 10.01 [14:59:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.70, 7.56, 7.25 [14:59:28] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.09, 2.54, 2.68 [14:59:48] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.89, 9.40, 8.28 [14:59:50] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.99, 9.24, 8.44 [15:00:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.98, 7.00, 7.04 [15:00:40] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.13, 8.94, 8.33 [15:00:56] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.87, 9.27, 9.44 [15:01:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.11, 6.98, 7.07 [15:02:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.50, 7.72, 7.27 [15:04:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.77, 7.87, 7.63 [15:06:20] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.85, 6.29, 6.71 [15:10:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 14.21, 8.93, 7.99 [15:11:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.52, 7.53, 7.26 [15:12:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.69, 7.13, 6.69 [15:12:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.47, 7.00, 6.88 [15:13:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.61, 7.41, 7.24 [15:14:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.62, 8.00, 7.75 [15:14:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.00, 7.55, 7.09 [15:15:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.61, 7.72, 7.37 [15:16:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.72, 8.15, 7.83 [15:17:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.73, 7.65, 7.38 [15:17:28] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [15:17:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.11, 7.50, 6.97 [15:19:52] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.00, 7.15, 6.89 [15:20:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 7.76, 7.90 [15:21:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.67, 8.04, 7.59 [15:21:28] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [15:22:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.20, 7.64, 7.49 [15:24:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.51, 7.65, 7.80 [15:25:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.12, 7.43, 7.51 [15:26:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.17, 7.64, 7.76 [15:26:16] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3ac51f21ea6d...e682998737ed [15:26:18] [02miraheze/mw-config] 07AgentIsai 03e682998 - Update SMTP configuration [15:26:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.02, 8.07, 7.97 [15:27:07] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [15:27:19] miraheze/mw-config - AgentIsai the build passed. [15:27:21] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [15:27:22] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [15:28:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.17, 7.36, 7.64 [15:28:31] PROBLEM - mwtask141 HTTPS on mwtask141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:28:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.85, 7.83, 7.88 [15:28:47] !log [agent@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 100s [15:29:36] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.50, 6.69, 6.80 [15:30:30] PROBLEM - mwtask141 HTTPS on mwtask141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.221 second response time [15:30:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.87, 8.10, 7.97 [15:30:59] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e682998737ed...9adceb9d0997 [15:31:00] [02miraheze/mw-config] 07AgentIsai 039adceb9 - Add tls:// to mail config [15:31:02] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [15:31:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.24, 7.41, 7.41 [15:31:15] !log [agent@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 13s [15:32:02] miraheze/mw-config - AgentIsai the build passed. [15:32:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.19, 7.61, 7.82 [15:33:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.78, 7.06, 7.28 [15:34:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.34, 7.46, 7.56 [15:36:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.93, 7.78, 7.80 [15:37:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.58, 7.40 [15:38:20] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.12, 7.38, 7.22 [15:38:26] PROBLEM - www.pomologia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.pomologia.org' expires in 15 day(s) (Thu 25 Jan 2024 15:13:08 GMT +0000). [15:38:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.56, 7.01, 6.93 [15:38:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c46dcb0c2a86...f1c61fd53337 [15:38:43] [02miraheze/ssl] 07MirahezeSSLBot 03f1c61fd - Bot: Update SSL cert for www.pomologia.org [15:39:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.29, 7.41, 7.35 [15:40:19] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.79, 7.48, 7.27 [15:40:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.15, 7.38, 7.07 [15:40:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 7.82, 7.82 [15:41:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.39, 7.90, 7.53 [15:42:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.57, 7.47, 7.15 [15:42:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.28, 8.80, 8.22 [15:43:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.98, 7.62, 7.47 [15:44:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.09, 7.51, 7.19 [15:44:22] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 12.13, 8.81, 7.80 [15:45:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.71, 8.28, 7.72 [15:46:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1a056f83dc7...55dddf3c1f58 [15:46:14] [02miraheze/puppet] 07paladox 0355dddf3 - Add wikitide.net to resolv.conf.erb [15:49:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.30, 7.76, 7.63 [15:51:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.82, 8.18, 7.80 [15:52:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8cd21845951d [15:52:42] [02miraheze/puppet] 07paladox 038cd2184 - Install bast181 [15:52:44] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [15:52:45] [02puppet] 07paladox opened pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:53:04] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.98, 7.67, 7.66 [15:53:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/8cd21845951d...d14d4741e261 [15:53:11] [02miraheze/puppet] 07paladox 03d14d474 - Create bast181.yaml [15:53:12] [02puppet] 07paladox synchronize pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:53:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/d14d4741e261...ad094aff04ef [15:53:29] [02miraheze/puppet] 07paladox 03ad094af - Create bast161.yaml [15:53:31] [02puppet] 07paladox synchronize pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:53:42] [02puppet] 07paladox edited pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:53:44] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03paladox-patch-15 [+0/-0/±37] 13https://github.com/miraheze/puppet/compare/ad094aff04ef...ff3a6eb0b205 [15:53:45] [02miraheze/puppet] 07github-actions 03ff3a6eb - CI: lint puppet code to standards [15:53:48] [02puppet] 07github-actions[bot] synchronize pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:54:12] [02puppet] 07paladox closed pull request 03#3599: Install bast181 - 13https://github.com/miraheze/puppet/pull/3599 [15:54:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±38] 13https://github.com/miraheze/puppet/compare/55dddf3c1f58...789cfde76891 [15:54:15] [02miraheze/puppet] 07paladox 03789cfde - Install bast181 (#3599) [15:54:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [15:54:19] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [15:54:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.36, 7.48, 7.52 [15:55:22] PROBLEM - ping6 on mon181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.38 ms [15:55:27] PROBLEM - ping6 on swiftobject181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.32 ms [15:55:28] PROBLEM - ping6 on mw182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.68 ms [15:55:30] PROBLEM - ping6 on bast181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.95 ms [15:55:31] PROBLEM - ping6 on db182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.91 ms [15:55:49] PROBLEM - ping6 on mwtask181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.63 ms [15:55:52] PROBLEM - ping6 on mw181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.39 ms [15:55:52] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.63 ms [15:56:00] PROBLEM - ping6 on db181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.93 ms [15:57:21] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [15:57:22] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [15:58:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bcc01a4175c9 [15:58:25] [02miraheze/puppet] 07paladox 03bcc01a4 - Install db181/db182 [15:58:26] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [15:58:27] [02puppet] 07paladox opened pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [15:58:56] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.97, 7.59, 7.95 [15:59:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/bcc01a4175c9...bf3a7af51821 [15:59:09] [02miraheze/puppet] 07paladox 03bf3a7af - Create db181.yaml [15:59:10] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [15:59:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/bf3a7af51821...110240af9ff4 [15:59:36] [02miraheze/puppet] 07paladox 03110240a - Create db182.yaml [15:59:38] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:00:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.02, 7.60, 7.57 [16:00:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.24, 7.66, 7.59 [16:00:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/110240af9ff4...e052af1c4e75 [16:00:55] [02miraheze/puppet] 07paladox 03e052af1 - Create db151.yaml [16:00:56] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:01:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/e052af1c4e75...5c6435819fdf [16:01:03] [02miraheze/puppet] 07paladox 035c64358 - Create db161.yaml [16:01:05] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:01:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/5c6435819fdf...c3cc9720cc50 [16:01:15] [02miraheze/puppet] 07paladox 03c3cc972 - Create db171.yaml [16:01:17] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:01:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3cc9720cc50...4ee8bd997ab1 [16:01:54] [02miraheze/puppet] 07paladox 034ee8bd9 - Update packages.pp [16:01:57] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:02:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.68, 7.20, 7.43 [16:02:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4ee8bd997ab1...0c90a83699b5 [16:02:39] [02miraheze/puppet] 07paladox 030c90a83 - Update config.pp [16:02:42] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:02:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.10, 7.48, 7.53 [16:02:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0c90a83699b5...4ad8a7aa0132 [16:03:00] [02miraheze/puppet] 07paladox 034ad8a7a - Update db181.yaml [16:03:02] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:03:05] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [16:03:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4ad8a7aa0132...94a611b9fc09 [16:03:09] [02miraheze/puppet] 07paladox 0394a611b - Update db171.yaml [16:03:11] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:03:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/94a611b9fc09...a97fa9f03783 [16:03:14] [02miraheze/puppet] 07paladox 03a97fa9f - Update db161.yaml [16:03:15] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:03:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a97fa9f03783...c54d4b233a29 [16:03:22] [02miraheze/puppet] 07paladox 03c54d4b2 - Update db151.yaml [16:03:24] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:03:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c54d4b233a29...d3aea00e7ea3 [16:03:30] [02miraheze/puppet] 07paladox 03d3aea00 - Update db182.yaml [16:03:33] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:04:02] [02puppet] 07paladox edited pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:04:41] [02puppet] 07paladox synchronize pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:04:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d3aea00e7ea3...31d06e5cea6d [16:04:43] [02miraheze/puppet] 07paladox 0331d06e5 - Update config.pp [16:05:03] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.151 second response time [16:05:19] [02puppet] 07paladox closed pull request 03#3600: Install db181/db182 - 13https://github.com/miraheze/puppet/pull/3600 [16:05:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-0/±3] 13https://github.com/miraheze/puppet/compare/789cfde76891...0e9df5fad0df [16:05:21] [02miraheze/puppet] 07paladox 030e9df5f - Install db181/db182 (#3600) [16:05:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [16:05:24] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [16:06:02] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:06:08] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:08:01] RECOVERY - www.pomologia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pomologia.org' will expire on Mon 08 Apr 2024 14:38:30 GMT +0000. [16:08:02] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.227 second response time [16:08:07] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.958 second response time [16:08:18] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private],Apt_key[Add key: 177F4010FE56CA3336300305F1656F24C74CD1D8 from Apt::Source mariadb_apt] [16:09:36] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.75, 1.91, 1.51 [16:10:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.94, 7.93, 7.64 [16:11:30] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.51, 1.73, 1.49 [16:12:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.63, 7.64, 7.56 [16:12:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-0/±28] 13https://github.com/miraheze/puppet/compare/0e9df5fad0df...562d9dbd8c8d [16:12:59] [02miraheze/puppet] 07paladox 03562d9db - Update apt module to 9.2.0 [16:13:24] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.36, 1.58, 1.46 [16:13:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.87, 4.02, 2.60 [16:14:18] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.95, 7.69, 7.62 [16:15:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.59, 3.68, 2.67 [16:16:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.10, 7.58, 7.41 [16:16:23] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 130.15 ms [16:18:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.56, 7.32, 7.35 [16:18:27] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.21 ms [16:18:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/562d9dbd8c8d...24df903a6d60 [16:18:59] [02miraheze/puppet] 07paladox 0324df903 - mariadb: fix fetching apt repo on Debian 12 [16:19:45] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.45, 3.60, 2.83 [16:19:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.21, 7.43, 7.94 [16:20:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.10, 7.87, 7.81 [16:20:25] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.78, 7.87, 7.65 [16:21:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/24df903a6d60...34b66622bd04 [16:21:34] [02miraheze/puppet] 07paladox 0334b6662 - mariadb: only use new apt repo on Debian 12 [16:22:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.53, 7.83, 7.68 [16:23:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.56, 3.18, 2.91 [16:24:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34b66622bd04...4d7f9f443d93 [16:24:18] [02miraheze/puppet] 07paladox 034d7f9f4 - Revert "mariadb: only use new apt repo on Debian 12" [16:24:23] PROBLEM - db181 MariaDB on db181 is CRITICAL: Can't connect to MySQL server on 'db181.wikitide.net' (115) [16:24:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.77, 8.23, 7.82 [16:24:36] PROBLEM - db181 Backups SQL on db181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [16:24:39] PROBLEM - db181 Backups SQL mhglobal on db181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-mhglobal-backup-weekly.log [16:24:41] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [16:24:41] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [16:25:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 8.31, 8.14 [16:25:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.73, 7.87, 7.78 [16:26:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4d7f9f443d93...bbe400011541 [16:26:35] [02miraheze/puppet] 07paladox 03bbe4000 - mariadb: fix installing mariadb 10.11 [16:27:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bbe400011541...37674145abec [16:27:14] [02miraheze/puppet] 07paladox 033767414 - Fix [16:27:48] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.71, 7.54, 7.66 [16:27:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 7.54, 7.94 [16:28:17] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.16, 7.65, 7.36 [16:28:20] [02mw-config] 07redbluegreenhat opened pull request 03#5430: T11591: Add templateeditor and extendedconfirmed to available rights … - 13https://github.com/miraheze/mw-config/pull/5430 [16:28:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/37674145abec...01efbac2bedf [16:28:35] [02miraheze/puppet] 07paladox 0301efbac - Fix [16:29:16] miraheze/mw-config - redbluegreenhat the build passed. [16:29:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.55, 7.87 [16:30:08] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:30:15] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.65, 7.33, 7.28 [16:30:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.78, 7.74, 7.78 [16:33:29] PROBLEM - ping6 on swiftproxy181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.05 ms [16:33:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.91, 7.58, 7.76 [16:33:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.97, 8.09, 8.04 [16:35:28] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.07 ms [16:35:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:35:59] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:36:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.62, 7.86, 7.75 [16:36:45] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.98, 10.60, 8.62 [16:36:52] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:37:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.59, 7.76 [16:37:31] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 3.66, 2.87 [16:37:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/789b5ae3819d [16:37:54] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.72, 10.56, 8.73 [16:37:55] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.700 second response time [16:37:55] [02miraheze/puppet] 07paladox 03789b5ae - Install mw181/182 [16:37:57] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [16:37:59] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.210 second response time [16:38:00] [02puppet] 07paladox opened pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:38:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.52, 7.69, 7.70 [16:38:23] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.43, 11.25, 9.51 [16:38:34] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/789b5ae3819d...7b35482897a2 [16:38:36] [02miraheze/puppet] 07github-actions 037b35482 - CI: lint puppet code to standards [16:38:37] [02puppet] 07github-actions[bot] synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:38:40] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.02, 9.09, 8.29 [16:38:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/7b35482897a2...b9d3b7af1361 [16:38:46] [02miraheze/puppet] 07paladox 03b9d3b7a - Create mw181.yaml [16:38:48] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.169 second response time [16:38:48] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:38:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b9d3b7af1361...4eb13043cb00 [16:38:57] [02miraheze/puppet] 07paladox 034eb1304 - Create mw182.yaml [16:38:59] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:00] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.16, 11.00, 9.33 [16:39:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/4eb13043cb00...1a05c2b27b9d [16:39:06] [02miraheze/puppet] 07paladox 031a05c2b - Create mw151.yaml [16:39:08] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1a05c2b27b9d...752f1482d7ae [16:39:18] [02miraheze/puppet] 07paladox 03752f148 - Create mw152.yaml [16:39:19] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.89, 7.86 [16:39:29] PROBLEM - ping6 on swiftproxy181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.71 ms [16:39:29] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.12, 3.47, 2.90 [16:39:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/752f1482d7ae...78b051c86e75 [16:39:32] [02miraheze/puppet] 07paladox 0378b051c - Create mw161.yaml [16:39:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.39, 9.75, 8.54 [16:39:35] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/78b051c86e75...47197b9350ae [16:39:39] [02miraheze/puppet] 07paladox 0347197b9 - Create mw162.yaml [16:39:40] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/47197b9350ae...8349ffb77eb1 [16:39:49] [02miraheze/puppet] 07paladox 038349ffb - Create mw171.yaml [16:39:50] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.79, 9.31, 8.49 [16:39:51] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:39:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/8349ffb77eb1...fc834554d2a4 [16:39:58] [02miraheze/puppet] 07paladox 03fc83455 - Create mw172.yaml [16:40:00] [02puppet] 07paladox synchronize pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:40:12] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.82, 7.87, 7.75 [16:40:23] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.93, 11.16, 9.69 [16:40:42] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.51, 1.85, 1.58 [16:40:50] [02puppet] 07paladox closed pull request 03#3601: Install mw181/182 - 13https://github.com/miraheze/puppet/pull/3601 [16:40:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+8/-0/±2] 13https://github.com/miraheze/puppet/compare/01efbac2bedf...0143433e2696 [16:40:53] [02miraheze/puppet] 07paladox 030143433 - Install mw181/182 (#3601) [16:40:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [16:40:57] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.47, 11.22, 9.65 [16:40:57] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [16:41:27] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.02, 2.64, 2.67 [16:41:28] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.29 ms [16:41:33] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 5.84, 8.73, 8.35 [16:42:23] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.64, 9.46, 9.24 [16:42:41] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.78, 1.51, 1.49 [16:42:54] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.82, 9.95, 9.36 [16:43:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0143433e2696...8011896f38a9 [16:43:11] [02miraheze/puppet] 07paladox 038011896 - Temporarily don't install cadvisor [16:44:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8011896f38a9...6365b9119c0a [16:44:49] [02miraheze/puppet] 07paladox 036365b91 - graylog: fix firewall [16:45:18] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 92 failures. Last run 1 minute ago with 92 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72],Package[cgroup-tools],Package[imagemagick],Package[webp] [16:45:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.08, 8.67, 7.87 [16:45:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.05, 7.14, 7.97 [16:46:08] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.12, 7.38, 7.61 [16:46:47] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.53, 10.48, 9.75 [16:47:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.55, 7.40, 7.82 [16:47:27] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 3.35, 2.57 [16:47:58] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 42 failures. Last run 2 minutes ago with 42 failures. Failed resources (up to 3 shown): Package[mcrouter],Package[cgroup-tools],Package[imagemagick],Package[webp] [16:48:44] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.68, 9.99, 9.67 [16:49:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.92, 7.98, 7.79 [16:49:24] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.61, 3.09, 2.58 [16:49:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.47, 7.57, 7.99 [16:51:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 7.95, 7.93 [16:52:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/f1c61fd53337...8290542cda0c [16:52:24] [02miraheze/ssl] 07MirahezeSSLBot 038290542 - Bot: Add SSL cert for en.clockup.wiki [16:52:55] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8290542cda0c...cb65856ce0bb [16:52:58] [02miraheze/ssl] 07paladox 03cb65856 - Remove duplicate en.clockup.wiki [16:53:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.58, 7.27, 7.67 [16:53:15] miraheze/ssl - MirahezeSSLBot the build has errored. [16:53:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.28, 6.28, 6.74 [16:54:15] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:54:32] PROBLEM - db182 MariaDB on db182 is CRITICAL: Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server [16:54:42] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw182.wikitide.net port 443 after 129 ms: Couldn't connect to server [16:54:50] PROBLEM - db182 Backups SQL icingaweb2 on db182 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-icingaweb2-backup-fortnightly.log [16:54:57] PROBLEM - db182 Backups SQL roundcubemail on db182 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-roundcubemail-backup-fortnightly.log [16:54:58] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: connect to address 2602:294:0:b12::106 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:55:02] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw181.wikitide.net port 443 after 129 ms: Couldn't connect to server [16:55:03] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.010 second response time [16:55:07] PROBLEM - db182 Backups SQL reports on db182 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-reports-backup-weekly.log [16:55:09] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: connect to address 2602:294:0:b12::105 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:55:10] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thro [16:55:10] /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} th [16:55:10] in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [16:55:25] PROBLEM - db182 Backups SQL icinga on db182 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-icinga-backup-fortnightly.log [16:55:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.46, 7.65, 7.86 [16:56:10] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.198 second response time [16:56:35] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.94, 13.40, 10.31 [16:56:58] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.272 second response time [16:57:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.74, 7.74 [16:57:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.82, 7.79, 7.89 [16:58:33] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.11, 10.38, 9.58 [16:58:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/cb65856ce0bb...fbe8b10363a9 [16:58:58] [02miraheze/ssl] 07MirahezeSSLBot 03fbe8b10 - Bot: Add SSL cert for en.clockup.wiki [16:58:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.37, 7.60 [16:59:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fbe8b10363a9...3866641b4d2f [16:59:15] [02miraheze/ssl] 07paladox 033866641 - Remove duplicate en.clockup.wiki [16:59:17] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.78, 11.94, 10.43 [16:59:43] miraheze/ssl - MirahezeSSLBot the build has errored. [17:00:32] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.20, 8.38, 8.95 [17:00:51] PROBLEM - ping6 on mwtask181 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 129.04 ms [17:00:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 7.59, 7.64 [17:01:04] PROBLEM - ping6 on mw181 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:01:09] PROBLEM - ping6 on mw182 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 129.67 ms [17:01:15] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.97, 10.88, 10.24 [17:02:21] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 1.80, 2.01, 1.66 [17:02:51] PROBLEM - ping6 on mwtask181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.64 ms [17:03:05] PROBLEM - ping6 on mw181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.13 ms [17:03:07] PROBLEM - ping6 on mw182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.69 ms [17:03:10] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [17:03:16] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.75, 7.72, 7.56 [17:03:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.46, 7.62, 7.49 [17:03:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.53, 8.38, 8.10 [17:04:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 7.50, 7.32 [17:04:17] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.04, 1.65, 1.57 [17:04:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.68, 7.70 [17:05:03] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.83, 3.08, 2.69 [17:06:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.40, 7.26, 7.25 [17:06:36] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [17:06:54] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [17:07:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.47, 3.12, 2.78 [17:07:12] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.17, 9.47, 9.89 [17:07:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.49, 7.97, 7.74 [17:07:26] PROBLEM - ping6 on swiftproxy181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.09 ms [17:07:51] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [17:07:57] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [17:08:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.23, 7.49, 7.32 [17:08:16] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private] [17:08:28] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private] [17:08:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.79, 7.90, 7.74 [17:08:52] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.33, 10.25, 9.14 [17:09:26] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.40 ms [17:09:28] PROBLEM - matomo121 Puppet on matomo121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [17:09:44] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private] [17:09:46] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.313 second response time [17:09:53] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.729 second response time [17:10:06] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.06, 7.24, 7.24 [17:10:23] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private] [17:10:42] PROBLEM - mw181 HTTPS on mw181 is WARNING: HTTP WARNING: HTTP/2 404 - 146 bytes in 0.408 second response time [17:10:47] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.12, 8.63, 8.68 [17:10:52] PROBLEM - mw181 MediaWiki Rendering on mw181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [17:11:20] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [17:11:25] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private] [17:11:40] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/ssl/private/miraheze.wiki.key] [17:11:45] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key] [17:11:49] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/wildcard.miraheze.org-2020-2.key],File[/etc/opensearch/ssl/opensearch-node-key.pem],File[/etc/opensearch/ssl/opensearch-admin-key.pem] [17:12:02] PROBLEM - mw182 MediaWiki Rendering on mw182 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [17:12:18] PROBLEM - mw182 HTTPS on mw182 is WARNING: HTTP WARNING: HTTP/2 404 - 146 bytes in 0.395 second response time [17:12:47] [02mw-config] 07Universal-Omega closed pull request 03#5430: T11591: Add templateeditor and extendedconfirmed to available rights … - 13https://github.com/miraheze/mw-config/pull/5430 [17:12:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9adceb9d0997...bce1879e360f [17:12:51] [02miraheze/mw-config] 07redbluegreenhat 03bce1879 - T11591: Add templateeditor and extendedconfirmed to available rights … (#5430) [17:13:52] miraheze/mw-config - Universal-Omega the build passed. [17:14:20] [02miraheze/MirahezeDebug] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/677f780ee16e...e0ce1a1600b9 [17:14:21] [02miraheze/MirahezeDebug] 07MacFan4000 03e0ce1a1 - add mw181/mw182 [17:15:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.99, 7.98, 8.00 [17:16:40] PROBLEM - ping6 on db182 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.10 ms [17:17:46] PROBLEM - ping6 on mon181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.08 ms [17:18:18] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.44, 6.94, 6.71 [17:18:43] PROBLEM - ping6 on db182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.23 ms [17:18:55] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:03] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.04, 7.72, 7.66 [17:19:49] PROBLEM - ping6 on mon181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.96 ms [17:20:05] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.98, 6.29, 6.79 [17:20:54] PROBLEM - mw181 MediaWiki Rendering on mw181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [17:21:23] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [17:21:51] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [17:22:16] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.57, 6.80, 6.71 [17:22:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.54, 7.84 [17:23:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.70, 7.79, 7.70 [17:23:18] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.164 second response time [17:23:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.94, 7.31, 7.89 [17:23:23] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.45, 9.80, 8.41 [17:23:47] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.51, 10.73, 8.78 [17:23:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.216 second response time [17:23:53] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw133 [17:24:21] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.39, 10.45, 8.94 [17:24:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6365b9119c0a...901030cee56a [17:24:26] [02miraheze/puppet] 07paladox 03901030c - Force use of cgroups v1 [17:25:19] PROBLEM - ping6 on mw181 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 135.40 ms [17:25:20] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.37, 7.55, 7.89 [17:25:23] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.88, 8.03, 7.94 [17:25:49] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [17:26:17] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.45, 9.16, 8.66 [17:26:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/901030cee56a...07cd89c244eb [17:26:50] [02miraheze/puppet] 07paladox 0307cd89c - mwdeploy: add mw181/mw182 [17:27:03] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.32, 7.05, 6.95 [17:27:17] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [17:27:18] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [17:27:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.09, 7.24, 7.73 [17:27:20] PROBLEM - ping6 on mw181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.27 ms [17:27:45] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.77, 9.16, 8.63 [17:28:01] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141] [17:28:01] miraheze/puppet - paladox the build passed. [17:29:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.49, 7.55, 7.14 [17:29:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.37, 7.71, 7.82 [17:29:34] !log [@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.miraheze.org [17:30:31] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.miraheze.org [17:30:36] !log [paladox@mwtask141] finished deploy of {'config': True, 'force': True, 'versions': ['1.40'], 'servers': ['mw181']} to [mw181] - FAIL: [3072] in 1s [17:30:45] !log [paladox@mwtask141] finished deploy of {'config': True, 'force': True, 'versions': ['1.40'], 'servers': ['mw182']} to [mw182] - FAIL: [3072] in 2s [17:31:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.41, 6.99, 6.98 [17:31:16] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.11, 7.31, 7.66 [17:31:36] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.miraheze.org [17:31:43] !log [paladox@mwtask141] finished deploy of {'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40'], 'servers': ['mw181', 'mw182']} to [mw181, mw182] - FAIL: [3072, 65280] in 3s [17:32:01] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [17:32:44] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:33:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.57, 7.75, 7.77 [17:33:17] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [17:33:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.25, 7.79, 7.65 [17:34:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.51, 7.46 [17:34:34] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:34:58] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:35:14] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.29, 7.52, 7.69 [17:35:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.81, 7.83, 7.69 [17:35:56] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:36:12] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-puppet141 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/8f121cdcbfdb [17:36:13] [02miraheze/dns] 07Universal-Omega 038f121cd - Remove puppet141 from DNS [17:36:14] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [17:36:14] [02dns] 07Universal-Omega created branch 03remove-puppet141 - 13https://github.com/miraheze/dns [17:36:16] [02dns] 07Universal-Omega opened pull request 03#455: Remove puppet141 from DNS - 13https://github.com/miraheze/dns/pull/455 [17:36:39] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:36:40] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:36:57] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-puppet141 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8f121cdcbfdb...63d6b8e9a3ba [17:36:58] [02miraheze/dns] 07Universal-Omega 0363d6b8e - Remove [17:37:01] [02dns] 07Universal-Omega synchronize pull request 03#455: Remove puppet141 from DNS - 13https://github.com/miraheze/dns/pull/455 [17:37:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.73, 8.10, 7.87 [17:37:26] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 11.92, 9.28, 8.23 [17:37:54] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:38:20] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 3.44, 2.29, 1.75 [17:38:59] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.30, 7.28, 7.10 [17:39:00] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.71, 7.80, 7.77 [17:39:21] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:25] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:45] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [17:40:17] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.42, 1.94, 1.68 [17:40:20] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:40:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 8.00, 7.78, 7.62 [17:40:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.42, 7.33, 7.14 [17:41:06] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:41:06] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.89, 6.78, 6.68 [17:41:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.05, 7.94, 7.83 [17:42:13] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.90, 1.56, 1.57 [17:42:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.59, 8.25, 7.82 [17:42:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.95, 7.41, 7.34 [17:44:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.84, 7.74 [17:44:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.84, 7.58, 7.41 [17:45:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.42, 6.38, 6.56 [17:45:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.18, 7.94, 7.90 [17:45:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.35, 7.99, 7.99 [17:47:33] [02miraheze/puppet] 07Universal-Omega created branch 03puppet181 13https://github.com/miraheze/puppet/commit/07cd89c244ebbdad00c5710ea433fd0f9d6786b4 [17:47:34] [02puppet] 07Universal-Omega created branch 03puppet181 - 13https://github.com/miraheze/puppet [17:48:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet181 [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/07cd89c244eb...b3dbc8280f63 [17:48:02] [02miraheze/puppet] 07Universal-Omega 03b3dbc82 - Use puppet181 by default [17:48:17] [02puppet] 07Universal-Omega opened pull request 03#3602: Use puppet181 by default - 13https://github.com/miraheze/puppet/pull/3602 [17:48:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.29, 7.73, 7.69 [17:48:54] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03puppet181 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3dbc8280f63...fd4e3ab1a6c4 [17:48:57] [02miraheze/puppet] 07github-actions 03fd4e3ab - CI: lint puppet code to standards [17:49:00] [02puppet] 07github-actions[bot] synchronize pull request 03#3602: Use puppet181 by default - 13https://github.com/miraheze/puppet/pull/3602 [17:50:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.69, 7.69 [17:50:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.24, 7.49, 7.16 [17:51:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/42cf1471c1fb [17:51:17] [02miraheze/puppet] 07Universal-Omega 0342cf147 - Use puppet8 [17:51:18] [02puppet] 07Universal-Omega created branch 03puppet8 - 13https://github.com/miraheze/puppet [17:51:20] [02puppet] 07Universal-Omega opened pull request 03#3603: Use puppet8 - 13https://github.com/miraheze/puppet/pull/3603 [17:51:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.10, 6.86, 6.73 [17:52:12] [02puppet] 07Universal-Omega closed pull request 03#3602: Use puppet181 by default - 13https://github.com/miraheze/puppet/pull/3602 [17:52:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/07cd89c244eb...c8d4c03a5ffe [17:52:16] [02miraheze/puppet] 07Universal-Omega 03c8d4c03 - Use puppet181 by default (#3602) [17:52:19] [02miraheze/puppet] 07Universal-Omega deleted branch 03puppet181 [17:52:22] [02puppet] 07Universal-Omega deleted branch 03puppet181 - 13https://github.com/miraheze/puppet [17:52:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.40, 7.77, 7.71 [17:52:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.60, 7.43, 7.17 [17:53:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.59, 7.71, 7.76 [17:54:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.04, 7.64, 7.27 [17:54:56] [02puppet] 07Universal-Omega edited pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [17:55:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.23, 8.18, 7.91 [17:55:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.68, 7.47, 6.97 [17:56:52] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.05, 7.48, 7.26 [17:57:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.56, 7.74, 7.77 [17:58:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.21, 7.23, 6.94 [17:58:56] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.01, 10.31, 9.33 [17:59:44] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:00:00] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.96, 6.76, 6.80 [18:00:19] PROBLEM - ping6 on puppet181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.61 ms [18:00:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.94, 7.91 [18:00:39] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.33, 10.85, 9.27 [18:01:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.14, 7.74, 7.74 [18:02:13] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.28, 9.61, 8.61 [18:02:19] PROBLEM - ping6 on puppet181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.49 ms [18:02:37] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.58, 10.32, 9.26 [18:02:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.19, 7.94, 7.96 [18:04:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.54, 6.82, 6.84 [18:04:12] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.13, 8.77, 8.41 [18:04:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.96, 7.91 [18:04:55] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.04, 9.86, 9.54 [18:04:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.11, 7.83, 7.90 [18:05:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.79, 7.95, 7.83 [18:06:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.87, 7.89 [18:06:31] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.71, 11.28, 9.86 [18:06:32] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.40, 4.20, 2.45 [18:07:32] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 7.94, 8.20, 7.96 [18:08:29] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.89, 11.09, 9.98 [18:08:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.47, 7.57, 7.76 [18:09:33] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.53, 7.93, 7.89 [18:10:26] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.94, 9.66, 9.59 [18:10:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.64, 3.47, 2.61 [18:10:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.06, 7.32, 7.13 [18:10:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.98, 7.41, 7.11 [18:10:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.48, 7.93, 7.87 [18:11:27] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:11:32] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:11:50] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:11:51] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:12:12] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:12:23] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.69, 2.47, 2.34 [18:12:26] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:12:41] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d],File[/home/agent],File[/home/owen],File[/home/paladox] [18:12:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.99, 7.35, 7.18 [18:12:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.36, 7.30, 7.10 [18:12:57] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:12:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.92, 7.73, 7.83 [18:13:05] PROBLEM - ping6 on mw182 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.62 ms [18:13:08] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:14:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.33, 7.72, 7.27 [18:15:04] PROBLEM - ping6 on mw182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.28 ms [18:15:14] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:15:19] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:15:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.10, 7.73, 7.80 [18:16:48] PROBLEM - swiftproxy181 Puppet on swiftproxy181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:16:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.94, 7.42, 7.22 [18:16:56] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:17:20] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:17:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.23, 7.70, 7.79 [18:18:10] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:18:21] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:20:04] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:20:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.60, 7.56 [18:20:29] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:20:30] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:20:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.33, 7.46, 7.25 [18:20:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.13, 7.86, 7.83 [18:21:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.08, 7.64, 7.72 [18:22:07] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:22:17] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:22:37] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:22:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.78, 8.49, 7.76 [18:22:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.39, 7.50, 7.29 [18:23:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.83, 7.39, 7.62 [18:23:44] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:24:16] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:24:26] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:24:44] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:24:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.62, 7.86, 7.62 [18:24:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.47, 7.74, 7.86 [18:25:43] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:26:01] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:26:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.93, 7.56, 7.31 [18:26:55] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:26:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.17, 7.98, 7.94 [18:27:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.32, 7.76, 7.72 [18:27:46] PROBLEM - cp34 Puppet on cp34 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:27:54] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:05] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:11] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:32] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.21, 7.95, 7.70 [18:28:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.83, 7.27, 7.24 [18:28:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.45, 7.78, 7.87 [18:29:25] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:29:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.89, 7.92, 7.79 [18:29:39] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:30:35] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:30:53] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:31:20] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:32:26] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:32:38] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:32:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.67, 7.44, 7.56 [18:32:53] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:32:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.29, 8.21, 8.01 [18:33:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/42cf1471c1fb...0b59e226c3a0 [18:33:20] [02miraheze/puppet] 07Universal-Omega 030b59e22 - Fix for ruby 3.2.0+ [18:33:20] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:33:21] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [18:33:35] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.14, 8.02, 7.85 [18:33:45] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:14] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0b59e226c3a0...6eb22df88989 [18:34:44] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:44] [02miraheze/puppet] 07Universal-Omega 036eb22df - Fix for ruby 3.2.0+ [18:34:45] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [18:34:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.70, 7.62, 7.82 [18:35:24] [02miraheze/puppet] 07Universal-Omega pushed 032 commits to 03puppet8 [+0/-0/±8] 13https://github.com/miraheze/puppet/compare/6eb22df88989...3aaad96d6854 [18:35:27] [02miraheze/puppet] 07Universal-Omega 033aaad96 - Merge branch 'master' into puppet8 [18:35:28] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [18:36:01] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.58, 6.58, 6.74 [18:36:33] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:36:57] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:37:05] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:37:59] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:38:13] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:38:26] PROBLEM - matomo121 Puppet on matomo121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:38:58] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:39:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3aaad96d6854...e0e823ec8d0d [18:39:20] [02miraheze/puppet] 07Universal-Omega 03e0e823e - Update puppet conf [18:39:23] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [18:40:06] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:40:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.42, 7.92 [18:40:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e0e823ec8d0d...5b902c2b8b95 [18:40:42] [02miraheze/puppet] 07Universal-Omega 035b902c2 - Update [18:40:44] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [18:41:25] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:41:29] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:41:45] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:41:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.95, 6.74, 6.72 [18:42:04] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:42:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 7.57, 7.90 [18:42:39] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.46 ms [18:43:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.07, 7.85, 7.96 [18:43:59] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.96, 6.55, 6.66 [18:44:38] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.93 ms [18:44:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.07, 7.54, 7.23 [18:44:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.12, 7.75, 7.74 [18:46:21] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:46:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.30, 7.73 [18:46:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.13, 7.64, 7.71 [18:47:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.09, 7.89, 7.94 [18:47:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.97, 7.14, 6.86 [18:48:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.11, 7.54, 7.38 [18:48:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.41, 7.29, 7.24 [18:48:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.80, 8.09, 7.86 [18:49:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.61, 7.80, 7.90 [18:50:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 7.50, 7.67 [18:50:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.18, 7.89, 7.48 [18:51:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.78, 8.03, 7.97 [18:52:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.82, 7.66, 7.71 [18:52:36] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.47, 7.15, 7.32 [18:52:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.21, 7.20, 7.27 [18:52:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.65, 7.52, 7.71 [18:53:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.58, 7.61, 7.81 [18:54:07] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [18:57:28] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:00:04] PROBLEM - ping6 on swiftobject181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.49 ms [19:00:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.19, 7.73, 7.63 [19:02:04] PROBLEM - ping6 on swiftobject181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 131.28 ms [19:02:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.75, 7.66 [19:02:44] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:01] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.01, 6.53, 6.72 [19:04:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.24, 7.81, 7.66 [19:05:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.75, 8.43, 7.99 [19:06:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.85, 7.70 [19:06:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.62, 7.20, 7.09 [19:06:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.24, 7.93, 7.75 [19:07:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.32, 6.81, 6.83 [19:08:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.77, 7.70 [19:08:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.75, 7.10, 7.08 [19:10:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.45, 7.53, 7.28 [19:11:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.16, 7.65, 7.86 [19:12:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 7.88, 7.74 [19:12:28] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.25, 7.54, 7.33 [19:13:59] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.86, 6.69, 6.80 [19:14:23] RECOVERY - swiftproxy181 Puppet on swiftproxy181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:14:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.12, 7.67, 7.31 [19:14:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.36, 7.61, 7.60 [19:15:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.32, 8.16, 8.02 [19:16:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.90, 7.80 [19:16:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.23, 7.58, 7.33 [19:16:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.86, 7.56, 7.60 [19:19:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.74, 7.62, 7.85 [19:20:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.14, 7.96, 7.52 [19:20:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.37, 7.86, 7.68 [19:21:13] PROBLEM - ping6 on swiftproxy181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.73 ms [19:21:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.61, 7.82, 7.88 [19:22:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.97, 7.80 [19:22:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.57, 7.49, 7.40 [19:22:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.29, 7.58, 7.59 [19:23:13] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.62 ms [19:24:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.62, 7.37, 7.60 [19:24:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 7.72, 7.64 [19:25:16] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:26:05] PROBLEM - ping6 on mw181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.58 ms [19:27:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.70, 7.87, 7.89 [19:27:59] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.68, 6.86, 6.85 [19:28:06] PROBLEM - ping6 on mw181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.92 ms [19:28:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.42, 7.50, 7.31 [19:28:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.52, 7.86, 7.75 [19:28:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.24, 7.54, 7.60 [19:30:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.40, 7.42, 7.30 [19:32:02] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchise.franchising.org.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:32:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.52, 7.67 [19:32:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.41, 7.40, 7.38 [19:32:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.20, 8.03, 7.75 [19:33:01] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [19:33:56] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.81, 8.20, 8.00 [19:33:59] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.37, 6.66, 6.78 [19:34:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.48, 6.06, 6.79 [19:34:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 7.17, 7.30 [19:34:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.11, 7.80, 7.71 [19:35:00] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.286 second response time [19:36:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 7.86, 8.09, 7.85 [19:37:46] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is UNKNOWN: NRPE: Unable to read output [19:38:08] PROBLEM - swiftobject111 APT on swiftobject111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:38:16] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bce1879e360f...bad85762ae01 [19:38:17] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 128.85 ms [19:38:18] [02miraheze/mw-config] 07AgentIsai 03bad8576 - Modify parameters for SMTP [19:38:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.38, 8.18, 7.85 [19:38:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.22, 7.84, 7.79 [19:39:05] PROBLEM - ping6 on mw182 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 130.51 ms [19:39:17] miraheze/mw-config - AgentIsai the build passed. [19:39:25] PROBLEM - cp24 Puppet on cp24 is UNKNOWN: NRPE: Unable to read output [19:39:27] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [19:39:45] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:40:00] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:40:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.45, 6.86, 6.85 [19:40:16] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.51 ms [19:40:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.23, 7.09, 7.03 [19:40:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.85, 7.97, 7.81 [19:40:26] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:40:28] PROBLEM - swiftobject111 APT on swiftobject111 is CRITICAL: APT CRITICAL: 86 packages available for upgrade (33 critical updates). [19:40:31] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:40:38] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:40:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.41, 7.69, 7.49 [19:40:59] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.10, 9.75, 8.65 [19:41:00] PROBLEM - swiftobject111 Puppet on swiftobject111 is UNKNOWN: NRPE: Unable to read output [19:41:04] PROBLEM - ping6 on mw182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.78 ms [19:41:32] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:41:32] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:42:11] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:42:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03puppet8 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5b902c2b8b95...10980afdc2e7 [19:42:39] [02miraheze/puppet] 07Universal-Omega 0310980af - Fix hiera [19:42:41] [02puppet] 07Universal-Omega synchronize pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [19:42:52] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.84, 7.58, 7.46 [19:42:58] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.96, 9.33, 8.65 [19:43:26] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c8d4c03a5ffe...e796c278b60c [19:43:27] [02miraheze/puppet] 07AgentIsai 03e796c27 - Change to smtp-relay [19:44:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.12, 7.43, 7.15 [19:44:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.56, 8.53, 8.05 [19:44:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.92, 8.74, 7.90 [19:45:50] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:46:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.56, 7.41, 7.18 [19:48:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.01, 7.51, 7.15 [19:48:10] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:48:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.02, 7.49, 7.23 [19:48:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.87, 7.97, 7.93 [19:49:34] PROBLEM - db121 APT on db121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [19:49:47] [02puppet] 07Universal-Omega closed pull request 03#3603: Upgrade to puppet 8 - 13https://github.com/miraheze/puppet/pull/3603 [19:49:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/e796c278b60c...24cc5409aef8 [19:49:53] [02miraheze/puppet] 07Universal-Omega 0324cc540 - Upgrade to puppet 8 (#3603) [19:49:56] [02miraheze/puppet] 07Universal-Omega deleted branch 03puppet8 [19:49:57] [02puppet] 07Universal-Omega deleted branch 03puppet8 - 13https://github.com/miraheze/puppet [19:50:05] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:50:06] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:50:07] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:50:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 8.14, 7.98 [19:50:32] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:50:48] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:50:59] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.69, 8.29, 7.92 [19:51:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.28, 1.75, 1.40 [19:51:35] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 92 packages available for upgrade (32 critical updates). [19:51:51] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:51:52] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:52:37] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:52:44] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:53:14] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.02, 1.51, 1.36 [19:53:29] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [19:53:41] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:53:52] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [19:54:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.15, 7.70, 7.42 [19:54:57] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:55:16] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:55:51] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [19:55:55] PROBLEM - mw181 Puppet on mw181 is UNKNOWN: NRPE: Unable to read output [19:56:11] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:56:18] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.54, 7.95, 7.54 [19:57:48] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [19:57:49] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [19:57:58] PROBLEM - mw182 Puppet on mw182 is UNKNOWN: NRPE: Unable to read output [19:58:01] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [19:58:09] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:58:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.50, 7.98, 7.87 [19:58:17] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.26, 7.46, 7.42 [19:59:41] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:00:15] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:00:16] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:00:24] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:00:42] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [20:01:39] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [20:01:54] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [20:02:22] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:23] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:02:44] !log upgraded puppetserver to puppet8, debian to debian bookworm, ruby to 3.1.2, and openjdk to 17.0.9 on puppet181 [20:02:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.09, 7.64, 7.93 [20:03:45] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.27, 3.46, 2.59 [20:04:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.34, 8.15, 7.89 [20:04:04] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:04:07] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:04:37] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:05:04] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:05:44] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.38, 3.09, 2.55 [20:06:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.41, 7.64, 7.47 [20:06:17] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:06:28] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:06:39] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:06:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.11, 7.38, 7.72 [20:07:04] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:11] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [20:07:33] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [20:07:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.47, 7.92, 7.85 [20:08:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.18, 7.34, 7.38 [20:08:43] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.89, 7.34, 7.65 [20:09:19] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:09:45] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 15.21, 9.85, 8.52 [20:09:51] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:09:53] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:10:12] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 10.83, 8.36, 7.72 [20:10:42] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.78, 8.63, 8.11 [20:11:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/24cc5409aef8...7cae51045238 [20:11:43] [02miraheze/puppet] 07Universal-Omega 037cae510 - Delete hieradata/hosts/cp32.yaml [20:12:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/7cae51045238...3c84ec30ca2b [20:12:07] [02miraheze/puppet] 07Universal-Omega 033c84ec3 - Delete hieradata/hosts/cp33.yaml [20:12:11] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.67, 7.77, 7.60 [20:12:40] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.15, 7.99, 7.94 [20:13:03] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:14:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c84ec30ca2b...0a41c3104c98 [20:14:25] [02miraheze/puppet] 07Universal-Omega 030a41c31 - puppet141 -> puppet181 [20:14:32] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-puppetdb121 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/eccb31c63377 [20:15:40] [02miraheze/puppet] 07Universal-Omega 03eccb31c - Remove puppetdb121 [20:15:41] [02puppet] 07Universal-Omega created branch 03remove-puppetdb121 - 13https://github.com/miraheze/puppet [20:15:43] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:15:44] [02puppet] 07Universal-Omega opened pull request 03#3604: Remove puppetdb121 - 13https://github.com/miraheze/puppet/pull/3604 [20:15:55] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-puppetdb121 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eccb31c63377...50255c64cb21 [20:16:26] [02miraheze/puppet] 07Universal-Omega 0350255c6 - Remove [20:16:28] [02puppet] 07Universal-Omega synchronize pull request 03#3604: Remove puppetdb121 - 13https://github.com/miraheze/puppet/pull/3604 [20:16:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.33, 8.21, 8.03 [20:17:12] [02puppet] 07Universal-Omega closed pull request 03#3604: Remove puppetdb121 - 13https://github.com/miraheze/puppet/pull/3604 [20:17:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/0a41c3104c98...b015eea6374e [20:17:14] [02miraheze/puppet] 07Universal-Omega 03b015eea - Remove puppetdb121 (#3604) [20:17:16] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-puppetdb121 [20:17:17] [02puppet] 07Universal-Omega deleted branch 03remove-puppetdb121 - 13https://github.com/miraheze/puppet [20:18:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-puppet141 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/e22854b3a7f4 [20:18:09] [02miraheze/puppet] 07Universal-Omega 03e22854b - Remove puppet141 [20:18:11] [02puppet] 07Universal-Omega created branch 03remove-puppet141 - 13https://github.com/miraheze/puppet [20:18:13] [02puppet] 07Universal-Omega opened pull request 03#3605: Remove puppet141 - 13https://github.com/miraheze/puppet/pull/3605 [20:19:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-puppet141 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e22854b3a7f4...65d247c8529e [20:19:31] [02miraheze/puppet] 07Universal-Omega 0365d247c - Remove [20:19:38] [02puppet] 07Universal-Omega synchronize pull request 03#3605: Remove puppet141 - 13https://github.com/miraheze/puppet/pull/3605 [20:22:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.37, 7.43, 7.82 [20:23:24] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.33, 7.12, 7.90 [20:25:14] PROBLEM - ping6 on swiftproxy181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.58 ms [20:25:30] PROBLEM - ping6 on mon181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.99 ms [20:26:07] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 7.56, 7.94 [20:26:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/559754f36716 [20:26:43] [02miraheze/puppet] 07paladox 03559754f - Install swiftobject181/swiftproxy181 [20:26:46] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [20:26:47] [02puppet] 07paladox opened pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:27:18] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.hsdfz.org.cn All nameservers failed to answer the query. [20:27:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/559754f36716...5a5bbb3bbba6 [20:27:29] [02miraheze/puppet] 07paladox 035a5bbb3 - Create swiftobject151.yaml [20:27:32] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:27:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/5a5bbb3bbba6...4564260f86fa [20:27:41] [02miraheze/puppet] 07paladox 034564260 - Create swiftobject161.yaml [20:27:43] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:27:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/4564260f86fa...f050c6d25b66 [20:28:01] [02miraheze/puppet] 07paladox 03f050c6d - Create swiftobject171.yaml [20:28:03] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:28:14] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.69, 9.61, 8.79 [20:28:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/f050c6d25b66...d7de8f00c266 [20:28:56] [02miraheze/puppet] 07paladox 03d7de8f0 - Create swiftproxy181.yaml [20:28:58] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:29:02] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.27, 10.10, 8.79 [20:29:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/d7de8f00c266...e3f38c517c5d [20:29:26] [02miraheze/puppet] 07paladox 03e3f38c5 - Create swiftproxy151.yaml [20:29:27] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:30:10] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.34, 9.09, 8.70 [20:30:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3f38c517c5d...1a97ee4d6637 [20:30:51] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [20:30:54] [02miraheze/puppet] 07paladox 031a97ee4 - Update site.pp [20:30:57] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:30:57] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.24, 9.51, 8.74 [20:31:01] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [20:31:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/1a97ee4d6637...a86bcf9a1d94 [20:31:14] [02miraheze/puppet] 07paladox 03a86bcf9 - Create swiftac171.yaml [20:31:16] [02puppet] 07paladox synchronize pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:31:45] [02puppet] 07paladox edited pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:31:57] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.29, 7.83, 7.89 [20:31:59] [02puppet] 07paladox closed pull request 03#3606: Install swiftobject181/swiftproxy181 - 13https://github.com/miraheze/puppet/pull/3606 [20:32:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+7/-0/±1] 13https://github.com/miraheze/puppet/compare/b015eea6374e...b1740b6d31f8 [20:32:04] [02miraheze/puppet] 07paladox 03b1740b6 - Install swiftobject181/swiftproxy181 (#3606) [20:32:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [20:32:06] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [20:33:48] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [20:33:54] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.56, 7.29, 7.69 [20:34:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-wt [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5ab9fc45ee32 [20:34:15] [02miraheze/puppet] 07Universal-Omega 035ab9fc4 - mwdeploy: temporary change to support wikitide.net in host [20:34:18] [02puppet] 07Universal-Omega created branch 03mwdeploy-wt - 13https://github.com/miraheze/puppet [20:34:21] [02puppet] 07Universal-Omega opened pull request 03#3607: mwdeploy: temporary change to support wikitide.net in host - 13https://github.com/miraheze/puppet/pull/3607 [20:35:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.70, 7.56, 7.64 [20:35:33] miraheze/puppet - Universal-Omega the build has errored. [20:37:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/60757875bc82 [20:37:19] [02miraheze/puppet] 07paladox 036075787 - swift::proxy: add support for python 3.11 [20:37:21] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [20:37:21] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.06, 7.45, 7.41 [20:37:23] [02puppet] 07paladox opened pull request 03#3608: swift::proxy: add support for python 3.11 - 13https://github.com/miraheze/puppet/pull/3608 [20:37:34] PROBLEM - swiftproxy181 Puppet on swiftproxy181 is UNKNOWN: NRPE: Unable to read output [20:37:37] [02puppet] 07paladox closed pull request 03#3608: swift::proxy: add support for python 3.11 - 13https://github.com/miraheze/puppet/pull/3608 [20:37:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [20:37:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1740b6d31f8...5691e859b8fc [20:37:41] [02miraheze/puppet] 07paladox 035691e85 - swift::proxy: add support for python 3.11 (#3608) [20:37:42] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [20:37:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.24, 7.78, 7.79 [20:38:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.42, 3.80, 2.46 [20:39:18] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.70, 7.12, 7.29 [20:39:31] RECOVERY - swiftproxy181 Puppet on swiftproxy181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:39:46] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.79, 7.75, 7.77 [20:40:22] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.15, 3.76, 2.63 [20:41:00] !log INSERT INTO slot_roles (role_id, role_name) VALUES (1, 'main'); (leyendablancawiki) [20:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:11] [02puppet] 07Universal-Omega synchronize pull request 03#3607: mwdeploy: temporary change to support wikitide.net in host - 13https://github.com/miraheze/puppet/pull/3607 [20:41:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-wt [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5ab9fc45ee32...206da23bc71b [20:41:15] [02miraheze/puppet] 07Universal-Omega 03206da23 - Fix [20:41:22] !log INSERT INTO content_models (model_id, model_name) VALUES ( 1, 'wikitext'); + css,javascript,Scribunto,json in that order (leyendablancawiki) [20:41:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:42:20] miraheze/puppet - Universal-Omega the build has errored. [20:42:22] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.63, 2.95, 2.46 [20:42:56] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/sql.php --wiki=leyendablancawiki (END - exit=2) [20:43:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.08, 7.69, 7.88 [20:45:07] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.14, 7.35, 7.26 [20:45:44] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.93, 11.60, 10.16 [20:46:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-wt [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/206da23bc71b...67cbd8fa1a94 [20:46:03] [02miraheze/puppet] 07Universal-Omega 0367cbd8f - Fix tests [20:46:06] [02puppet] 07Universal-Omega synchronize pull request 03#3607: mwdeploy: temporary change to support wikitide.net in host - 13https://github.com/miraheze/puppet/pull/3607 [20:47:20] miraheze/puppet - Universal-Omega the build passed. [20:47:39] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.86, 10.42, 9.90 [20:47:58] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:47:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:48:11] [02puppet] 07paladox closed pull request 03#3607: mwdeploy: temporary change to support wikitide.net in host - 13https://github.com/miraheze/puppet/pull/3607 [20:48:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/5691e859b8fc...3c79bcc1c515 [20:48:15] [02miraheze/puppet] 07Universal-Omega 033c79bcc - mwdeploy: temporary change to support wikitide.net in host (#3607) [20:48:18] [02miraheze/puppet] 07paladox deleted branch 03mwdeploy-wt [20:48:20] [02puppet] 07paladox deleted branch 03mwdeploy-wt - 13https://github.com/miraheze/puppet [20:49:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.70, 7.93, 7.83 [20:49:29] miraheze/puppet - paladox the build passed. [20:49:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.61, 7.86, 7.72 [20:49:33] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.69, 7.34, 7.06 [20:49:35] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.99, 9.53, 9.62 [20:49:37] [02puppet] 07Universal-Omega closed pull request 03#3605: Remove puppet141 - 13https://github.com/miraheze/puppet/pull/3605 [20:49:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/puppet/compare/3c79bcc1c515...10a23534b24b [20:49:41] [02miraheze/puppet] 07Universal-Omega 0310a2353 - Remove puppet141 (#3605) [20:49:43] [02miraheze/puppet] 07Universal-Omega deleted branch 03remove-puppet141 [20:49:44] [02puppet] 07Universal-Omega deleted branch 03remove-puppet141 - 13https://github.com/miraheze/puppet [20:49:50] [02dns] 07Universal-Omega closed pull request 03#455: Remove puppet141 from DNS - 13https://github.com/miraheze/dns/pull/455 [20:49:52] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-puppet141 [20:49:53] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.222 second response time [20:49:53] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/7f5d60180289...32d5cdcf5494 [20:49:56] [02miraheze/dns] 07Universal-Omega 0332d5cdc - Remove puppet141 from DNS (#455) [20:49:57] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.756 second response time [20:49:59] [02dns] 07Universal-Omega deleted branch 03remove-puppet141 - 13https://github.com/miraheze/dns [20:51:04] !log [paladox@mwtask141] finished deploy of {'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40'], 'servers': ['mw181', 'mw182']} to [mw181, mw182] - SUCCESS in 25s [20:51:06] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/32d5cdcf5494...d5046dc897ef [20:51:08] [02miraheze/dns] 07Universal-Omega 03d5046dc - Remove puppetdb121 from DNS [20:51:20] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 200 - 320 bytes in 0.401 second response time [20:51:30] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.41, 7.38, 7.11 [20:52:37] RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 200 - 320 bytes in 0.403 second response time [20:52:52] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.02, 10.75, 9.55 [20:53:45] PROBLEM - ping6 on bast181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.21 ms [20:54:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.60, 7.99, 7.39 [20:54:16] PROBLEM - swiftobject181 Swift Object Service on swiftobject181 is CRITICAL: connect to address 2602:294:0:b12::108 and port 6000: Connection refused [20:54:55] PROBLEM - ping6 on mwtask181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.17 ms [20:55:10] PROBLEM - ping6 on swiftobject181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.24 ms [20:55:47] PROBLEM - ping6 on bast181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.08 ms [20:56:42] PROBLEM - wiki.hsdfz.org.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hsdfz.org.cn could not be found [20:56:50] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.83, 7.93, 7.74 [20:57:27] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.05, 10.24, 9.04 [20:57:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.95, 7.64, 7.40 [20:58:14] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 8.52, 10.31, 10.03 [20:58:23] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:58:51] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.41, 10.52, 9.35 [20:59:09] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.30, 7.64, 7.86 [20:59:21] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [20:59:23] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.86, 9.85, 9.01 [20:59:53] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [21:00:11] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.71, 11.16, 10.34 [21:00:19] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.206 second response time [21:00:51] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.82, 12.01, 10.05 [21:01:05] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.62, 7.83, 7.89 [21:01:18] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.25, 11.18, 9.60 [21:01:19] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 2.172 second response time [21:01:54] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [21:02:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.75, 11.12, 10.45 [21:02:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/77a89814e1b8 [21:02:41] [02miraheze/puppet] 07paladox 0377a8981 - Install mwtask181 [21:02:44] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:02:47] [02puppet] 07paladox opened pull request 03#3609: Install mwtask181 - 13https://github.com/miraheze/puppet/pull/3609 [21:02:50] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.83, 10.94, 9.91 [21:03:01] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.60, 7.70, 7.84 [21:03:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/77a89814e1b8...cd92033f1cb5 [21:03:09] [02miraheze/puppet] 07paladox 03cd92033 - Update site.pp [21:03:12] [02puppet] 07paladox synchronize pull request 03#3609: Install mwtask181 - 13https://github.com/miraheze/puppet/pull/3609 [21:03:14] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.58, 10.18, 9.42 [21:03:14] [02puppet] 07paladox closed pull request 03#3609: Install mwtask181 - 13https://github.com/miraheze/puppet/pull/3609 [21:03:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [21:03:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/10a23534b24b...7e812d8b0937 [21:03:19] [02miraheze/puppet] 07paladox 037e812d8 - Install mwtask181 (#3609) [21:03:20] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:03:49] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.180 second response time [21:03:57] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.18, 10.94, 9.14 [21:04:04] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.36, 9.65, 9.99 [21:04:11] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.80, 3.58, 2.99 [21:04:37] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.95, 9.76, 9.97 [21:04:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e812d8b0937...9c3336256670 [21:04:40] [02miraheze/puppet] 07paladox 039c33362 - Fix [21:04:50] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.48, 9.59, 9.53 [21:04:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.82, 8.08, 7.96 [21:05:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.19, 7.60, 7.41 [21:05:55] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.71, 9.18, 8.70 [21:07:14] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.76, 10.53, 9.74 [21:07:34] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.42, 7.55, 7.42 [21:07:34] PROBLEM - mwtask181 APT on mwtask181 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:08:09] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.71, 3.16 [21:09:14] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.27, 9.54, 9.48 [21:09:30] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:10:07] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.02, 3.57, 3.19 [21:10:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 10.03, 8.08, 7.70 [21:12:05] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.45, 3.26, 3.12 [21:12:53] [02miraheze/MirahezeDebug] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/e0ce1a1600b9...b2f7be25707a [21:12:56] [02miraheze/MirahezeDebug] 07MacFan4000 03b2f7be2 - Update popup.html [21:13:19] [02miraheze/MirahezeDebug] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/b2f7be25707a...80c0f926cc84 [21:13:21] [02miraheze/MirahezeDebug] 07MacFan4000 0380c0f92 - fix [21:15:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.96, 8.17, 7.71 [21:17:08] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.59, 7.89, 7.67 [21:17:14] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.57, 10.29, 10.06 [21:19:11] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.14, 9.75, 9.90 [21:21:08] PROBLEM - mwtask181 Puppet on mwtask181 is UNKNOWN: NRPE: Unable to read output [21:22:51] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.50, 10.66, 9.83 [21:24:05] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.523 second response time [21:24:09] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 2602:294:0:b12::107 and port 10044: Connection refused [21:24:11] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 10.41, 9.77 [21:24:12] PROBLEM - mwtask181 HTTPS on mwtask181 is WARNING: HTTP WARNING: HTTP/2 404 - 146 bytes in 0.398 second response time [21:24:22] PROBLEM - mwtask181 JobRunner Service on mwtask181 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:24:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.95, 4.67, 3.62 [21:25:10] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:25:52] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.74, 10.25, 10.12 [21:26:10] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.96, 10.01, 9.72 [21:26:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.04, 3.20, 3.21 [21:27:59] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.129 second response time on 2602:294:0:b12::107 port 10044 [21:28:17] RECOVERY - mwtask181 JobRunner Service on mwtask181 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:28:37] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:28:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.77, 7.96, 7.52 [21:28:48] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.95, 9.72, 9.74 [21:29:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/adfd2675b7cb [21:29:52] [02miraheze/puppet] 07paladox 03adfd267 - mediawiki::deploy: install langcodes with --break-system-packages [21:29:53] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:29:54] [02puppet] 07paladox opened pull request 03#3610: mediawiki::deploy: install langcodes with --break-system-packages - 13https://github.com/miraheze/puppet/pull/3610 [21:29:58] [02puppet] 07paladox closed pull request 03#3610: mediawiki::deploy: install langcodes with --break-system-packages - 13https://github.com/miraheze/puppet/pull/3610 [21:30:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9c3336256670...b9a88c6a4f08 [21:30:02] [02miraheze/puppet] 07paladox 03b9a88c6 - mediawiki::deploy: install langcodes with --break-system-packages (#3610) [21:30:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [21:30:07] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.22, 11.34, 10.29 [21:30:08] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [21:32:02] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [21:32:06] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.13, 10.72, 10.22 [21:33:46] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.05, 9.27, 9.85 [21:34:45] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.82, 7.96, 7.69 [21:35:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b9a88c6a4f08...6c2b08be90a1 [21:35:21] [02miraheze/puppet] 07paladox 036c2b08b - mwdeploy: add mwtask181 [21:36:04] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.71, 9.91, 10.01 [21:36:17] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['']} to [mwtask181] [21:36:18] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['']} to [mwtask181] - SUCCESS in 0s [21:36:19] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['']} to [mwtask181] [21:36:20] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['']} to [mwtask181] - SUCCESS in 0s [21:36:32] miraheze/puppet - paladox the build passed. [21:36:38] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] [21:36:39] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 0s [21:36:40] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] [21:37:09] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [21:37:32] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 200 - 324 bytes in 0.409 second response time [21:38:10] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:12] [02miraheze/mediawiki-repos] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/d406a5f54d4b...c9bb9a934b4b [21:40:13] [02miraheze/mediawiki-repos] 07paladox 03c9bb9a9 - Lock StandardDialogs to _branch_ [21:42:18] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] [21:42:25] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 6s [21:42:26] !log [paladox@mwtask181] starting deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] [21:42:28] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [21:42:45] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 10.00, 7.66, 7.53 [21:44:21] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.532 second response time [21:45:33] !log [paladox@mwtask181] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40']} to [mwtask181] - SUCCESS in 194s [21:51:32] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.44, 9.11, 8.82 [21:51:54] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.40, 9.95, 9.67 [21:53:53] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 10.13, 9.81, 9.64 [21:57:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.97, 11.45, 9.93 [21:58:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.98, 7.58, 7.94 [22:00:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/cdaa77533b0b [22:00:33] [02miraheze/puppet] 07paladox 03cdaa775 - Install mon181 [22:00:35] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [22:00:38] [02puppet] 07paladox opened pull request 03#3611: Install mon181 - 13https://github.com/miraheze/puppet/pull/3611 [22:00:53] PROBLEM - test131 Puppet on test131 is UNKNOWN: NRPE: Unable to read output [22:00:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cdaa77533b0b...a558b943322e [22:00:57] [02miraheze/puppet] 07paladox 03a558b94 - Update site.pp [22:00:57] [02puppet] 07paladox synchronize pull request 03#3611: Install mon181 - 13https://github.com/miraheze/puppet/pull/3611 [22:01:01] [02puppet] 07paladox closed pull request 03#3611: Install mon181 - 13https://github.com/miraheze/puppet/pull/3611 [22:01:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [22:01:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/6c2b08be90a1...874d9f460b9e [22:01:09] [02miraheze/puppet] 07paladox 03874d9f4 - Install mon181 (#3611) [22:01:10] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [22:01:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/874d9f460b9e...3020823de1be [22:01:53] [02miraheze/puppet] 07paladox 033020823 - Fix [22:02:46] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.52, 10.89, 10.16 [22:02:51] !log [paladox@mwtask141] finished deploy of {'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'force': True, 'versions': ['1.40'], 'servers': ['mw181', 'mw182']} to [mw181, mw182] - SUCCESS in 4331s [22:03:22] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [22:03:25] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.95, 10.44, 9.64 [22:03:39] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [22:04:45] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.71, 9.43, 9.72 [22:05:21] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.50, 10.91, 10.59 [22:05:24] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.43, 9.94, 9.55 [22:05:39] PROBLEM - mon181 Puppet on mon181 is UNKNOWN: NRPE: Unable to read output [22:06:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3020823de1be...9f79c1cbd1ee [22:06:12] [02miraheze/puppet] 07paladox 039f79c1c - monitoring: fix installing mariadb for bookworm [22:09:19] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.15, 8.75, 9.79 [22:10:38] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.42, 8.08, 7.84 [22:10:41] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.16, 11.05, 10.16 [22:11:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03commit [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ac1c53114658 [22:11:18] [02miraheze/puppet] 07Universal-Omega 03ac1c531 - Support commit in mediawiki-repos [22:11:20] [02puppet] 07Universal-Omega created branch 03commit - 13https://github.com/miraheze/puppet [22:11:22] [02puppet] 07Universal-Omega opened pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:12:23] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.23, 13.21, 10.83 [22:13:14] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.52, 11.71, 10.89 [22:13:18] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.76, 10.81, 9.97 [22:13:32] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [22:14:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03commit [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ac1c53114658...2240b73cd7de [22:14:36] [02miraheze/puppet] 07Universal-Omega 032240b73 - Add commit support [22:14:37] [02puppet] 07Universal-Omega synchronize pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:14:39] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.00, 10.33, 10.12 [22:14:53] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.11, 10.66, 8.16 [22:15:13] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 11.26, 12.04, 11.13 [22:15:17] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.43, 10.17, 9.86 [22:16:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.39, 9.79, 9.96 [22:16:51] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.83, 9.75, 8.13 [22:19:13] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.28, 10.62, 10.11 [22:20:36] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.19, 7.77, 7.97 [22:24:55] [02puppet] 07Universal-Omega synchronize pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:24:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03commit [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2240b73cd7de...f85c0ceb24af [22:24:59] [02miraheze/puppet] 07Universal-Omega 03f85c0ce - Fix [22:25:10] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.73, 10.12, 10.06 [22:25:11] PROBLEM - mon181 IRC Log Server Bot on mon181 is UNKNOWN: NRPE: Command 'check_IRC Log Server Bot' not defined [22:25:23] PROBLEM - mon181 Check correctness of the icinga configuration on mon181 is UNKNOWN: NRPE: Command 'check_Check correctness of the icinga configuration' not defined [22:25:24] PROBLEM - mon181 IRC-Discord Relay Bot on mon181 is UNKNOWN: NRPE: Command 'check_IRC-Discord Relay Bot' not defined [22:25:28] PROBLEM - mon181 icinga.miraheze.org HTTPS on mon181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:25:38] PROBLEM - mon181 Backups Grafana on mon181 is UNKNOWN: NRPE: Command 'check_Backups Grafana' not defined [22:25:48] PROBLEM - mon181 IRC RC Bot on mon181 is UNKNOWN: NRPE: Command 'check_IRC RC Bot' not defined [22:25:49] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.32, 11.65, 11.51 [22:25:51] PROBLEM - mon181 HTTPS on mon181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:25:55] PROBLEM - mon181 IRCEcho on mon181 is UNKNOWN: NRPE: Command 'check_IRCEcho' not defined [22:25:57] PROBLEM - mon181 IRC Log Bot on mon181 is UNKNOWN: NRPE: Command 'check_IRC Log Bot' not defined [22:26:03] PROBLEM - mon181 php-fpm on mon181 is UNKNOWN: NRPE: Command 'check_php-fpm' not defined [22:26:19] PROBLEM - mon181 grafana.miraheze.org HTTPS on mon181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:08] PROBLEM - mon141 Puppet on mon141 is UNKNOWN: NRPE: Unable to read output [22:28:18] [02miraheze/puppet] 07Universal-Omega pushed 034 commits to 03commit [+2/-0/±5] 13https://github.com/miraheze/puppet/compare/f85c0ceb24af...7db5b8ed985e [22:28:20] [02miraheze/puppet] 07Universal-Omega 037db5b8e - Merge branch 'master' into commit [22:28:23] [02puppet] 07Universal-Omega synchronize pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:28:35] PROBLEM - ping6 on mon181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.04 ms [22:29:07] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.85, 10.52, 10.24 [22:30:37] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/c9bb9a934b4b...5536a8835017 [22:30:38] [02miraheze/mediawiki-repos] 07Universal-Omega 035536a88 - Use commit for WikiSEO [22:30:39] PROBLEM - ping6 on mon181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.14 ms [22:30:55] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:31:34] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:32:20] [02puppet] 07Universal-Omega synchronize pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:32:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03commit [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7db5b8ed985e...876fe41b12a2 [22:32:22] [02miraheze/puppet] 07Universal-Omega 03876fe41 - Fix [22:33:00] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.81, 11.06, 11.51 [22:33:05] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.71, 9.63, 10.00 [22:33:20] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:33:44] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:34:59] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 11.59, 11.64 [22:35:17] PROBLEM - mwtask141 Puppet on mwtask141 is UNKNOWN: NRPE: Unable to read output [22:35:47] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.69, 5.49, 6.68 [22:37:21] PROBLEM - ping6 on mw181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.34 ms [22:37:29] [02puppet] 07Universal-Omega synchronize pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [22:37:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03commit [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/876fe41b12a2...dcb4206d294c [22:37:34] [02miraheze/puppet] 07Universal-Omega 03dcb4206 - Remove fetch [22:37:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.82, 7.17, 7.95 [22:39:23] PROBLEM - ping6 on mw181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.01 ms [22:41:16] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.58, 9.21, 10.13 [22:42:53] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.77, 11.13, 11.64 [22:43:26] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.11, 7.52, 7.83 [22:43:58] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.88, 10.64, 10.36 [22:45:57] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.80, 9.37, 9.91 [22:46:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.86, 7.08, 6.90 [22:46:44] PROBLEM - wiki.walkscape.app - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.walkscape.app All nameservers failed to answer the query. [22:47:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.76, 7.82, 7.90 [22:48:17] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.30, 6.55, 6.74 [22:49:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.36, 8.61, 8.17 [22:50:36] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.11, 10.43, 9.94 [22:52:35] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.72, 9.28, 9.57 [22:52:46] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.62, 8.28, 9.88 [22:54:13] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:54:53] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.71, 7.69, 7.98 [22:56:30] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 10.79, 10.12 [22:57:48] PROBLEM - ping6 on cloud18 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.42 ms [22:57:54] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.03, 6.74, 6.67 [22:58:29] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.28, 10.35, 10.05 [22:58:49] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 7.92, 8.17, 8.12 [22:59:40] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.63, 10.69, 10.52 [22:59:48] PROBLEM - ping6 on cloud18 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.88 ms [22:59:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.27, 6.45, 6.57 [23:00:27] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.02, 11.24, 10.40 [23:01:00] [02puppet] 07Universal-Omega closed pull request 03#3612: Support commit in mediawiki-repos - 13https://github.com/miraheze/puppet/pull/3612 [23:01:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/9f79c1cbd1ee...e83f275f7b2d [23:01:04] [02miraheze/puppet] 07Universal-Omega 03e83f275 - Support commit in mediawiki-repos (#3612) [23:01:06] [02miraheze/puppet] 07Universal-Omega deleted branch 03commit [23:01:08] [02puppet] 07Universal-Omega deleted branch 03commit - 13https://github.com/miraheze/puppet [23:02:26] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.36, 10.35, 10.19 [23:02:47] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [23:03:02] PROBLEM - db181 Puppet on db181 is UNKNOWN: NRPE: Unable to read output [23:03:08] PROBLEM - db182 Puppet on db182 is UNKNOWN: NRPE: Unable to read output [23:03:37] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.41, 9.19, 10.01 [23:04:24] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.03, 9.03, 9.72 [23:05:32] RECOVERY - mon181 grafana.miraheze.org HTTPS on mon181 is OK: HTTP OK: HTTP/1.1 302 Found - 429 bytes in 0.543 second response time [23:05:57] RECOVERY - mon181 HTTPS on mon181 is OK: HTTP OK: HTTP/2 302 - 374 bytes in 0.399 second response time [23:08:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e83f275f7b2d...9e84cf6762e8 [23:08:10] [02miraheze/puppet] 07paladox 039e84cf6 - Fix [23:08:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e84cf6762e8...0e342a55034f [23:08:54] [02miraheze/puppet] 07paladox 030e342a5 - Fix [23:09:36] RECOVERY - mon181 icinga.miraheze.org HTTPS on mon181 is OK: HTTP OK: HTTP/1.1 302 Found - 297 bytes in 0.669 second response time [23:09:46] [02puppet] 07Universal-Omega closed pull request 03#3588: Fixed discord author parsing in replied messages for MirahezeLogbot - 13https://github.com/miraheze/puppet/pull/3588 [23:09:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e342a55034f...4eaad47dcd27 [23:09:49] [02miraheze/puppet] 07pixDeVl 034eaad47 - Fixed discord author parsing in replied messages for MirahezeLogbot (#3588) [23:11:02] miraheze/puppet - Universal-Omega the build passed. [23:13:45] . [23:15:15] . [23:15:15] . [23:15:35] I thought that might happen lol [23:15:35] I thought that might happen lol [23:16:20] oh boy [23:16:21] oh boy [23:16:26] what now [23:16:26] what now [23:16:39] @cosmicalpha what have you done [23:16:39] @cosmicalpha what have you done [23:16:57] We are setting up new infra so have 2 bots running IRC side now lol [23:16:57] We are setting up new infra so have 2 bots running IRC side now lol [23:17:10] a lmao [23:17:11] a lmao [23:17:20] do logs double? [23:17:20] do logs double? [23:17:23] PROBLEM - ping6 on bast181 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.71 ms [23:17:37] Probably [23:17:37] Probably [23:19:24] PROBLEM - ping6 on bast181 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.44 ms [23:22:15] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [23:25:05] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 11.09, 10.05 [23:28:23] PROBLEM - ping6 on db182 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.17 ms [23:29:02] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.99, 11.32, 10.40 [23:30:22] PROBLEM - ping6 on db182 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.16 ms [23:30:53] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:31:01] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.28, 10.06, 10.04 [23:39:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4eaad47dcd27...a94277f262fa [23:39:28] [02miraheze/puppet] 07paladox 03a94277f - nrpe: allow mon181 to connect [23:40:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a94277f262fa...4e0c5233746d [23:40:34] [02miraheze/puppet] 07paladox 034e0c523 - nginx: fix firewall [23:42:04] RECOVERY - mon181 IRC Log Server Bot on mon181 is OK: PROCS OK: 1 process with args 'irclogserverbot.py' [23:42:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.04, 6.60, 7.56 [23:46:46] !log test [23:47:33] h [23:47:40] well the echo stfu [23:47:47] !log test [23:47:51] aww, bot ded [23:47:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:47:57] oh [23:47:59] It works Discord side not IRC still [23:48:04] which one was that [23:48:08] h o w [23:48:13] !log buddy you wake? [23:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:00] I think because discord_author is not set but it is used even if not from Discord an undefined variable does not seem to be liked. [23:49:07] !log test [23:49:33] Is should default to author if discord auth dont exist [23:49:58] But it is checked for the username mapping [23:50:13] outside of the discord check [23:50:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5f019b9047e9 [23:50:21] [02miraheze/puppet] 07paladox 035f019b9 - Switch mon141 to mon181 [23:50:22] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [23:50:24] [02puppet] 07paladox opened pull request 03#3613: Switch mon141 to mon181 - 13https://github.com/miraheze/puppet/pull/3613 [23:50:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f019b9047e9...ffaf46d3324a [23:50:45] [02miraheze/puppet] 07paladox 03ffaf46d - Update default.vcl [23:50:48] [02puppet] 07paladox synchronize pull request 03#3613: Switch mon141 to mon181 - 13https://github.com/miraheze/puppet/pull/3613 [23:50:53] [02puppet] 07paladox closed pull request 03#3613: Switch mon141 to mon181 - 13https://github.com/miraheze/puppet/pull/3613 [23:50:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/4e0c5233746d...b4922e602fec [23:50:55] [02miraheze/puppet] 07paladox 03b4922e6 - Switch mon141 to mon181 (#3613) [23:50:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [23:50:59] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [23:51:03] eidiufheufihg which line [23:52:33] [23:53:25] dont like the undefined? [23:53:35] aight, should be able to just add else i think? [23:53:48] dont like the undefined? [23:53:48] aight, should be able to just add else i think? [23:53:59] PROBLEM - txt-dex.com - reverse DNS on sslhost is UNKNOWN: [23:54:02] RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct [23:54:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [23:54:12] Here we go again [23:54:35] echo [23:54:37] echo [23:54:58] RECOVERY - wiki.walkscape.app - reverse DNS on sslhost is OK: SSL OK - wiki.walkscape.app reverse DNS resolves to cp24.miraheze.org - CNAME OK [23:55:05] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/grafana-backup.log [23:56:30] RECOVERY - mon181 php-fpm on mon181 is OK: PROCS OK: 21 processes with command name 'php-fpm8.2' [23:56:50] RECOVERY - mon181 IRCEcho on mon181 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [23:57:00] RECOVERY - mon181 IRC RC Bot on mon181 is OK: PROCS OK: 1 process with args 'ircrcbot.py' [23:57:20] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [23:57:26] RECOVERY - mon181 IRC Log Bot on mon181 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [23:58:02] RECOVERY - mon181 IRC-Discord Relay Bot on mon181 is OK: PROCS OK: 2 processes with args 'relaybot'