[00:01:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.58, 7.96, 7.85 [00:03:20] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.15, 4.71, 2.41 [00:03:24] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.41, 9.76, 6.24 [00:03:40] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw143 [00:03:49] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [00:03:54] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:04:09] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw143 [00:04:22] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.96, 9.34, 6.99 [00:04:36] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.73, 11.47, 7.86 [00:04:37] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 16.54, 10.53, 5.44 [00:05:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.03, 3.70, 2.32 [00:05:23] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.33, 8.79, 6.33 [00:05:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.65, 8.00, 7.86 [00:05:40] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:05:49] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:05:50] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.133 second response time [00:06:09] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:06:22] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.05, 8.58, 7.02 [00:06:36] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.42, 9.90, 7.73 [00:07:10] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.03, 2.84, 2.17 [00:07:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.71, 7.65, 7.76 [00:10:22] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw132.miraheze.org port 443 after 0 ms: Couldn't connect to server [00:12:09] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [00:12:23] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.230 second response time [00:12:37] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.26, 7.78, 6.72 [00:14:09] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:14:37] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.90, 6.56, 6.40 [00:15:00] !log depool mw131 and mw132 [00:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.55, 7.88, 7.83 [00:16:23] !log repool mw131 and mw132 [00:16:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:16:57] [02mw-config] 07Universal-Omega synchronize pull request 03#5424: Add CategoryTests to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5424 [00:17:37] [02mw-config] 07Universal-Omega synchronize pull request 03#5424: Add CategoryTests to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5424 [00:17:55] [02mw-config] 07Universal-Omega synchronize pull request 03#5267: Install RealMe - 13https://github.com/miraheze/mw-config/pull/5267 [00:18:05] miraheze/mw-config - Universal-Omega the build has errored. [00:18:41] miraheze/mw-config - Universal-Omega the build passed. [00:18:41] [02mw-config] 07Universal-Omega synchronize pull request 03#5267: Install RealMe - 13https://github.com/miraheze/mw-config/pull/5267 [00:18:49] miraheze/mw-config - Universal-Omega the build has errored. [00:19:01] da fac depool mean [00:19:47] miraheze/mw-config - Universal-Omega the build passed. [00:21:01] If a server is pooled, that means it is receiving production traffic. If it isn't, then it's not. [00:21:27] Ah, pull off the web for maintaince? [00:21:36] Or for testing [00:23:52] [02miraheze/mediawiki-repos] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/commit/598fdc1d0470 [00:23:54] [02miraheze/mediawiki-repos] 07paladox 03598fdc1 - Update WikiSeo to b05926d5fdf8bf76b7eb5882734e0d9f5c2c3315 [00:23:56] [02mediawiki-repos] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mediawiki-repos [00:23:59] [02mediawiki-repos] 07paladox opened pull request 03#2: Update WikiSeo to b05926d5fdf8bf76b7eb5882734e0d9f5c2c3315 - 13https://github.com/miraheze/mediawiki-repos/pull/2 [00:24:40] [02mediawiki-repos] 07Universal-Omega closed pull request 03#2: Update WikiSeo to b05926d5fdf8bf76b7eb5882734e0d9f5c2c3315 - 13https://github.com/miraheze/mediawiki-repos/pull/2 [00:24:43] [02miraheze/mediawiki-repos] 07Universal-Omega deleted branch 03paladox-patch-1 [00:24:46] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/16c271b22e09...d406a5f54d4b [00:24:48] [02miraheze/mediawiki-repos] 07paladox 03d406a5f - Update WikiSeo to b05926d5fdf8bf76b7eb5882734e0d9f5c2c3315 (#2) [00:24:51] [02mediawiki-repos] 07Universal-Omega deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mediawiki-repos [00:29:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.68, 7.66, 7.97 [00:33:12] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [00:33:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.75, 8.18, 8.06 [00:34:23] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_40 WikiSEO] [00:35:44] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.71, 3.58, 3.06 [00:37:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.34, 7.46, 7.82 [00:37:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.76, 3.24, 3.00 [00:39:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.69, 7.59, 7.80 [00:39:37] !log [universalomega@mwtask141] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [00:39:38] !log [universalomega@mwtask141] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 0s [00:39:39] !log [universalomega@mwtask141] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [00:39:53] !log [universalomega@mwtask141] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [00:39:54] !log [universalomega@mwtask141] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 0s [00:39:55] !log [universalomega@mwtask141] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [00:40:49] !log [universalomega@mwtask141] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiSEO']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 56s [00:40:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.69, 2.45, 1.61 [00:41:12] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.550 second response time [00:41:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.97, 7.25, 7.64 [00:42:20] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:42:55] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.17, 1.95, 1.53 [00:45:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.35, 7.75, 7.69 [00:47:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.62, 7.57, 7.63 [00:51:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.23, 8.30, 7.88 [00:53:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.88, 7.99, 7.81 [00:55:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.11, 8.44, 7.99 [00:57:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.57, 7.56, 7.95 [00:59:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.28, 7.32, 7.69 [00:59:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 7.99, 7.93, 8.05 [01:01:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.26, 8.26, 8.00 [01:13:54] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:15:50] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.141 second response time [01:17:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.77, 7.70, 8.00 [01:19:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.26, 7.79, 7.99 [01:19:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.13, 7.30, 7.99 [01:21:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.92, 7.75, 7.94 [01:21:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.61, 8.09, 8.19 [01:22:41] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:24:37] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.217 second response time [01:27:17] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [01:27:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 7.71, 7.99 [01:28:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e3ba71045d8e...6e8f887d2097 [01:28:07] [02miraheze/mw-config] 07Universal-Omega 036e8f887 - Update CONTRIBUTING for mediawiki-repos [01:29:01] miraheze/mw-config - Universal-Omega the build passed. [01:29:12] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.138 second response time [01:29:41] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.03, 7.74, 7.95 [01:33:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.45, 7.56, 7.56 [01:35:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.85, 7.54, 7.54 [01:38:18] I was taking those servers off of production because they were being unresponsive when I was trying to fix no extensions loading by resetting ManageWiki cache. All other servers worked immediately without further intervention but those two just decided to not respond so I depooled to fix the issue. But as soon as I did they started working again so repooled. [01:38:41] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10006 milliseconds [01:39:26] This message perfectly encapsulates systems administration [01:39:38] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [01:39:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.73, 7.63, 7.94 [01:40:07] !log [@mwtask141] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 29s [01:40:37] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.393 second response time [01:42:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6e8f887d2097...effec13bdb82 [01:42:54] [02miraheze/mw-config] 07Universal-Omega 03effec13 - Disable chameleon for now [01:43:04] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [01:43:29] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 25s [01:43:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 10.06, 8.92, 8.36 [01:43:49] miraheze/mw-config - Universal-Omega the build passed. [01:44:18] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.57, 1.49, 1.08 [01:45:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.83, 7.42, 7.45 [01:46:17] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 1.07, 1.26, 1.04 [01:47:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.34, 7.27, 7.39 [01:55:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.83, 6.89, 7.76 [02:07:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.43, 7.71, 7.31 [02:07:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.07, 7.08, 7.31 [02:09:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.19, 7.35, 7.22 [02:09:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.15, 7.09, 7.29 [02:21:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.49, 7.52, 7.30 [02:25:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:25:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.56, 6.82, 7.10 [02:27:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.32, 7.97, 7.38 [02:27:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 11.07, 7.83, 7.40 [02:31:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.32, 7.14, 7.17 [02:39:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.33, 7.90, 8.00 [02:47:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.51, 7.39, 7.14 [02:51:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.70, 8.47, 8.11 [02:53:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.62, 8.00, 7.54 [03:02:03] PROBLEM - mon141 Backups Grafana on mon141 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209720 seconds old and 785 bytes [03:03:30] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.60, 7.36, 7.34 [03:07:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.60, 7.26, 7.31 [03:09:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.24, 7.60, 7.43 [03:10:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:11:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.38, 7.45, 7.39 [03:12:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:15:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.21, 7.28, 7.25 [03:15:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.04, 7.53, 8.00 [03:16:41] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:18:37] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.222 second response time [03:19:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.28, 7.49, 7.36 [03:21:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.15, 7.61, 7.41 [03:23:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.73, 7.13, 7.25 [03:25:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.02, 7.55, 7.37 [03:25:46] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.80, 8.19, 7.98 [03:27:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.09, 7.40, 7.35 [03:27:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/71248604ff0f...aee3df5ceae1 [03:27:45] [02miraheze/ssl] 07MirahezeSSLBot 03aee3df5 - Bot: Add SSL cert for wiki.buryland.net [03:29:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.94, 7.81, 7.89 [03:32:58] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:33:41] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:35:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 12.44, 8.86, 8.18 [03:38:25] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:39:02] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:39:24] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:39:32] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:39:41] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/aee3df5ceae1...4e81f74a0dd9 [03:39:44] [02miraheze/ssl] 07MirahezeSSLBot 034e81f74 - Bot: Add SSL cert for wiki.buryland.net [03:40:16] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:40:33] miraheze/ssl - MirahezeSSLBot the build has errored. [03:42:25] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [03:42:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:43:00] [02miraheze/ssl] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4e81f74a0dd9...33e5b235de61 [03:43:02] [02miraheze/ssl] 07OAuthority 0333e5b23 - remove duplicate entry [03:43:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.14, 6.90, 6.91 [03:46:53] !log [oa@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [03:47:05] !log [oa@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 12s [03:49:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.42, 7.74, 7.32 [04:00:43] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:00:59] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:01:04] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [04:01:49] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:02:41] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.733 second response time [04:03:02] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.173 second response time [04:05:31] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:06:34] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:06:49] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:07:15] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:08:15] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.38, 6.47, 6.80 [04:08:25] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.10, 11.17, 7.53 [04:08:34] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [04:09:38] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:10:19] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:11:03] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:11:28] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10007 milliseconds with 0 bytes received [04:11:28] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [04:12:10] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.77, 6.92, 6.89 [04:12:33] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.151 second response time [04:13:00] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.159 second response time [04:13:25] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.156 second response time [04:13:27] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.198 second response time [04:14:06] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.10, 6.58, 6.77 [04:16:31] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 5.33, 10.37, 9.55 [04:17:28] PROBLEM - swiftobject122 NTP time on swiftobject122 is WARNING: NTP WARNING: Offset 0.1122740805 secs [04:18:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.62, 7.40, 7.09 [04:18:26] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.08, 8.87, 9.10 [04:23:03] PROBLEM - deadprem.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for deadprem.wiki could not be found [04:23:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.39, 7.23, 7.01 [04:27:44] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.39, 7.39, 7.17 [04:28:10] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/de8e6a8bc4f1...887716ee4617 [04:28:11] [02miraheze/dns] 07AgentIsai 03887716e - Initial setup of Gmail [04:29:01] PROBLEM - www.deadprem.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.deadprem.wiki could not be found [04:33:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.91, 6.28, 6.79 [04:34:34] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [04:35:06] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 3.05, 1.75 [04:36:08] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 10.38, 12.02, 9.30 [04:36:29] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.345 second response time [04:37:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.80, 2.54, 1.72 [04:38:02] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.54, 10.08, 8.91 [04:38:19] RECOVERY - swiftobject122 NTP time on swiftobject122 is OK: NTP OK: Offset 0.09598490596 secs [04:42:33] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3478c7e93606...a24d5cd83429 [04:42:35] [02miraheze/puppet] 07AgentIsai 03a24d5cd - Replace SSH key for self [04:47:09] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.04, 6.94, 6.74 [04:52:58] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.25, 6.51, 6.61 [05:04:46] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:05:47] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.32, 7.36, 6.85 [05:06:19] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:07:43] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.93, 7.47, 6.95 [05:13:33] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.30, 6.50, 6.73 [05:18:53] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/887716ee4617...3fd0d470a542 [05:18:56] [02miraheze/dns] 07AgentIsai 033fd0d47 - Add site verification for miraheze.org [05:21:39] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/3fd0d470a542...48ca6579bcb5 [05:21:41] [02miraheze/dns] 07AgentIsai 0348ca657 - Disable mail121 momentarily [05:22:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.97, 6.93, 6.87 [05:23:14] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/48ca6579bcb5...b6fdf2d4a5b5 [05:23:15] [02miraheze/dns] 07AgentIsai 03b6fdf2d - Add DKIM key for miraheze.org [05:25:15] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b6fdf2d4a5b5...92ae7f7205fa [05:25:16] [02miraheze/dns] 07AgentIsai 0392ae7f7 - Update SPF record for miraheze.org [05:30:06] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.73, 6.08, 6.59 [05:34:01] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.66, 6.82, 6.77 [05:37:54] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.64, 6.62, 6.73 [05:47:40] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.58, 7.15, 6.82 [05:49:36] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.57, 6.79, 6.72 [05:53:31] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.10, 7.57, 7.06 [05:59:20] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 5.19, 6.84, 6.98 [06:05:53] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [06:10:36] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:14:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.12, 6.15, 6.68 [06:15:43] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.27, 10.20, 8.59 [06:17:43] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.29, 9.46, 8.52 [06:18:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.28, 7.04, 6.95 [06:24:34] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:30:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.09, 6.45, 6.79 [06:37:39] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:38:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.27, 6.99, 6.95 [06:40:07] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [06:44:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.41, 6.11, 6.62 [06:48:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.91, 7.08, 6.94 [06:50:33] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:58:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.74, 6.49, 6.78 [07:00:33] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/effec13bdb82...a4d76ebc2d13 [07:00:34] [02miraheze/mw-config] 07AgentIsai 03a4d76eb - Update SMTP settings [07:01:16] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:01:30] miraheze/mw-config - AgentIsai the build passed. [07:01:32] !log [agent@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 16s [07:03:25] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [07:03:39] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [07:05:22] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.157 second response time [07:05:38] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.200 second response time [07:06:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=breadwiki --no-updates --username-prefix=wikia:the-bread-game /home/reception/ImportDump/breadwiki-20240108010847.xml (START) [07:06:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.41, 6.76, 6.81 [07:09:38] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a4d76ebc2d13...1c8b863e1ae0 [07:09:40] [02miraheze/mw-config] 07AgentIsai 031c8b863 - Try different IDHost [07:09:46] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:09:47] !log [agent@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [07:09:48] !log [@mwtask141] starting deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:09:59] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:10:03] !log [@mwtask141] finished deploy of {'config': True, 'versions': ['']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 15s [07:10:27] !log [agent@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 28s [07:10:31] miraheze/mw-config - AgentIsai the build passed. [07:10:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=breadwiki --no-updates --username-prefix=wikia:the-bread-game /home/reception/ImportDump/breadwiki-20240108010847.xml (END - exit=0) [07:10:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=breadwiki (START) [07:10:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.98, 6.58, 6.75 [07:12:13] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=breadwiki (END - exit=0) [07:12:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=breadwiki --active --update (END - exit=0) [07:16:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.75, 7.27, 6.96 [07:24:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.70, 3.86, 2.84 [07:26:18] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.75, 3.02, 2.65 [07:34:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.04, 7.09, 6.99 [07:36:13] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1c8b863e1ae0...9b670ac1436b [07:36:16] [02miraheze/mw-config] 07AgentIsai 039b670ac - Update SMTP settings [07:36:36] !log [agent@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] [07:36:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.64, 7.10, 7.00 [07:37:11] miraheze/mw-config - AgentIsai the build passed. [07:37:40] !log [agent@mwtask141] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mwtask141] - SUCCESS in 66s [07:39:02] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:41:25] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [07:43:22] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.145 second response time [07:47:36] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a24d5cd83429...22d314329573 [07:47:37] [02miraheze/puppet] 07AgentIsai 0322d3143 - Update Phabricator email settings [07:51:27] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22d314329573...64096cde7210 [07:51:29] [02miraheze/puppet] 07AgentIsai 0364096cd - Update matomo SMTP settings [07:52:47] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/64096cde7210...8a3f70f7888d [07:52:50] [02miraheze/puppet] 07AgentIsai 038a3f70f - Update Grafana mail settings [07:54:08] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/92ae7f7205fa...a81f15b0561f [07:54:09] [02miraheze/dns] 07AgentIsai 03a81f15b - Swap around MX priorities [07:54:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.90, 6.22, 6.74 [07:59:44] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.93, 1.71, 1.14 [08:01:44] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.34, 1.54, 1.15 [08:06:55] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.70, 2.99, 2.13 [08:08:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:08:55] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.98, 2.58, 2.08 [08:12:07] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8a3f70f7888d...de17d5b44dff [08:12:10] [02miraheze/puppet] 07AgentIsai 03de17d5b - Fix Phabricator email [08:15:27] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.02, 6.95, 6.66 [08:17:23] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.83, 7.23, 6.80 [08:23:13] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.20, 6.72, 6.72 [08:28:16] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [08:28:37] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:29:05] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.95, 6.80, 6.71 [08:30:16] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.521 second response time [08:30:35] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.631 second response time [08:31:02] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.63, 7.22, 6.87 [08:34:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.10, 7.27, 6.97 [08:36:49] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [08:36:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.49, 7.56, 7.10 [08:38:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.94, 7.14, 6.99 [08:42:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.91, 6.43, 6.73 [09:04:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.43, 7.19, 8.00 [09:06:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.04, 8.05, 8.22 [09:07:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:08:36] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:16:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 10.11, 7.87, 7.03 [09:22:13] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.20, 7.34, 7.09 [09:28:03] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.63, 6.34, 6.78 [09:36:52] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:42:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.26, 6.87, 6.73 [09:52:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.82, 6.53, 6.75 [10:06:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:16:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.25, 7.19, 6.71 [10:18:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.14, 6.90, 6.67 [10:22:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.79, 6.30, 6.52 [10:34:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.97, 7.01, 6.65 [10:35:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:36:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.42, 6.70, 6.57 [10:40:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.28, 7.53, 6.94 [10:42:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.35, 7.14, 6.88 [10:46:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.16, 7.67, 7.12 [10:48:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.59, 7.24, 7.03 [10:52:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.54, 7.57, 7.97 [10:54:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.10, 7.89, 8.04 [10:56:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.40, 7.64, 7.92 [11:00:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.20, 6.03, 6.59 [11:08:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.63, 7.74, 7.57 [11:12:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.80, 7.71, 7.60 [11:16:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.64, 7.39, 6.78 [11:18:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.49, 6.99, 6.70 [11:22:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.23, 7.18, 7.24 [11:22:51] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.99, 6.42, 6.60 [11:28:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.63, 7.53, 7.43 [11:28:46] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [11:32:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.73, 7.84, 7.51 [11:37:33] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 GMT +0000. [11:40:50] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.00, 7.04, 6.57 [11:42:51] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.45, 7.07, 6.63 [11:43:43] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.55, 8.70, 7.06 [11:44:30] PROBLEM - barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - barengreza.my.id All nameservers failed to answer the query. [11:44:50] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.47, 6.73, 6.56 [11:45:43] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.42, 8.42, 7.16 [11:55:43] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.27, 10.95, 8.87 [11:59:43] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.53, 10.53, 9.19 [12:02:26] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.50, 6.88, 6.58 [12:03:43] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.25, 8.83, 8.86 [12:04:56] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 2.91, 1.70 [12:06:23] RECOVERY - barengreza.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'barengreza.my.id' will expire on Sun 17 Mar 2024 17:33:19 GMT +0000. [12:06:58] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.70, 2.92, 1.86 [12:08:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.42, 7.33, 7.91 [12:08:15] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.34, 6.78, 6.63 [12:10:06] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.47, 7.64, 7.94 [12:12:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.35, 7.50, 7.86 [12:14:00] PROBLEM - barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for barengreza.my.id could not be found [12:14:18] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/MirahezeMagic/compare/f82691865182...81aff3441ea4 [12:14:20] [02miraheze/MirahezeMagic] 07translatewiki 0381aff34 - Localisation updates from https://translatewiki.net. [12:18:00] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.49, 7.52, 7.03 [12:22:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.12, 8.20, 7.91 [12:23:25] miraheze/MirahezeMagic - translatewiki the build passed. [12:23:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.18, 5.97, 6.55 [12:24:08] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.43, 7.84, 7.81 [12:26:56] !log [@test131] starting deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] [12:26:57] !log [@test131] finished deploy of {'folders': '1.41/extensions/MirahezeMagic', 'versions': ['']} to [test131] - SUCCESS in 0s [12:30:46] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:31:36] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.81, 6.71, 6.65 [12:32:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.94, 7.83, 7.76 [12:32:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.81, 7.28, 7.90 [12:34:55] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.09, 7.54, 7.91 [12:35:29] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 9.17, 7.60, 6.98 [12:36:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.86, 7.86, 7.78 [12:37:25] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.17, 7.41, 6.99 [12:38:05] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.31, 8.00, 7.84 [12:38:25]