[00:00:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.18, 7.99, 7.57 [00:00:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.32, 6.80, 6.92 [00:01:17] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:02:47] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.72, 6.72, 6.88 [00:02:50] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.94, 8.02, 3.55 [00:04:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 7.75, 7.59 [00:05:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.53, 7.48, 6.83 [00:06:47] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.32, 6.57, 6.78 [00:06:53] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.05, 3.10, 1.77 [00:07:38] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:08:14] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.48, 6.87, 6.40 [00:08:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 3.69, 2.52 [00:09:28] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.64, 7.16, 6.89 [00:09:34] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.60, 10.57, 7.50 [00:09:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.59, 11.00, 7.44 [00:09:51] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [00:10:02] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.96, 3.50, 2.80 [00:10:04] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:10:10] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.82, 6.39, 6.27 [00:10:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.93, 9.62, 7.10 [00:11:34] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.67, 10.08, 7.74 [00:11:43] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.56, 10.83, 7.90 [00:11:43] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.527 second response time [00:11:49] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.683 second response time [00:12:02] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.17, 2.98, 2.69 [00:12:14] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.38, 7.76, 7.53 [00:12:14] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.68, 8.44, 6.95 [00:12:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.45, 3.84, 2.88 [00:12:57] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.17, 3.99, 2.75 [00:13:39] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.81, 9.68, 7.81 [00:13:55] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.669 second response time [00:14:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.13, 4.37, 3.19 [00:15:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.78, 6.51, 6.71 [00:15:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.43, 7.38, 6.99 [00:16:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.73, 7.84, 7.62 [00:16:55] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.66, 3.36, 2.78 [00:17:49] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.19, 7.20, 6.97 [00:18:50] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.50, 7.30, 7.53 [00:19:51] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.55, 6.58, 6.77 [00:22:27] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to [mwtask181] [00:22:29] !log [universalomega@mwtask181] finished deploy of {'config': True, 'versions': 'all'} to [mwtask181] - SUCCESS in 1s [00:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.68, 3.68, 3.36 [00:22:51] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 5.88, 5.35, 6.60 [00:23:51] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.69, 6.88, 6.89 [00:25:00] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [00:25:52] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.52, 6.41, 6.72 [00:26:10] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': 'all'} to [mwtask181] [00:26:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.14, 3.29, 3.31 [00:27:06] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [00:27:09] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 2s [00:27:11] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': 'all'} to [mwtask181] - SUCCESS in 61s [00:27:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.38, 7.51, 7.40 [00:28:43] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_extensions': ['CreateWiki', 'DataDump', 'GlobalNewFiles', 'ImportDump', 'IncidentReporting', 'ManageWiki', 'MatomoAnalytics', 'MirahezeMagic', 'PDFEmbed', 'RemovePII', 'RottenLinks', 'SpriteSheet', 'WikiDiscover', 'YouTube'], 'upgrade_skins': []} to [mwtask181] [00:28:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:02] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_extensions': ['CreateWiki', 'DataDump', 'GlobalNewFiles', 'ImportDump', 'IncidentReporting', 'ManageWiki', 'MatomoAnalytics', 'MirahezeMagic', 'PDFEmbed', 'RemovePII', 'RottenLinks', 'SpriteSheet', 'WikiDiscover', 'YouTube'], 'upgrade_skins': []} to [mwtask181] - SUCCESS in 19s [00:29:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.11, 7.33, 7.35 [00:31:42] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_extensions': ['CreateWiki', 'DataDump', 'GlobalNewFiles', 'ImportDump', 'IncidentReporting', 'ManageWiki', 'MatomoAnalytics', 'MirahezeMagic', 'PDFEmbed', 'RemovePII', 'RottenLinks', 'SpriteSheet', 'WikiDiscover', 'YouTube'], 'upgrade_skins': []} to [mwtask181] [00:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:00] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_extensions': ['CreateWiki', 'DataDump', 'GlobalNewFiles', 'ImportDump', 'IncidentReporting', 'ManageWiki', 'MatomoAnalytics', 'MirahezeMagic', 'PDFEmbed', 'RemovePII', 'RottenLinks', 'SpriteSheet', 'WikiDiscover', 'YouTube'], 'upgrade_skins': []} to [mwtask181] - SUCCESS in 17s [00:32:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:08] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.91, 6.58, 6.52 [00:33:17] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:33:23] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:33:32] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_extensions': ['CreateWiki', 'DataDump', 'GlobalNewFiles', 'ImportDump', 'IncidentReporting', 'ManageWiki', 'MatomoAnalytics', 'MirahezeMagic', 'PDFEmbed', 'RemovePII', 'RottenLinks', 'SpriteSheet', 'WikiDiscover', 'YouTube'], 'upgrade_skins': []} to [mwtask181] [00:33:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:09] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mwtask181] [00:36:30] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mwtask181] - SUCCESS in 20s [00:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:53] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.13, 6.84, 6.80 [00:37:13] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mwtask181] [00:37:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:31] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mwtask181] - SUCCESS in 18s [00:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:48] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mw181] [00:37:49] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to [mw181] - SUCCESS in 0s [00:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:04] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.83, 6.82, 7.97 [00:38:13] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to all [00:38:14] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.08, 5.78, 6.61 [00:38:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:31] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to all - SUCCESS in 18s [00:38:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:46] !log [universalomega@mwtask181] starting deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to all [00:38:53] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.39, 3.94, 3.19 [00:38:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.01, 6.29, 6.60 [00:38:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:04] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_pack': 'miraheze'} to all - SUCCESS in 18s [00:39:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:33] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to all [00:39:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:05] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 3.56, 5.70, 6.32 [00:40:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 3.75, 3.32 [00:40:59] !log [universalomega@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.wikitide.net1 [00:41:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:41:37] nice logging seems to still work (and better) [00:42:02] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.51, 2.87 [00:42:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.48, 3.20, 3.17 [00:43:37] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.22, 8.71, 7.10 [00:44:02] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.10, 2.98, 2.76 [00:44:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/66ec6188f442...dd7538912d22 [00:44:19] [02miraheze/puppet] 07Universal-Omega 03dd75389 - Update [00:44:21] [02puppet] 07Universal-Omega synchronize pull request 03#3629: mwdeploy: improve logging - 13https://github.com/miraheze/puppet/pull/3629 [00:44:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 3.75, 5.68, 7.58 [00:45:35] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.40, 8.25, 7.11 [00:45:40] miraheze/puppet - Universal-Omega the build passed. [00:46:53] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.57, 3.86, 3.55 [00:50:04] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.22, 5.77, 6.74 [00:50:53] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.64, 3.24, 3.37 [00:54:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.32, 5.81, 6.79 [00:54:34] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [00:54:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 3.37, 3.10 [00:56:51] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.84, 3.21, 3.09 [00:58:22] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [00:58:23] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 0s [00:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.42, 3.54, 3.25 [01:01:17] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [01:01:22] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:02:51] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.87, 3.16, 3.14 [01:06:06] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.35, 1.85, 1.07 [01:08:24] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to [all] [01:08:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:50] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:09:38] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': ['all']} to [all] [01:09:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:10:48] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.517 second response time [01:11:15] !log [universalomega@mwtask181] starting deploy of {'versions': ['all'], 'upgrade_extensions': 'all'} to [mwtask181] [01:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:38] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 4.13, 3.58 [01:13:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.64, 3.85, 3.54 [01:16:54] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.20, 6.29, 5.65 [01:16:59] [02puppet] 07Universal-Omega synchronize pull request 03#3629: mwdeploy: improve logging - 13https://github.com/miraheze/puppet/pull/3629 [01:17:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd7538912d22...eb49c538c408 [01:17:04] [02miraheze/puppet] 07Universal-Omega 03eb49c53 - Update [01:18:24] miraheze/puppet - Universal-Omega the build passed. [01:18:34] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to mwtask181 [01:18:36] !log [universalomega@mwtask181] finished deploy of {'config': True, 'versions': 'all'} to mwtask181 - SUCCESS in 1s [01:18:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:54] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.50, 5.80, 5.54 [01:18:56] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to all [01:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:24] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.54, 2.76, 3.21 [01:19:42] !log [universalomega@mwtask181] starting deploy of {'config': True, 'versions': 'all'} to [mw181, mw182] [01:19:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:50] !log [universalomega@mwtask181] finished deploy of {'config': True, 'versions': 'all'} to [mw181, mw182] - SUCCESS in 7s [01:19:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:15] [02puppet] 07Universal-Omega synchronize pull request 03#3629: mwdeploy: improve logging - 13https://github.com/miraheze/puppet/pull/3629 [01:25:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwdeploy-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eb49c538c408...1a9d42b253af [01:25:18] [02miraheze/puppet] 07Universal-Omega 031a9d42b - Don't always use version [01:25:59] !log [universalomega@mwtask181] starting deploy of {'config': True} to [mw181, mw182] [01:26:04] !log [universalomega@mwtask181] finished deploy of {'config': True} to [mw181, mw182] - SUCCESS in 5s [01:26:11] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:26:28] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:26:43] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:26:50] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:27:03] miraheze/puppet - Universal-Omega the build passed. [01:27:06] !log [@test131] starting deploy of {'config': True, 'versions': ['']} to [test131] [01:27:07] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.222 second response time [01:27:07] !log [@test131] finished deploy of {'config': True, 'versions': ['']} to [test131] - SUCCESS in 1s [01:27:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:40] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': 'all'} to mwtask181 [01:27:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:06] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.246 second response time [01:28:22] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.666 second response time [01:28:40] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.584 second response time [01:28:40] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': 'all'} to mwtask181 - SUCCESS in 60s [01:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:48] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.536 second response time [01:29:08] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.708 second response time [01:29:25] [02puppet] 07Universal-Omega closed pull request 03#3629: mwdeploy: improve logging - 13https://github.com/miraheze/puppet/pull/3629 [01:29:28] [02miraheze/puppet] 07Universal-Omega deleted branch 03mwdeploy-logging [01:29:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/306efe4eafbb...0db00cda2af2 [01:29:31] [02miraheze/puppet] 07Universal-Omega 030db00cd - mwdeploy: improve logging (#3629) [01:29:33] [02puppet] 07Universal-Omega deleted branch 03mwdeploy-logging - 13https://github.com/miraheze/puppet [01:30:44] miraheze/puppet - Universal-Omega the build passed. [01:33:50] [02puppet] 07Universal-Omega edited pull request 03#3629: mwdeploy: improve logging - 13https://github.com/miraheze/puppet/pull/3629 [01:39:45] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.17, 4.16, 3.34 [01:41:40] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.87, 3.44, 3.19 [01:43:35] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.47, 2.40, 2.84 [01:52:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0db00cda2af2...6bcd711d6d09 [01:52:12] [02miraheze/puppet] 07Universal-Omega 036bcd711 - Enable email notifications for myself [02:01:08] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [02:01:44] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:02:30] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:03:03] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [02:03:12] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [02:03:55] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:03:55] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [02:04:00] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [02:04:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw132 mw141 mw133 mw134 mw143 [02:04:19] !log [universalomega@test131] starting deploy of {'config': True} to all [02:04:20] !log [universalomega@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:04:27] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.521 second response time [02:04:36] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw131 mw141 mw133 mw143 [02:04:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:09] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.522 second response time [02:05:17] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.509 second response time [02:05:43] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 17 backends are healthy [02:05:43] !log [@test131] starting deploy of {'config': True} to all [02:05:44] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [02:05:53] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.582 second response time [02:05:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:36] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 17 backends are healthy [02:07:36] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 6.15, 4.40, 3.09 [02:07:57] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:08:08] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.860 second response time [02:09:32] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.66, 3.91, 3.06 [02:11:32] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.81, 4.14, 3.24 [02:13:33] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.08, 3.33, 3.06 [02:15:37] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.27, 1.58, 1.96 [02:23:26] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.14, 1.26, 1.64 [02:31:02] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [02:31:40] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [02:38:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6bcd711d6d09...0adce5cc15d8 [02:38:40] [02miraheze/puppet] 07Universal-Omega 030adce5c - Use workspace for postfix relayhost [02:43:09] !log [@test131] starting deploy of {'config': True} to all [02:43:10] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:43:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:58] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [02:44:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.97, 6.77, 6.16 [02:45:09] CUSTOM - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures universalomega test [02:46:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.24, 6.18, 6.02 [02:46:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.97, 6.50, 6.14 [02:50:11] CUSTOM - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures universalomega test [02:50:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.17, 6.99, 6.41 [02:52:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.56, 6.34, 6.24 [03:00:18] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [03:01:41] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [03:17:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0adce5cc15d8...e3d3665de3fd [03:17:57] [02miraheze/puppet] 07Universal-Omega 03e3d3665 - mwdeploy: fix [03:19:14] miraheze/puppet - Universal-Omega the build passed. [03:20:25] !log [@test131] starting deploy of {'config': True} to all [03:20:26] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:20:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:21:40] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [03:27:05] !log [@test131] starting deploy of {'config': True} to all [03:27:06] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:27:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:04] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [03:56:58] !log [@test131] starting deploy of {'config': True} to all [03:56:59] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:57:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:22:02] PROBLEM - eurocom.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eurocom.wiki could not be found [04:25:42] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5b3f47f8a153...a8f2889eaf3b [04:25:43] [02miraheze/dns] 07Universal-Omega 03a8f2889 - Update mail records [04:27:18] !log [@test131] starting deploy of {'config': True} to all [04:27:19] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [04:27:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:27:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:32:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:51:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-5/±0] 13https://github.com/miraheze/puppet/commit/41430377b365 [04:51:50] [02miraheze/puppet] 07Universal-Omega 034143037 - Remove roundcubemail [04:51:52] [02puppet] 07Universal-Omega created branch 03remove-roundcubemail - 13https://github.com/miraheze/puppet [04:51:54] [02puppet] 07Universal-Omega opened pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:52:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/41430377b365...8c8e1a5fb5d9 [04:52:42] [02miraheze/puppet] 07Universal-Omega 038c8e1a5 - - [04:52:43] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:53:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/8c8e1a5fb5d9...d848fad96e89 [04:53:12] [02miraheze/puppet] 07Universal-Omega 03d848fad - - [04:53:13] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:54:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d848fad96e89...7246ede9890e [04:54:24] [02miraheze/puppet] 07Universal-Omega 037246ede - - [04:54:26] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:55:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7246ede9890e...4e664f557741 [04:55:11] [02miraheze/puppet] 07Universal-Omega 034e664f5 - - [04:55:13] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:55:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4e664f557741...5e519bf11ad0 [04:55:52] [02miraheze/puppet] 07Universal-Omega 035e519bf - - [04:55:55] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:56:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e519bf11ad0...3cd700dec2b8 [04:56:38] [02miraheze/puppet] 07Universal-Omega 033cd700d - - [04:56:41] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:57:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3cd700dec2b8...1f8b3b02c984 [04:57:22] !log [@test131] starting deploy of {'config': True} to all [04:57:22] [02miraheze/puppet] 07Universal-Omega 031f8b3b0 - -backend [04:57:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [04:57:24] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:57:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1f8b3b02c984...ad422d95b4ea [04:57:53] [02miraheze/puppet] 07Universal-Omega 03ad422d9 - -role [04:57:54] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [04:58:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-roundcubemail [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ad422d95b4ea...01d900501624 [04:58:46] [02miraheze/puppet] 07Universal-Omega 0301d9005 - -webmail [04:58:49] [02puppet] 07Universal-Omega synchronize pull request 03#3632: Remove roundcubemail - 13https://github.com/miraheze/puppet/pull/3632 [05:01:06] RECOVERY - db182 Backups SQL icingaweb2 on db182 is OK: FILE_AGE OK: /var/log/sql-icingaweb2-backup-fortnightly.log is 54 seconds old and 125 bytes [05:01:37] RECOVERY - db182 Backups SQL roundcubemail on db182 is OK: FILE_AGE OK: /var/log/sql-roundcubemail-backup-fortnightly.log is 85 seconds old and 219 bytes [05:01:41] RECOVERY - db182 Backups SQL icinga on db182 is OK: FILE_AGE OK: /var/log/sql-icinga-backup-fortnightly.log is 24 seconds old and 120 bytes [05:03:01] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [05:05:06] PROBLEM - eurocom.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address eurocom.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:27:28] !log [@test131] starting deploy of {'config': True} to all [05:27:29] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:27:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:01] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:32:04] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [05:33:14] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [05:37:13] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [05:57:20] !log [@test131] starting deploy of {'config': True} to all [05:57:21] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:03:00] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [06:23:47] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:27:18] !log [@test131] starting deploy of {'config': True} to all [06:27:20] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 2s [06:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:00] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:51:55] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:57:33] !log [@test131] starting deploy of {'config': True} to all [06:57:34] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:04:59] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:19:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:26:39] !log [@test131] starting deploy of {'config': True} to all [07:26:40] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [07:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:31:29] RECOVERY - eurocom.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'eurocom.wiki' will expire on Fri 29 Mar 2024 04:19:54 PM GMT +0000. [07:32:04] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:32:59] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:48:06] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [07:57:06] !log [@test131] starting deploy of {'config': True} to all [07:57:07] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:57:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:00:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:02:42] PROBLEM - eurocom.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address eurocom.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:09:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 6.85, 6.59 [08:11:09] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.46, 6.79, 6.61 [08:25:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.86, 7.99, 6.96 [08:26:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.62, 7.22, 6.72 [08:26:36] !log [@test131] starting deploy of {'config': True} to all [08:26:37] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [08:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:28:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.14, 8.11, 7.09 [08:34:59] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [08:36:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.36, 7.38, 6.36 [08:38:34] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.88, 6.85, 6.29 [08:44:21] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.25, 6.35, 6.31 [08:44:34] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.51, 6.84, 6.31 [08:52:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.58, 6.66, 6.40 [08:54:04] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.43, 6.56, 6.40 [08:56:36] !log [@test131] starting deploy of {'config': True} to all [08:56:37] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [08:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:00:08] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.26, 6.60, 6.63 [09:00:58] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [09:02:04] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [09:04:04] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.41, 6.91, 6.73 [09:06:01] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.54, 6.77, 6.71 [09:07:45] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.27, 7.88, 6.87 [09:09:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.52, 6.61, 6.17 [09:09:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.14, 7.70, 7.11 [09:11:33] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.51, 6.96, 6.33 [09:13:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.73, 6.42, 6.21 [09:13:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 5.72, 7.43, 7.06 [09:15:44] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.24, 6.59, 6.79 [09:19:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 4.75, 7.07, 7.41 [09:21:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.40, 7.11, 6.48 [09:22:40] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.06, 7.36, 7.02 [09:23:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 7.02, 6.52 [09:24:36] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.30, 6.87, 6.88 [09:26:44] !log [@test131] starting deploy of {'config': True} to all [09:26:45] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [09:26:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.36, 6.73, 6.51 [09:28:28] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 4.16, 5.82, 6.49 [09:30:04] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [09:35:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.59, 7.21, 7.15 [09:36:11] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.56, 6.91, 6.61 [09:37:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.34, 6.86, 7.03 [09:39:45] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.08, 7.37, 7.20 [09:40:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.54, 7.62, 6.99 [09:41:45]