[00:01:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.01, 8.43, 8.13 [00:02:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 6.85, 6.65 [00:03:49] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.77, 7.75, 3.59 [00:04:17] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:04:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.55, 6.52, 6.55 [00:09:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.73, 7.70, 8.00 [00:09:20] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:09:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/e83512cda280...81291b9b9d39 [00:09:41] [02miraheze/ssl] 07MirahezeSSLBot 0381291b9 - Bot: Add SSL cert for kinnikupedia.org [00:09:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/81291b9b9d39...2c3655292ff6 [00:09:58] [02miraheze/ssl] 07MirahezeSSLBot 032c36552 - Bot: Add SSL cert for kinnikupedia.org [00:10:15] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:10:33] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:10:35] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:10:39] miraheze/ssl - MirahezeSSLBot the build has errored. [00:10:48] miraheze/ssl - MirahezeSSLBot the build has errored. [00:10:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:11:08] [02miraheze/ssl] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2c3655292ff6...5b9d6f1c5233 [00:11:09] [02miraheze/ssl] 07OAuthority 035b9d6f1 - no idea what went on there [00:11:17] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:12:04] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:12:17] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.545 second response time [00:12:29] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildtextindex.php --wiki=trollpastawiki (END - exit=0) [00:12:31] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.603 second response time [00:12:33] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.693 second response time [00:12:35] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.65, 3.20, 2.21 [00:12:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:47] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.70, 5.56, 3.45 [00:12:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.552 second response time [00:13:12] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [00:13:40] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.78, 6.45, 5.69 [00:13:59] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.714 second response time [00:13:59] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:14:35] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.15, 3.21, 2.33 [00:14:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.05, 5.33, 3.62 [00:18:44] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw181.wikitide.net port 443 after 0 ms: Couldn't connect to server [00:18:58] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw181 [00:19:00] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: connect to address 2602:294:0:b12::105 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:19:01] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw181 [00:19:14] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw181 [00:19:27] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw181 [00:19:49] PROBLEM - txt-dex.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - txt-dex.com All nameservers failed to answer the query. [00:20:35] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.53, 3.34, 2.65 [00:22:35] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.16, 3.02, 2.62 [00:24:44] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 200 - 320 bytes in 0.008 second response time [00:24:56] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 17 backends are healthy [00:25:01] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 17 backends are healthy [00:25:14] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 17 backends are healthy [00:25:27] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 17 backends are healthy [00:26:45] PROBLEM - cp34 Puppet on cp34 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:26:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.85, 8.21, 5.73 [00:27:12] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.256 second response time [00:28:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.49, 6.82, 5.52 [00:29:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.20, 7.61, 7.52 [00:30:22] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:30:35] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:30:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.25, 5.63, 5.25 [00:31:02] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:32:04] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:33] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:32:40] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:34:05] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:35:43] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:37:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.19, 7.74, 7.67 [00:39:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.33, 7.75, 7.67 [00:39:13] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:39:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ssl/compare/5b9d6f1c5233...7cb823a511a5 [00:39:22] [02miraheze/ssl] 07MirahezeSSLBot 037cb823a - Bot: Add SSL cert for kinnikupedia.org [00:39:28] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:39:42] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:40:10] miraheze/ssl - MirahezeSSLBot the build has errored. [00:40:13] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:40:15] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7cb823a511a5...d47d432097aa [00:40:16] [02miraheze/ssl] 07Universal-Omega 03d47d432 - - [00:41:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 7.52, 7.60 [00:41:09] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:41:54] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.74, 3.50, 2.81 [00:42:26] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:43:31] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:43:49] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.601 second response time [00:43:57] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.29, 3.39, 2.86 [00:45:13] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/d47d432097aa...2ea31eaff434 [00:45:16] [02miraheze/ssl] 07Universal-Omega 032ea31ea - Remove [00:45:33] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2ea31eaff434...7eccb88b504c [00:45:34] [02miraheze/ssl] 07Universal-Omega 037eccb88 - Remove [00:46:37] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:48:00] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.69, 3.59, 3.06 [00:48:37] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:49:59] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.55, 3.17, 2.97 [00:50:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 6.82, 6.36 [00:51:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.03, 7.76, 7.62 [00:52:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.45, 6.28 [00:53:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.23, 7.42, 7.51 [00:54:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7eccb88b504c...ad4c94507fe1 [00:54:55] [02miraheze/ssl] 07MirahezeSSLBot 03ad4c945 - Bot: Add SSL cert for kinnikupedia.org [00:56:45] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:58:22] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:58:43] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:59:10] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:04] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [01:00:40] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:01:57] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [01:02:33] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:03:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.13, 7.57, 7.49 [01:03:43] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:03:47] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [01:03:56] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.580 second response time [01:06:37] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:06:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [01:07:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.92, 7.87, 7.64 [01:07:18] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:07:31] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:08:22] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:08:37] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.755 second response time [01:08:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.557 second response time [01:09:09] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:09:45] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.27, 11.11, 8.70 [01:09:52] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.90, 3.62, 3.25 [01:10:17] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [01:11:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.07, 8.68, 7.99 [01:11:28] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:45] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.18, 9.41, 8.37 [01:11:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.52, 3.26, 3.17 [01:12:00] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:12:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.05, 6.63 [01:14:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.22, 6.56, 6.52 [01:16:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.79, 7.23, 6.03 [01:16:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/da59d73570b0...4def352bb32a [01:16:46] [02miraheze/puppet] 07Universal-Omega 034def352 - Pull puppet and ssl repos when running puppet [01:18:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.55, 7.01, 6.10 [01:20:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03srv-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ff107e5ad9b4 [01:20:38] [02miraheze/puppet] 07Universal-Omega 03ff107e5 - Puppetize /srv/ssl/ssl [01:20:38] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.03, 6.51, 6.01 [01:20:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.76, 6.87, 6.67 [01:20:40] [02puppet] 07Universal-Omega created branch 03srv-ssl - 13https://github.com/miraheze/puppet [01:20:42] [02puppet] 07Universal-Omega opened pull request 03#3625: Puppetize /srv/ssl/ssl - 13https://github.com/miraheze/puppet/pull/3625 [01:21:07] [02puppet] 07Universal-Omega synchronize pull request 03#3625: Puppetize /srv/ssl/ssl - 13https://github.com/miraheze/puppet/pull/3625 [01:21:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03srv-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff107e5ad9b4...a3ec79394bd1 [01:21:09] [02miraheze/puppet] 07Universal-Omega 03a3ec793 - Fix [01:25:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.79, 7.89, 7.95 [01:26:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.25, 7.54, 7.00 [01:27:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.77, 8.19, 8.05 [01:28:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.41, 7.01 [01:29:31] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [01:29:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:07] [02puppet] 07Universal-Omega closed pull request 03#3625: Puppetize /srv/ssl/ssl - 13https://github.com/miraheze/puppet/pull/3625 [01:31:09] [02miraheze/puppet] 07Universal-Omega deleted branch 03srv-ssl [01:31:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4def352bb32a...2f016bfe1be6 [01:31:12] [02miraheze/puppet] 07Universal-Omega 032f016bf - Puppetize /srv/ssl/ssl (#3625) [01:31:15] [02puppet] 07Universal-Omega deleted branch 03srv-ssl - 13https://github.com/miraheze/puppet [01:32:40] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [01:39:17] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl] [01:39:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2f016bfe1be6...ed9d6e7f136d [01:39:35] [02miraheze/puppet] 07Universal-Omega 03ed9d6e7 - Change order and comment for now [01:41:17] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [01:43:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03proxy [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/843db3914f06 [01:43:59] [02miraheze/puppet] 07Universal-Omega 03843db39 - Uncomment and add ProxyCommand [01:44:01] [02puppet] 07Universal-Omega created branch 03proxy - 13https://github.com/miraheze/puppet [01:44:02] [02puppet] 07Universal-Omega opened pull request 03#3626: Uncomment and add ProxyCommand - 13https://github.com/miraheze/puppet/pull/3626 [01:44:09] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.83, 6.99, 6.45 [01:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.12, 7.80, 7.24 [01:45:21] [02puppet] 07Universal-Omega closed pull request 03#3626: Uncomment and add ProxyCommand - 13https://github.com/miraheze/puppet/pull/3626 [01:45:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ed9d6e7f136d...17b209944e1e [01:45:25] [02miraheze/puppet] 07Universal-Omega 0317b2099 - Uncomment and add ProxyCommand (#3626) [01:45:27] [02miraheze/puppet] 07Universal-Omega deleted branch 03proxy [01:45:28] [02puppet] 07Universal-Omega deleted branch 03proxy - 13https://github.com/miraheze/puppet [01:45:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.96, 5.66, 5.16 [01:47:55] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.55, 5.72, 5.26 [01:48:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.32, 7.52, 7.26 [01:49:13] PROBLEM - txt-dex.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [01:49:57] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.67, 7.38, 6.92 [01:50:10] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=2) [01:50:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:32] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (START) [01:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:50:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.18, 9.04, 7.83 [01:50:58] ^rerunning in screen. [01:51:35] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 6.98, 6.31 [01:51:38] !log (rerunning in screen) [01:51:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:53:35] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.60, 6.41, 6.18 [01:55:45] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.21, 6.73, 6.79 [01:56:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.15, 7.92, 7.73 [01:58:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 7.97, 7.77 [02:00:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/17b209944e1e...73cfea6afba9 [02:00:41] [02miraheze/puppet] 07Universal-Omega 0373cfea6 - Move [02:00:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.77, 7.73 [02:01:18] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [02:02:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 8.39, 7.98 [02:04:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.91, 7.63, 7.76 [02:04:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.55, 6.02, 4.78 [02:06:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/73cfea6afba9...68a099181da4 [02:06:41] [02miraheze/puppet] 07Universal-Omega 0368a0991 - mediawiki: remove mwupgradetool [02:06:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.06, 7.81, 7.80 [02:08:02] miraheze/puppet - Universal-Omega the build passed. [02:08:47] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.34, 6.42, 5.24 [02:08:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/68a099181da4...40558e02c3c1 [02:08:53] [02miraheze/puppet] 07Universal-Omega 0340558e0 - Remove unused grafana_db_host [02:09:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/40558e02c3c1...55b8fa2a69a8 [02:09:46] [02miraheze/puppet] 07Universal-Omega 0355b8fa2 - Remove unused grafana_password [02:12:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.67, 9.23, 6.76 [02:12:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:13:01] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:13:08] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [02:14:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/55b8fa2a69a8...7c7bbf116bf0 [02:14:21] [02miraheze/puppet] 07Universal-Omega 037c7bbf1 - icingaweb2: update default DB [02:14:22] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.41, 7.21, 6.82 [02:14:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.526 second response time [02:14:58] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.519 second response time [02:15:08] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.585 second response time [02:15:56] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.64, 6.37, 5.72 [02:16:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c7bbf116bf0...78fa46060f2e [02:16:05] [02miraheze/puppet] 07Universal-Omega 0378fa460 - icinga2: update default DB [02:16:28] [02miraheze/puppet] 07Universal-Omega pushed 032 commits to 03mariadb-remove-ssl [+0/-0/±17] 13https://github.com/miraheze/puppet/compare/02b79fedc2ac...749ad82776fa [02:16:29] [02miraheze/puppet] 07Universal-Omega 03d509187 - mariadb: remove SSL [02:16:30] [02miraheze/puppet] 07github-actions 03749ad82 - CI: lint puppet code to standards [02:16:31] [02puppet] 07Universal-Omega synchronize pull request 03#3582: mariadb: remove SSL - 13https://github.com/miraheze/puppet/pull/3582 [02:17:56] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.56, 6.44, 5.83 [02:18:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.49, 7.52, 7.10 [02:18:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.49, 7.92, 7.00 [02:19:01] PROBLEM - txt-dex.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - txt-dex.com All nameservers failed to answer the query. [02:20:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 13.60, 10.20, 7.95 [02:21:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78fa46060f2e...459b7321b125 [02:21:33] [02miraheze/puppet] 07Universal-Omega 03459b732 - - [02:23:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/459b7321b125...f891f743ba37 [02:23:08] [02miraheze/puppet] 07Universal-Omega 03f891f74 - Remove unused puppet_major_version [02:23:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f891f743ba37...560b351abf8e [02:23:59] [02miraheze/puppet] 07Universal-Omega 03560b351 - Default puppet_major_version to 8 [02:25:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/560b351abf8e...798b8bdae05e [02:25:46] [02miraheze/puppet] 07Universal-Omega 03798b8bd - Default puppet_major_version to 8 [02:25:58] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.82, 6.31, 6.68 [02:30:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 7.39, 7.87 [02:34:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.77, 7.90 [02:36:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.27, 6.76, 7.89 [02:36:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 7.63, 7.84 [02:38:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.68, 8.31, 8.05 [02:40:04] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:40:47] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.96, 3.57, 6.31 [02:40:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [02:41:59] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.295 second response time [02:42:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.550 second response time [02:58:31] RECOVERY - deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [03:00:40] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:01:25] RECOVERY - db101 Backups SQL on db101 is OK: FILE_AGE OK: /var/log/sql-backup.log is 83 seconds old and 0 bytes [03:02:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.44, 7.86 [03:07:18] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.68, 6.65, 5.85 [03:09:17] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.30, 6.66, 5.96 [03:16:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 7.44, 7.55 [03:18:38] PROBLEM - txt-dex.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [03:18:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 7.61, 7.61 [03:20:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.21, 9.16, 8.22 [03:21:09] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.38, 6.68, 6.06 [03:21:31] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.02, 6.91, 6.60 [03:23:07] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.16, 6.81, 6.19 [03:23:27] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.28, 6.78, 6.60 [03:25:06] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.06, 6.46, 6.13 [03:30:31] PROBLEM - deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:33:17] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [03:35:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.39, 7.92 [03:40:35] PROBLEM - ping6 on cloud10 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 129.01 ms [03:41:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.16, 8.01, 7.91 [03:42:35] PROBLEM - ping6 on cloud10 is WARNING: PING WARNING - Packet loss = 0%, RTA = 128.67 ms [03:43:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.89, 8.00, 7.93 [03:45:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.42, 8.44, 8.10 [03:47:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.43, 7.62, 7.84 [03:51:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.12, 6.61, 7.83 [03:53:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.58, 6.99, 7.80 [03:53:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.88, 7.86 [03:55:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.50, 6.80, 7.63 [03:55:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.50, 7.72 [03:59:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.06, 7.75, 7.82 [04:01:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.33, 7.51, 7.72 [04:01:17] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [04:05:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.86, 8.14, 7.91 [04:05:06] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [04:06:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.11, 6.81, 5.99 [04:07:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.93, 7.75, 7.80 [04:08:47] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 4.64, 6.04, 5.81 [04:15:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.73, 7.87, 7.73 [04:15:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.77, 7.55 [04:17:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.90, 7.04, 7.44 [04:17:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.32, 7.52, 7.47 [04:18:15] PROBLEM - txt-dex.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - txt-dex.com All nameservers failed to answer the query. [04:19:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.87, 7.98, 7.64 [04:23:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.78, 7.97, 7.66 [04:25:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.00, 7.26, 7.44 [04:25:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.99, 7.09, 7.50 [04:27:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.66, 7.81, 7.71 [04:29:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.56, 7.60, 7.64 [04:31:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.38, 7.14, 7.27 [04:31:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.71, 7.67 [04:32:40] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [04:33:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.87, 6.80, 7.14 [04:33:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.48, 7.37, 7.55 [04:35:40] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:37:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.22, 7.48, 7.35 [04:37:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.15, 7.60, 7.55 [04:37:30] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.71, 6.86, 6.10 [04:39:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.64, 6.84, 7.13 [04:39:29] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.62, 6.34, 6.00 [04:41:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.44, 7.15, 7.38 [04:49:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.98, 7.65, 7.22 [04:49:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.84, 8.18, 7.55 [04:51:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.73, 7.24, 7.12 [04:51:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 7.61, 7.42 [04:51:11] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 3.37, 1.80, 1.01 [04:53:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.05, 7.52, 7.23 [04:53:07] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.15, 1.52, 1.00 [04:55:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.58, 8.29, 7.67 [04:57:10] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 3.495 second response time [04:57:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 7.67, 7.52 [04:57:26] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:58:42] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.62, 10.88, 6.90 [04:58:59] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [04:59:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.81, 7.56, 7.37 [04:59:07] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.630 second response time [04:59:10] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.66, 9.01, 6.09 [04:59:26] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.833 second response time [05:00:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.76, 12.10, 8.64 [05:00:38] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.81, 10.11, 7.13 [05:00:40] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.18, 10.53, 7.93 [05:00:55] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.547 second response time [05:01:08] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.31, 8.31, 6.22 [05:02:32] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.30, 9.79, 8.21 [05:02:36] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.16, 9.69, 7.94 [05:03:17] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [05:04:20] RECOVERY - www.deadprem.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'deadprem.wiki' will expire on Wed 31 Jan 2024 12:16:06 PM GMT +0000. [05:06:17] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.57, 6.26, 5.79 [05:08:16] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.63, 6.23, 5.83 [05:11:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.99, 6.44, 6.80 [05:11:12] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.99, 6.62, 6.25 [05:13:08] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.32, 6.52, 6.26 [05:13:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.67, 7.48 [05:15:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.71, 6.69, 6.82 [05:15:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.28, 7.18, 7.33 [05:17:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.72, 6.36, 6.69 [05:17:52] PROBLEM - txt-dex.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:19:06] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/semanticmediawiki.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/SemanticMediaWiki/maintenance/setupStore.php (END - exit=0) [05:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:21:02] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.54, 6.61, 6.12 [05:21:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 7.50, 7.11 [05:21:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.43, 7.62, 7.36 [05:21:33] !log [universalomega@mwtask181] starting deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] [05:21:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:22:15] !log [universalomega@mwtask181] finished deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json', 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask181] - SUCCESS in 41s [05:22:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:01] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.36, 6.30, 6.13 [05:27:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.57, 7.43 [05:29:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.29, 6.20, 6.68 [05:30:40] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:31:17] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [05:34:54] PROBLEM - www.deadprem.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.deadprem.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:47:07] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.51, 1.81, 1.09 [05:47:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.86, 8.02, 7.51 [05:47:39] PROBLEM - txt-dex.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - txt-dex.com All nameservers failed to answer the query. [05:47:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.97, 7.78, 7.00 [05:48:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.38, 7.07, 6.10 [05:49:07] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.02, 1.44, 1.04 [05:49:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.43, 7.55, 7.03 [05:50:50] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.80, 6.83, 6.12 [05:52:49] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.17, 6.34, 6.04 [05:56:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.18, 11.38, 6.85 [06:00:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.69, 6.56, 5.96 [06:01:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 8.00, 7.97 [06:07:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.37, 7.59, 7.21 [06:09:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.49, 7.26, 7.13 [06:15:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 6.75, 7.23 [06:17:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 7.04, 7.29 [06:17:28] PROBLEM - txt-dex.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [06:17:33] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.34, 6.81, 6.52 [06:18:08] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [06:21:24] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.28, 6.65, 6.53 [06:22:08] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [06:25:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.74, 6.27, 6.69 [06:27:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.12, 7.79, 7.57 [06:29:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.70, 7.56 [06:39:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.84, 7.22, 7.26 [06:45:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.53, 7.42 [06:46:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.08, 6.30, 5.95 [06:47:16] PROBLEM - txt-dex.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - txt-dex.com All nameservers failed to answer the query. [06:47:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.87, 6.98, 6.79 [06:48:38] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.17, 5.81, 5.81 [06:49:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.23, 6.64, 6.69 [06:53:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.63, 7.13, 6.87 [06:57:14] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.89, 7.16, 6.88 [06:59:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.83, 7.07, 6.89 [07:01:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.10, 7.51, 7.05 [07:03:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.35, 7.15, 6.99 [07:04:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 6.96, 6.71, 6.25 [07:05:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.39, 8.28, 7.42 [07:05:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.57, 8.11, 7.39 [07:07:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.08, 7.36, 6.31 [07:08:15] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.82, 7.04, 6.41 [07:09:11] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.47, 6.61, 6.16 [07:14:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 8.88, 7.77, 7.03 [07:18:38] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 5.01, 6.95, 6.92 [07:19:42] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:20:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.72, 7.89, 7.18 [07:21:03] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.70, 7.08, 6.57 [07:22:10] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 7.83, 7.25 [07:22:38]