[00:01:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc343e722d66...0e94e5a3418c [00:01:36] [02miraheze/puppet] 07Universal-Omega 030e94e5a - mwdeploy: remove test131 [00:02:58] miraheze/puppet - Universal-Omega the build passed. [00:03:36] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/93f2b5f07b48...11999f62edbe [00:03:39] [02miraheze/MirahezeDebug] 07Universal-Omega 0311999f6 - Remove test131 [00:05:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e94e5a3418c...85f8e7147959 [00:05:06] [02miraheze/puppet] 07Universal-Omega 0385f8e71 - Remove test131 Varnish backend [00:06:18] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 3.34, 1.94 [00:07:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03test151 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c267ba594cf7 [00:07:40] [02miraheze/mw-config] 07Universal-Omega 03c267ba5 - Use test151 [00:07:42] [02mw-config] 07Universal-Omega created branch 03test151 - 13https://github.com/miraheze/mw-config [00:07:43] [02mw-config] 07Universal-Omega opened pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:08:38] miraheze/mw-config - Universal-Omega the build passed. [00:08:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03test151 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c267ba594cf7...fe423e76c870 [00:08:41] [02miraheze/mw-config] 07Universal-Omega 03fe423e7 - -test131 [00:08:43] [02mw-config] 07Universal-Omega synchronize pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:09:38] miraheze/mw-config - Universal-Omega the build passed. [00:10:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03test151 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fe423e76c870...26bb0db4330c [00:10:19] [02miraheze/mw-config] 07Universal-Omega 0326bb0db - Update [00:10:22] [02mw-config] 07Universal-Omega synchronize pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:11:10] miraheze/mw-config - Universal-Omega the build passed. [00:11:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03test151 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/26bb0db4330c...e0b3b5a680f9 [00:12:00] [02miraheze/mw-config] 07Universal-Omega 03e0b3b5a - Update [00:12:02] [02mw-config] 07Universal-Omega synchronize pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:12:58] miraheze/mw-config - Universal-Omega the build passed. [00:13:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03test151 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e0b3b5a680f9...65b96032bcb6 [00:13:25] [02miraheze/mw-config] 07Universal-Omega 0365b9603 - Remove test131 [00:13:27] [02mw-config] 07Universal-Omega synchronize pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:13:34] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 3.40, 2.62 [00:13:48] PROBLEM - cp35 Nginx Backend for test131 on cp35 is CRITICAL: connect to address localhost and port 8180: Connection refused [00:14:07] [02mw-config] 07Universal-Omega closed pull request 03#5439: Use test151 - 13https://github.com/miraheze/mw-config/pull/5439 [00:14:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/mw-config/compare/9f073178c2fe...98e22e94792c [00:14:12] [02miraheze/mw-config] 07Universal-Omega 0398e22e9 - Use test151 (#5439) [00:14:15] [02miraheze/mw-config] 07Universal-Omega deleted branch 03test151 [00:14:16] miraheze/mw-config - Universal-Omega the build passed. [00:14:17] [02mw-config] 07Universal-Omega deleted branch 03test151 - 13https://github.com/miraheze/mw-config [00:15:02] miraheze/mw-config - Universal-Omega the build passed. [00:15:31] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.64, 3.46, 2.74 [00:17:21] !log [universalomega@test151] starting deploy of {'pull': 'config', 'config': True} to all [00:17:28] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.68, 3.82, 2.95 [00:17:34] !log [universalomega@test151] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 13s [00:17:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.01, 7.08, 6.49 [00:18:32] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:19:27] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.80, 3.25, 2.84 [00:19:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.57, 6.21, 6.23 [00:22:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-test131 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/b9e3b101223b [00:22:41] [02miraheze/puppet] 07Universal-Omega 03b9e3b10 - Remove test131 [00:22:42] [02puppet] 07Universal-Omega created branch 03remove-test131 - 13https://github.com/miraheze/puppet [00:22:44] [02puppet] 07Universal-Omega opened pull request 03#3651: Remove test131 - 13https://github.com/miraheze/puppet/pull/3651 [00:23:03] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.53, 3.81, 3.67 [00:23:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-test131 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b9e3b101223b...a99364b3041c [00:23:28] [02miraheze/puppet] 07Universal-Omega 03a99364b - Remove [00:23:30] [02puppet] 07Universal-Omega synchronize pull request 03#3651: Remove test131 - 13https://github.com/miraheze/puppet/pull/3651 [00:28:07] PROBLEM - cp34 Nginx Backend for test131 on cp34 is CRITICAL: connect to address localhost and port 8180: Connection refused [00:28:54] PROBLEM - cp25 Nginx Backend for test131 on cp25 is CRITICAL: connect to address localhost and port 8180: Connection refused [00:28:57] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.65, 3.03, 3.33 [00:30:36] PROBLEM - cp24 Nginx Backend for test131 on cp24 is CRITICAL: connect to address localhost and port 8180: Connection refused [00:31:37] !log [@mwtask181] starting deploy of {'config': True} to all [00:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:38] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 61s [00:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:23] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.02923932672 secs [00:45:51] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.55, 3.58, 3.14 [00:46:52] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:53:37] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.47, 4.00, 3.62 [00:55:33] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.91, 3.24, 3.38 [01:02:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.68, 3.85, 3.25 [01:06:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 4.23, 3.51 [01:18:44] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.67, 3.66, 3.71 [01:22:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.82, 2.65, 3.28 [01:29:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03test151-multiversion [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/21f5bd5c3f3a [01:29:57] [02miraheze/puppet] 07Universal-Omega 0321f5bd5 - test151: enable the use of true multiversion [01:29:58] [02puppet] 07Universal-Omega created branch 03test151-multiversion - 13https://github.com/miraheze/puppet [01:29:59] [02puppet] 07Universal-Omega opened pull request 03#3652: test151: enable the use of true multiversion - 13https://github.com/miraheze/puppet/pull/3652 [01:31:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85f8e7147959...214e15a07892 [01:31:35] [02miraheze/puppet] 07Universal-Omega 03214e15a - test_mwdeploy: use mw151 and mw152 [01:32:58] miraheze/puppet - Universal-Omega the build passed. [01:36:57] [02puppet] 07Universal-Omega closed pull request 03#3652: test151: enable the use of true multiversion - 13https://github.com/miraheze/puppet/pull/3652 [01:36:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/214e15a07892...3034cf5fb05a [01:37:00] [02miraheze/puppet] 07Universal-Omega 033034cf5 - test151: enable the use of true multiversion (#3652) [01:37:02] [02miraheze/puppet] 07Universal-Omega deleted branch 03test151-multiversion [01:37:05] [02puppet] 07Universal-Omega deleted branch 03test151-multiversion - 13https://github.com/miraheze/puppet [01:40:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.78, 4.44, 3.46 [01:41:33] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [01:42:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.50, 3.40, 3.21 [01:47:51] !log [@test151] starting deploy of {'folders': '1.40/extensions/MirahezeMagic'} to all [01:47:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.84, 6.41 [01:57:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.40, 6.79, 6.44 [01:59:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03common-multiversion [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1ec8818532d4 [01:59:07] [02miraheze/puppet] 07Universal-Omega 031ec8818 - common: enable the use of true multiversion [01:59:08] [02puppet] 07Universal-Omega created branch 03common-multiversion - 13https://github.com/miraheze/puppet [01:59:11] [02puppet] 07Universal-Omega opened pull request 03#3653: common: enable the use of true multiversion - 13https://github.com/miraheze/puppet/pull/3653 [02:03:55] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [02:04:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:09:51] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [02:11:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.35, 6.94, 6.71 [02:13:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.48, 6.74, 6.66 [02:13:53] !log [universalomega@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.40'} to all [02:14:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:29] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.42/skins/Femiwiki/node_modules],Exec[git_clone_MediaWiki-REL1_40 Mermaid],Exec[MediaWiki-REL1_40 MirahezeMagic Sync] [02:18:11] !log [universalomega@test151] finished deploy of {'world': True, 'l10n': True, 'versions': '1.40'} to all - SUCCESS in 258s [02:18:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3034cf5fb05a...f680332b8f81 [02:21:15] [02miraheze/puppet] 07Universal-Omega 03f680332 - Update mwdeploy [02:23:04] miraheze/puppet - Universal-Omega the build passed. [02:23:24] !log [universalomega@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.42'} to test151 [02:23:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:31] !log [universalomega@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:51] !log [universalomega@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.42'} to test151 [02:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:57] !log [universalomega@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:24:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:54] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/1f28242f1435...7e1e1529c17d [02:27:57] [02miraheze/mediawiki-repos] 07Universal-Omega 037e1e152 - Support psy/psysh ^0.12.0 [02:28:27] !log [universalomega@test151] starting deploy of {'world': True, 'l10n': True, 'versions': '1.42'} to test151 [02:28:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:45] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 7.08, 4.56, 2.96 [02:33:20] !log [universalomega@test151] finished deploy of {'world': True, 'l10n': True, 'versions': '1.42'} to test151 - SUCCESS in 293s [02:33:22] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 2.93, 1.60 [02:33:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:20] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.13, 2.64, 1.65 [02:36:26] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:38:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.04, 3.43, 3.10 [02:40:45] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.15, 3.09, 3.02 [02:41:07] [02puppet] 07Universal-Omega closed pull request 03#3653: common: enable the use of true multiversion - 13https://github.com/miraheze/puppet/pull/3653 [02:41:08] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f680332b8f81...00ec42e25c8f [02:41:11] [02miraheze/puppet] 07Universal-Omega 0300ec42e - common: enable the use of true multiversion (#3653) [02:41:14] [02miraheze/puppet] 07Universal-Omega deleted branch 03common-multiversion [02:41:17] [02puppet] 07Universal-Omega deleted branch 03common-multiversion - 13https://github.com/miraheze/puppet [02:45:46] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.04, 6.94, 6.31 [02:47:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.96, 6.25, 6.13 [02:50:16] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [02:50:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:31] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [02:57:38] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:00:25] !log [universalomega@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.41'} to mwtask181 [03:00:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:44] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:00:56] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:01:34] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [03:02:36] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:02:44] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [03:03:05] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 1 failures [03:03:43] PROBLEM - test131 HTTPS on test131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test131.miraheze.org port 443 after 128 ms: Couldn't connect to server [03:04:07] !log [universalomega@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': '1.41'} to mwtask181 - SUCCESS in 221s [03:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:11] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:05:40] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [03:05:58] PROBLEM - test131 JobChron Service on test131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [03:06:02] PROBLEM - test131 memcached on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 11211: Connection refused [03:06:25] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: connect to address 2a10:6740::6:406 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [03:06:33] PROBLEM - test131 php-fpm on test131 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [03:06:46] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:07:43] !log [universalomega@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.41'} to all [03:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:51] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:08:05] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:08:11] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:08:18] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:09:02] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.101641804 secs [03:13:04] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:13:32] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules] [03:13:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 7.06, 6.34 [03:15:53] !log [universalomega@mwtask181] starting deploy of {'world': True, 'l10n': True, 'versions': '1.41'} to all [03:15:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.57, 6.60, 6.28 [03:21:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 7.25, 6.67 [03:23:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 8.26, 7.14 [03:25:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 7.63, 7.04 [03:25:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.32, 7.04, 6.56 [03:27:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.95, 6.26, 6.33 [03:31:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.66, 7.56, 7.08 [03:31:34] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:31:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.90, 7.03, 6.58 [03:33:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.61, 6.48, 6.41 [03:35:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.39, 6.75, 6.88 [03:36:18] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [03:41:32] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [03:53:37] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.99, 3.00, 1.74 [03:53:48] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:55:21] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.41, 6.74 [03:55:30] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:55:35] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.99, 4.06, 2.28 [03:55:48] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 4.024 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:56:20] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:56:43] PROBLEM - mw142 PowerDNS Recursor on mw142 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:56:45] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 28.75, 16.65, 9.86 [03:56:54] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [03:56:57] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.33, 9.58, 6.75 [03:57:07] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw141 mw142 [03:57:15] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw142 [03:57:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.96, 10.86, 6.88 [03:58:19] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.515 second response time [03:58:28] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:41] RECOVERY - mw142 PowerDNS Recursor on mw142 is OK: DNS OK: 1.740 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:58:45] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw143 [03:58:50] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw143 [03:58:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.497 second response time [03:59:15] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [03:59:32] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.512 second response time [03:59:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.92, 9.54, 6.89 [04:00:24] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.900 second response time [04:00:35] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:00:45] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [04:00:50] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [04:00:55] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.67, 10.09, 7.82 [04:01:09] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [04:01:34] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [04:02:40] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.27, 10.61, 9.41 [04:03:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.10, 6.70, 6.22 [04:04:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.05, 6.88 [04:04:40] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.71, 8.65, 8.84 [04:05:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.76, 6.20, 6.09 [04:06:14] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [04:07:10] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:07:51] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:08:11] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:08:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.62, 6.69, 6.80 [04:11:24] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.67, 2.94, 3.45 [04:13:21] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.19, 2.01, 3.04 [04:16:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.02, 7.37, 6.63 [04:18:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 7.50, 7.11 [04:18:23] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.88, 6.65, 6.44 [04:23:38] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:25:31] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:26:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.77, 6.51, 6.78 [04:26:56] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:29:34] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [04:33:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.64, 7.06, 6.47 [04:34:46] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:35:44] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.76, 6.79, 6.44 [04:39:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.72, 7.12, 6.68 [04:41:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.31, 6.68, 6.59 [04:44:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.85, 6.67 [04:48:11] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [04:48:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.53, 6.30, 6.51 [05:16:11] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:21:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.72, 7.27, 6.55 [05:29:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.51, 6.51 [05:46:44] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.85, 6.73, 6.23 [05:47:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.93, 7.64, 6.89 [05:48:38] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.56, 6.71, 6.29 [05:49:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 7.03, 6.76 [05:50:11] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [05:53:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.83, 6.59, 6.70 [06:12:20] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [06:14:19] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.781 second response time [06:18:10] !log [universalomega@mwtask181] finished deploy of {'world': True, 'l10n': True, 'versions': '1.41'} to all - SUCCESS in 10938s [06:18:11] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:20:19] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:34:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.10, 6.65, 6.04 [06:36:17] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.83, 6.38, 6.02 [06:49:39] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:00:17] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:12:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.73, 6.55, 6.18 [07:14:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.83, 5.81, 5.95 [07:18:11] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:18:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:30:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:35:42] [02puppet] 07Universal-Omega synchronize pull request 03#3582: mariadb: remove SSL - 13https://github.com/miraheze/puppet/pull/3582 [07:35:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mariadb-remove-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/43973e1b00c0...3c99d9b4939f [07:35:46] [02miraheze/puppet] 07Universal-Omega 033c99d9b - + [07:45:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 6.72, 6.14 [07:47:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.42, 6.10 [07:48:11] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:48:18] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [07:48:29] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:59:43] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:11:45] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:27:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.36, 6.97, 6.33 [08:29:27] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:29:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.88, 6.54, 6.25 [08:30:42] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - with.cpt-ra.bid All nameservers failed to answer the query. [08:40:02] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:44:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 7.91, 6.81 [08:46:29] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:46:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.56, 6.83 [08:48:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.97, 6.73, 6.61 [08:49:57] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:50:00] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:50:11] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [08:50:22] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:50:22] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:50:54] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [08:51:02] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [08:51:56] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.724 second response time [08:51:57] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.534 second response time [08:52:06] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.697 second response time [08:52:19] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.643 second response time [08:52:19] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.602 second response time [08:52:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.513 second response time [08:52:59] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.604 second response time [08:59:15] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp35.miraheze.org - CNAME OK [09:03:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.53, 7.64, 6.65 [09:05:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.27, 6.79, 6.47 [09:33:04] PROBLEM - itadminwiki.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itadminwiki.pl' expires in 15 day(s) (Fri 02 Feb 2024 09:29:29 AM GMT +0000). [09:50:11] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [09:52:54] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [09:53:01] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [09:53:16] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [09:54:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.37, 3.78, 2.07 [09:54:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.517 second response time [09:54:59] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.641 second response time [09:55:30] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [09:57:15] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.605 second response time [09:57:26] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.496 second response time [09:58:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.39, 3.30, 2.35 [09:58:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:12:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.20, 6.26 [10:14:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.80, 6.27, 6.02 [10:18:11] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:28:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:39:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.21, 6.78, 6.00 [10:41:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.96, 6.77, 6.08 [10:41:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.20, 6.41, 5.96 [10:43:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.24, 6.28, 6.00 [10:54:20] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [10:56:19] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3459 bytes in 0.666 second response time [11:00:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.30, 2.87 [11:03:34] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [11:10:45] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.24, 3.40, 3.13 [11:31:34] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.71, 7.23, 6.40 [11:38:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.09, 7.59, 6.61 [11:40:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.19, 6.94, 6.48 [11:42:27] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.28, 6.36, 6.32 [11:47:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.47, 3.29, 3.00 [11:49:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.67, 2.96, 2.91 [11:53:34] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.00, 6.87, 6.62 [11:55:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.17, 6.23, 6.41 [12:07:16] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [12:07:30] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [12:08:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.07, 6.68 [12:09:11] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3459 bytes in 0.604 second response time [12:09:26] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.546 second response time [12:10:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.21, 6.68, 6.58 [12:10:35] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.20, 4.55, 2.81 [12:12:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.14, 3.70, 2.72 [12:14:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.33, 3.35, 2.71 [12:22:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/00ec42e25c8f...dc4efbfcad18 [12:22:29] [02miraheze/puppet] 07paladox 03dc4efbf - matomo: enable CustomVariables plugin [12:28:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.08, 6.73 [12:30:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.69, 6.21, 6.46 [12:51:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.59, 7.65, 6.62 [12:52:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.55, 7.15, 6.54 [12:54:58] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.37, 6.45, 6.35 [12:56:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.09, 7.60, 6.57 [12:58:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.47, 7.00, 6.46 [13:00:23] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.35, 6.73, 6.42 [13:06:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.85, 3.34, 2.06 [13:08:34] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.56, 3.28, 2.20 [13:12:34] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.96, 3.24, 2.44 [13:12:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 6.74, 6.43 [13:14:46] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.19, 6.10, 6.23 [13:18:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc4efbfcad18...598576da0375 [13:18:12] [02miraheze/puppet] 07paladox 03598576d - Revert "matomo: enable CustomVariables plugin" [13:28:54] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [13:29:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.57, 4.67, 3.41 [13:30:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/598576da0375...a386ec2231a2 [13:30:05] [02miraheze/puppet] 07paladox 03a386ec2 - matomo: enable CustomVariables plugin [13:30:50] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.662 second response time [13:31:39] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.67, 10.20, 7.75 [13:32:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.47, 9.50, 7.86 [13:32:08] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.38, 4.38, 3.17 [13:32:50] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.95, 11.73, 9.39 [13:32:57] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.24, 10.27, 8.10 [13:33:35] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.50, 9.34, 7.74 [13:33:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.85, 3.29 [13:34:07] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.39, 8.82, 7.82 [13:34:57] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.59, 9.17, 7.95 [13:35:55] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.65, 3.50, 3.24 [13:36:01] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.54, 3.98, 3.29 [13:36:50] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.33, 10.04, 9.24 [13:37:52] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.68, 3.30, 3.19 [13:39:53] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.72, 3.17, 3.09 [13:43:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.58, 6.92 [13:47:25] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.44, 3.50, 3.84 [13:48:29] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [13:49:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.37, 8.01, 7.22 [13:51:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.60, 7.17 [13:53:18] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.69, 2.00, 3.09 [14:03:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.58, 8.04, 7.49 [14:03:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.32, 7.14, 6.43 [14:05:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.68, 6.86, 6.39 [14:07:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 7.72, 7.55 [14:07:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.52, 6.73, 6.40 [14:09:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.47, 7.81, 7.58 [14:11:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.31, 6.95, 6.60 [14:13:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.66, 7.57 [14:15:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.51, 7.86, 7.63 [14:15:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.15, 9.25, 7.59 [14:16:29] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:17:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 7.72, 7.64 [14:17:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.94, 7.95, 7.32 [14:21:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.36, 9.42, 8.31 [14:25:43]