[03:00:49] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 48 seconds old and 0 bytes [04:35:38] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:42:55] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [05:00:37] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:11:48] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp24.miraheze.org - CNAME OK [05:42:55] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [05:56:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:11:48] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp25.miraheze.org - CNAME OK [06:26:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:24:07] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.55, 3.16, 2.45 [08:26:07] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 2.11, 2.86, 2.43 [08:30:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:46:31] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.49, 2.92, 2.54 [08:48:26] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.88, 2.55, 2.46 [08:53:16] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.40, 3.01, 4.00 [08:55:16] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 5.95, 3.97, 4.22 [09:11:51] PROBLEM - crustypedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crustypedia.org' expires in 15 day(s) (Fri 03 Nov 2023 09:01:30 GMT +0000). [09:18:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/125ad5f1d6c5...2d3d776501ac [09:18:32] [02miraheze/ssl] 07MirahezeSSLBot 032d3d776 - Bot: Update SSL cert for crustypedia.org [09:30:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:41:12] RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Tue 16 Jan 2024 08:18:21 GMT +0000. [09:56:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:26:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:39:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/080ecf3278e0 [11:39:23] [02miraheze/mw-config] 07paladox 03080ecf3 - Remove duplicate configs [11:39:25] [02mw-config] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [11:39:27] [02mw-config] 07paladox opened pull request 03#5337: Remove duplicate configs - 13https://github.com/miraheze/mw-config/pull/5337 [11:39:46] [02mw-config] 07paladox closed pull request 03#5337: Remove duplicate configs - 13https://github.com/miraheze/mw-config/pull/5337 [11:39:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d018cd7f21f7...3ce993eddbee [11:39:50] [02miraheze/mw-config] 07paladox 033ce993e - Remove duplicate configs (#5337) [11:39:52] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-5 [11:39:53] [02mw-config] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [11:40:36] miraheze/mw-config - paladox the build passed. [11:40:56] miraheze/mw-config - paladox the build passed. [11:56:41] !log [@test131] starting deploy of {'config': True} to all [11:56:42] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [11:56:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:56:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:50] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.52, 1.67, 1.00 [12:00:29] RECOVERY - phab121 APT on phab121 is OK: APT OK: 5 packages available for upgrade (0 critical updates). [12:01:31] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:05:17] PROBLEM - jobchron121 APT on jobchron121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:05:26] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.44, 0.54, 0.20 [12:09:08] !log [@mwtask141] starting deploy of {'config': True} to all [12:09:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:16] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:09:17] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 9s [12:09:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:11:35] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.96, 1.91, 1.24 [12:13:51] PROBLEM - jobchron121 Current Load on jobchron121 is CRITICAL: connect to address 2a10:6740::6:306 port 5666: No route to hostconnect to host 2a10:6740::6:306 port 5666: No route to host [12:14:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cc2996d1b834 [12:14:38] [02miraheze/puppet] 07paladox 03cc2996d - jobchron121: switch to php 8.2 [12:14:40] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:14:43] [02puppet] 07paladox opened pull request 03#3452: jobchron121: switch to php 8.2 - 13https://github.com/miraheze/puppet/pull/3452 [12:15:48] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.91, 0.83, 0.32 [12:16:16] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:17:44] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 1.53, 0.67, 0.25 [12:18:23] [02puppet] 07paladox closed pull request 03#3452: jobchron121: switch to php 8.2 - 13https://github.com/miraheze/puppet/pull/3452 [12:18:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/98172e44fa67...cd75a2050b59 [12:18:26] [02miraheze/puppet] 07paladox 03cd75a20 - jobchron121: switch to php 8.2 (#3452) [12:18:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [12:18:30] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:18:33] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 208 packages available for upgrade (2 critical updates). [12:19:13] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:19:14] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:19:30] PROBLEM - mw142 JobRunner Service on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:19:30] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [12:20:47] PROBLEM - jobchron121 Puppet on jobchron121 is UNKNOWN: NRPE: Unable to read output [12:22:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/580a8396b936 [12:22:32] [02miraheze/puppet] 07paladox 03580a839 - phab121: upgrade php to 8.2 [12:22:33] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:22:36] [02puppet] 07paladox opened pull request 03#3453: phab121: upgrade php to 8.2 - 13https://github.com/miraheze/puppet/pull/3453 [12:22:44] [02puppet] 07paladox closed pull request 03#3453: phab121: upgrade php to 8.2 - 13https://github.com/miraheze/puppet/pull/3453 [12:22:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cd75a2050b59...5d78438c8ed8 [12:22:48] [02miraheze/puppet] 07paladox 035d78438 - phab121: upgrade php to 8.2 (#3453) [12:22:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [12:22:53] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [12:24:06] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:25:06] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:25:07] RECOVERY - mw141 JobRunner Service on mw141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:25:15] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:25:25] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [12:27:22] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 4.07, 2.12, 1.05 [12:28:28] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.89, 1.68, 1.03 [12:30:46] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [12:34:17] PROBLEM - jobchron121 Current Load on jobchron121 is CRITICAL: CRITICAL - load average: 2.69, 2.07, 1.39 [12:38:12] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 1.50, 1.69, 1.39 [12:41:49] PROBLEM - phab121 conntrack_table_size on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:41:55] PROBLEM - phab121 PowerDNS Recursor on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:06] PROBLEM - phab121 Backups Phabricator Static on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:14] PROBLEM - phab121 APT on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:42] PROBLEM - phab121 phd on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:42] PROBLEM - phab121 Puppet on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:46] PROBLEM - phab121 ferm_active on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:42:51] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 212 packages available for upgrade (6 critical updates). [12:43:10] PROBLEM - phab121 NTP time on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:43:11] PROBLEM - phab121 Disk Space on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:43:30] PROBLEM - phab121 php-fpm on phab121 is CRITICAL: connect to address 2a10:6740::6:311 port 5666: Connection refusedconnect to host 2a10:6740::6:311 port 5666: Connection refused [12:44:51] PROBLEM - jobchron121 APT on jobchron121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:46:53] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 193 packages available for upgrade (11 critical updates). [12:53:02] PROBLEM - jobchron121 APT on jobchron121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:54:48] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:55:44] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.90, 1.89, 1.63 [12:55:55] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.033 second response time [12:57:08] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 126 packages available for upgrade (11 critical updates). [12:57:15] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [12:59:08] PROBLEM - jobchron121 APT on jobchron121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:59:37] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 1.42, 1.66, 1.60 [13:01:10] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 21 packages available for upgrade (0 critical updates). [13:10:39] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:12:32] RECOVERY - phab121 ferm_active on phab121 is OK: OK ferm input default policy is set [13:12:42] RECOVERY - phab121 phd on phab121 is OK: PROCS OK: 1 process with args 'phd' [13:12:43] PROBLEM - phab121 Puppet on phab121 is UNKNOWN: NRPE: Unable to read output [13:13:02] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.53, 1.21, 1.52 [13:13:03] RECOVERY - phab121 NTP time on phab121 is OK: NTP OK: Offset 0.00371414423 secs [13:13:04] RECOVERY - phab121 Disk Space on phab121 is OK: DISK OK - free space: / 23825MiB (53% inode=90%); [13:13:16] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:13:26] PROBLEM - wariopedia.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4081504344940186 seconds [13:13:33] RECOVERY - phab121 PowerDNS Recursor on phab121 is OK: DNS OK: 0.316 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [13:13:49] RECOVERY - phab121 conntrack_table_size on phab121 is OK: OK: nf_conntrack is 0 % full [13:14:06] RECOVERY - phab121 Backups Phabricator Static on phab121 is OK: FILE_AGE OK: /var/log/phabricator-backup.log is 296437 seconds old and 96 bytes [13:22:42] PROBLEM - phab121 phd on phab121 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [13:22:58] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:22:58] PROBLEM - mw132 JobRunner Service on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:23:27] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:24:01] PROBLEM - mw142 JobRunner Service on mw142 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:24:06] PROBLEM - mwtask141 JobRunner Service on mwtask141 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:24:39] PROBLEM - mw131 JobRunner Service on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [13:25:30] RECOVERY - phab121 php-fpm on phab121 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [13:25:58] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20260 bytes in 3.735 second response time [13:26:42] RECOVERY - phab121 phd on phab121 is OK: PROCS OK: 1 process with args 'phd' [13:27:03] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:32:01] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:36:58] RECOVERY - mw141 JobRunner Service on mw141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:36:58] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:40:00] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:40:06] RECOVERY - mwtask141 JobRunner Service on mwtask141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:40:39] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [13:42:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e2d063b57adc [13:42:14] [02miraheze/mw-config] 07paladox 03e2d063b - Support newer xhprof php extension [13:42:16] [02mw-config] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [13:42:18] [02mw-config] 07paladox opened pull request 03#5338: Support newer xhprof php extension - 13https://github.com/miraheze/mw-config/pull/5338 [13:42:35] [02mw-config] 07paladox synchronize pull request 03#5338: Support newer xhprof php extension - 13https://github.com/miraheze/mw-config/pull/5338 [13:42:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e2d063b57adc...8707d0d82536 [13:42:39] [02miraheze/mw-config] 07paladox 038707d0d - Update LocalSettings.php [13:42:47] PROBLEM - wariopedia.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wariopedia.com All nameservers failed to answer the query. [13:43:23] miraheze/mw-config - paladox the build has errored. [13:43:50] miraheze/mw-config - paladox the build has errored. [13:45:12] [02mw-config] 07paladox synchronize pull request 03#5338: Support newer xhprof php extension - 13https://github.com/miraheze/mw-config/pull/5338 [13:45:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8707d0d82536...787a56383908 [13:45:17] [02miraheze/mw-config] 07paladox 03787a563 - Update LocalSettings.php [13:46:13] miraheze/mw-config - paladox the build passed. [13:46:35] [02mw-config] 07paladox closed pull request 03#5338: Support newer xhprof php extension - 13https://github.com/miraheze/mw-config/pull/5338 [13:46:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3ce993eddbee...783799166f32 [13:46:41] [02miraheze/mw-config] 07paladox 037837991 - Support newer xhprof php extension (#5338) [13:46:43] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-5 [13:46:45] [02mw-config] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/mw-config [13:47:11] miraheze/mw-config - paladox the build passed. [13:49:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+4/-2/±1] 13https://github.com/miraheze/puppet/compare/5d78438c8ed8...29c23b435165 [13:49:15] [02miraheze/puppet] 07paladox 0329c23b4 - mediawiki: add support for php 8.2 (partially) [13:50:38] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [13:53:11] [02miraheze/jobrunner-service] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/jobrunner-service/commit/b080127051b5 [13:53:13] [02miraheze/jobrunner-service] 07paladox 03b080127 - Add support for php 8.2 [13:53:15] [02jobrunner-service] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/jobrunner-service [13:53:17] [02jobrunner-service] 07paladox opened pull request 03#10: Add support for php 8.2 - 13https://github.com/miraheze/jobrunner-service/pull/10 [13:54:08] [02miraheze/jobrunner-service] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/jobrunner-service/compare/b080127051b5...1aef698e3e51 [13:54:08] [02miraheze/jobrunner-service] 07paladox 031aef698 - Update composer.json [13:54:09] [02jobrunner-service] 07paladox synchronize pull request 03#10: Add support for php 8.2 - 13https://github.com/miraheze/jobrunner-service/pull/10 [13:55:30] [02miraheze/jobrunner-service] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/jobrunner-service/compare/1aef698e3e51...4e58313522b0 [13:55:32] [02miraheze/jobrunner-service] 07paladox 034e58313 - Update composer.json [13:55:34] [02jobrunner-service] 07paladox synchronize pull request 03#10: Add support for php 8.2 - 13https://github.com/miraheze/jobrunner-service/pull/10 [13:56:07] !log [@test131] starting deploy of {'config': True} to all [13:56:07] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:56:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:33] [02miraheze/jobrunner-service] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/jobrunner-service/compare/4e58313522b0...bced2de98a7a [13:56:35] [02miraheze/jobrunner-service] 07paladox 03bced2de - Update redisJobChronService [13:56:38] [02jobrunner-service] 07paladox synchronize pull request 03#10: Add support for php 8.2 - 13https://github.com/miraheze/jobrunner-service/pull/10 [13:57:58] [02jobrunner-service] 07paladox closed pull request 03#10: Add support for php 8.2 - 13https://github.com/miraheze/jobrunner-service/pull/10 [13:57:59] [02miraheze/jobrunner-service] 07paladox pushed 035 commits to 03miraheze [+0/-0/±6] 13https://github.com/miraheze/jobrunner-service/compare/58f08f5d7b66...0ef93c66d089 [13:58:00] [02miraheze/jobrunner-service] 07paladox 030ef93c6 - Merge pull request #10 from miraheze/paladox-patch-1 [13:58:02] [02miraheze/jobrunner-service] 07paladox deleted branch 03paladox-patch-1 [13:58:03] [02jobrunner-service] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/jobrunner-service [14:05:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c068e22e0356 [14:05:41] [02miraheze/puppet] 07paladox 03c068e22 - php: Don't install json on 8.2+ [14:05:44] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:05:45] [02puppet] 07paladox opened pull request 03#3454: php: Don't install json on 8.2+ - 13https://github.com/miraheze/puppet/pull/3454 [14:08:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c068e22e0356...e147a0067746 [14:08:55] [02miraheze/puppet] 07paladox 03e147a00 - Update default_extensions.pp [14:08:56] [02puppet] 07paladox synchronize pull request 03#3454: php: Don't install json on 8.2+ - 13https://github.com/miraheze/puppet/pull/3454 [14:09:29] [02puppet] 07paladox closed pull request 03#3454: php: Don't install json on 8.2+ - 13https://github.com/miraheze/puppet/pull/3454 [14:09:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/29c23b435165...956065dbf8df [14:09:32] !log [@mwtask141] starting deploy of {'config': True} to all [14:09:34] [02miraheze/puppet] 07paladox 03956065d - php: Don't install json on 8.2+ (#3454) [14:09:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:09:36] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:09:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:42] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [14:09:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:12:33] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:15:25] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:20:44] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:24:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4c542969aaf4 [14:24:58] [02miraheze/puppet] 07paladox 034c54296 - php: fix syntax [14:25:01] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:25:02] [02puppet] 07paladox opened pull request 03#3455: php: fix syntax - 13https://github.com/miraheze/puppet/pull/3455 [14:25:08] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:25:30] [02puppet] 07paladox closed pull request 03#3455: php: fix syntax - 13https://github.com/miraheze/puppet/pull/3455 [14:25:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/956065dbf8df...02bbe06cb31d [14:25:32] [02miraheze/puppet] 07paladox 0302bbe06 - php: fix syntax (#3455) [14:25:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:25:38] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:26:30] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:28:46] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:35:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/06f5fb37dffb [14:35:02] [02miraheze/puppet] 07paladox 0306f5fb3 - mon141: upgrade php to 8.2 [14:35:04] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:35:08] [02puppet] 07paladox opened pull request 03#3456: mon141: upgrade php to 8.2 - 13https://github.com/miraheze/puppet/pull/3456 [14:35:21] [02puppet] 07paladox closed pull request 03#3456: mon141: upgrade php to 8.2 - 13https://github.com/miraheze/puppet/pull/3456 [14:35:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/02bbe06cb31d...f55297257a79 [14:35:26] [02miraheze/puppet] 07paladox 03f552972 - mon141: upgrade php to 8.2 (#3456) [14:35:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [14:35:30] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [14:36:04] PROBLEM - mon141 APT on mon141 is CRITICAL: APT CRITICAL: 185 packages available for upgrade (2 critical updates). [14:38:06] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:44:12] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:45:55] PROBLEM - mon141 Current Load on mon141 is CRITICAL: CRITICAL - load average: 6.07, 8.39, 4.36 [14:46:26] PROBLEM - mon141 APT on mon141 is CRITICAL: APT CRITICAL: 209 packages available for upgrade (8 critical updates). [14:49:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: /usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (5.1.0) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_li [14:49:24] t', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba2966d92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permis [14:49:24] denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:50:07] PROBLEM - db101 MariaDB Connections on db101 is UNKNOWN: execvpe(php) failed: No such file or directory [14:50:07] PROBLEM - mon141 icinga.miraheze.org HTTPS on mon141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.027 second response time [14:50:12] PROBLEM - db131 MariaDB Connections on db131 is UNKNOWN: execvpe(php) failed: No such file or directory [14:51:34] PROBLEM - ping6 on bast121 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:301CRITICAL - Could not interpret output from ping command [14:51:40] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: execvpe(php) failed: No such file or directory [14:51:41] PROBLEM - Host swiftobject122 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:317CRITICAL - Could not interpret output from ping command [14:51:43] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: execvpe(php) failed: No such file or directory [14:51:46] PROBLEM - ping6 on graylog131 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:512CRITICAL - Could not interpret output from ping command [14:51:47] PROBLEM - ping6 on db112 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:211CRITICAL - Could not interpret output from ping command [14:51:48] PROBLEM - ping6 on mw142 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:503CRITICAL - Could not interpret output from ping command [14:51:50] PROBLEM - Host test131 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:406CRITICAL - Could not interpret output from ping command [14:51:51] PROBLEM - ping6 on os131 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:401CRITICAL - Could not interpret output from ping command [14:51:51] PROBLEM - Host swiftobject121 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:316CRITICAL - Could not interpret output from ping command [14:51:51] PROBLEM - Host cloud14 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:500CRITICAL - Could not interpret output from ping command [14:51:53] PROBLEM - ping6 on cp34 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2607:5300:205:200::3121CRITICAL - Could not interpret output from ping command [14:51:53] PROBLEM - Host bast141 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:509CRITICAL - Could not interpret output from ping command [14:51:53] PROBLEM - ping6 on swiftobject113 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:205CRITICAL - Could not interpret output from ping command [14:51:53] PROBLEM - Host swiftobject111 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:203CRITICAL - Could not interpret output from ping command [14:51:54] PROBLEM - Host mem131 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:405CRITICAL - Could not interpret output from ping command [14:51:54] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: execvpe(php) failed: No such file or directory [14:51:55] PROBLEM - Host db101 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:102CRITICAL - Could not interpret output from ping command [14:51:55] PROBLEM - mon141 php-fpm on mon141 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.4' [14:51:55] PROBLEM - Host mw141 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:502CRITICAL - Could not interpret output from ping command [14:51:56] PROBLEM - Host mw142 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:503CRITICAL - Could not interpret output from ping command [14:51:56] PROBLEM - Host reports121 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:312CRITICAL - Could not interpret output from ping command [14:51:56] PROBLEM - ping6 on mw131 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:403CRITICAL - Could not interpret output from ping command [14:52:08] PROBLEM - Host db112 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:211CRITICAL - Could not interpret output from ping command [14:52:08] PROBLEM - Host ldap141 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:506CRITICAL - Could not interpret output from ping command [14:52:09] PROBLEM - Host cp35 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2607:5300:205:200::1c93CRITICAL - Could not interpret output from ping command [14:52:09] PROBLEM - Host cloud10 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:100CRITICAL - Could not interpret output from ping command [14:52:10] PROBLEM - ping6 on swiftproxy131 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:315CRITICAL - Could not interpret output from ping command [14:52:10] PROBLEM - Host mw131 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:403CRITICAL - Could not interpret output from ping command [14:52:11] PROBLEM - ping6 on matomo121 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:207CRITICAL - Could not interpret output from ping command [14:52:11] PROBLEM - ping6 on cloud13 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:400CRITICAL - Could not interpret output from ping command [14:52:12] PROBLEM - Host phab121 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:311CRITICAL - Could not interpret output from ping command [14:52:12] PROBLEM - Host cp25 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2001:41d0:801:2000::3a18CRITICAL - Could not interpret output from ping command [14:52:13] PROBLEM - ping6 on jobchron121 is UNKNOWN: /bin/ping -6 -n -U -w 10 -c 5 2a10:6740::6:306CRITICAL - Could not interpret output from ping command [14:52:13] PROBLEM - Host os141 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:501CRITICAL - Could not interpret output from ping command [14:52:14] PROBLEM - Host swiftobject112 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:204CRITICAL - Could not interpret output from ping command [14:52:14] PROBLEM - Host mw134 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:410CRITICAL - Could not interpret output from ping command [14:52:15] PROBLEM - Host mail121 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:307CRITICAL - Could not interpret output from ping command [14:52:15] PROBLEM - Host os131 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:401CRITICAL - Could not interpret output from ping command [14:52:16] PROBLEM - Host mw143 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:513CRITICAL - Could not interpret output from ping command [14:52:16] PROBLEM - Host matomo121 is DOWN: /bin/ping -6 -n -U -w 30 -c 5 2a10:6740::6:207CRITICAL - Could not interpret output from ping command [14:59:57] RECOVERY - Host swiftobject112 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [15:00:02] RECOVERY - Host db112 is UP: PING OK - Packet loss = 0%, RTA = 1.26 ms [15:00:02] RECOVERY - Host phab121 is UP: PING OK - Packet loss = 0%, RTA = 1.38 ms [15:00:07] RECOVERY - Host mw131 is UP: PING OK - Packet loss = 0%, RTA = 3.85 ms [15:00:12] RECOVERY - Host bast121 is UP: PING OK - Packet loss = 0%, RTA = 2.67 ms [15:00:17] RECOVERY - Host cp34 is UP: PING OK - Packet loss = 0%, RTA = 79.55 ms [15:00:21] RECOVERY - Host mw143 is UP: PING OK - Packet loss = 0%, RTA = 1.03 ms [15:00:26] RECOVERY - Host swiftproxy131 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [15:00:28] RECOVERY - Host matomo121 is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [15:00:32] RECOVERY - Host cloud10 is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms [15:00:36] RECOVERY - Host mw134 is UP: PING OK - Packet loss = 0%, RTA = 3.65 ms [15:00:36] RECOVERY - Host mwtask141 is UP: PING OK - Packet loss = 0%, RTA = 1.30 ms [15:00:38] RECOVERY - Host mail121 is UP: PING OK - Packet loss = 0%, RTA = 1.69 ms [15:00:39] RECOVERY - ping6 on swiftproxy131 is OK: PING OK - Packet loss = 0%, RTA = 1.61 ms [15:00:43] RECOVERY - Host db131 is UP: PING OK - Packet loss = 0%, RTA = 1.96 ms [15:00:45] RECOVERY - Host jobchron121 is UP: PING OK - Packet loss = 0%, RTA = 2.02 ms [15:00:48] RECOVERY - Host cloud13 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [15:00:49] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 104.70 ms [15:00:52] RECOVERY - ping6 on jobchron121 is OK: PING OK - Packet loss = 0%, RTA = 1.28 ms [15:00:53] RECOVERY - Host cloud12 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms [15:00:53] RECOVERY - Host os131 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [15:00:54] RECOVERY - Host prometheus131 is UP: PING OK - Packet loss = 0%, RTA = 1.01 ms [15:00:56] RECOVERY - Host mw132 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [15:00:58] RECOVERY - Host graylog131 is UP: PING OK - Packet loss = 0%, RTA = 1.19 ms [15:01:21] RECOVERY - ping6 on bast121 is OK: PING OK - Packet loss = 0%, RTA = 2.15 ms [15:01:22] RECOVERY - ping6 on graylog131 is OK: PING OK - Packet loss = 0%, RTA = 0.52 ms [15:01:25] RECOVERY - Host swiftobject122 is UP: PING OK - Packet loss = 0%, RTA = 2.16 ms [15:01:27] RECOVERY - Host test131 is UP: PING OK - Packet loss = 0%, RTA = 3.31 ms [15:01:28] RECOVERY - Host swiftobject121 is UP: PING OK - Packet loss = 0%, RTA = 1.93 ms [15:01:29] RECOVERY - ping6 on db112 is OK: PING OK - Packet loss = 0%, RTA = 3.04 ms [15:01:30] RECOVERY - ping6 on os131 is OK: PING OK - Packet loss = 0%, RTA = 2.80 ms [15:01:33] RECOVERY - Host mw142 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [15:01:34] RECOVERY - Host mem131 is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [15:01:35] RECOVERY - Host bast141 is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [15:01:35] RECOVERY - Host cloud14 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [15:01:38] RECOVERY - ping6 on cp34 is OK: PING OK - Packet loss = 0%, RTA = 79.91 ms [15:01:39] RECOVERY - Host mem141 is UP: PING OK - Packet loss = 0%, RTA = 1.99 ms [15:01:39] RECOVERY - Host db101 is UP: PING OK - Packet loss = 0%, RTA = 4.62 ms [15:01:39] RECOVERY - Host swiftobject113 is UP: PING OK - Packet loss = 0%, RTA = 1.66 ms [15:01:40] RECOVERY - ping6 on swiftobject113 is OK: PING OK - Packet loss = 0%, RTA = 2.94 ms [15:01:40] RECOVERY - ping6 on phab121 is OK: PING OK - Packet loss = 0%, RTA = 2.75 ms [15:01:40] RECOVERY - Host mw141 is UP: PING OK - Packet loss = 0%, RTA = 2.59 ms [15:01:41] RECOVERY - Host swiftobject101 is UP: PING OK - Packet loss = 0%, RTA = 4.30 ms [15:01:42] RECOVERY - Host cp35 is UP: PING OK - Packet loss = 0%, RTA = 82.79 ms [15:01:43] RECOVERY - Host db121 is UP: PING OK - Packet loss = 0%, RTA = 2.50 ms [15:01:43] RECOVERY - Host swiftobject111 is UP: PING OK - Packet loss = 0%, RTA = 2.58 ms [15:01:44] RECOVERY - Host sslhost is UP: PING OK - Packet loss = 0%, RTA = 0.10 ms [15:01:44] RECOVERY - Host reports121 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms [15:01:44] RECOVERY - ping6 on mw131 is OK: PING OK - Packet loss = 0%, RTA = 1.91 ms [15:01:45] RECOVERY - Host swiftproxy111 is UP: PING OK - Packet loss = 0%, RTA = 4.18 ms [15:01:45] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [15:02:00] RECOVERY - ping6 on matomo121 is OK: PING OK - Packet loss = 0%, RTA = 1.83 ms [15:02:05] RECOVERY - ping6 on cloud12 is OK: PING OK - Packet loss = 0%, RTA = 0.65 ms [15:02:15] RECOVERY - ping6 on cloud13 is OK: PING OK - Packet loss = 0%, RTA = 0.89 ms [15:02:50] PROBLEM - mon141 HTTPS on mon141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:03:15] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 2.17, 2.59, 3.11 [15:03:25] RECOVERY - ping6 on mw142 is OK: PING OK - Packet loss = 0%, RTA = 0.31 ms [15:03:35] RECOVERY - mon141 APT on mon141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:03:35] RECOVERY - ping6 on mon141 is OK: PING OK - Packet loss = 0%, RTA = 0.08 ms [15:06:40] RECOVERY - db131 MariaDB Connections on db131 is OK: OK connection usage: 4.8%Current connections: 24 [15:07:09] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 4%Current connections: 20 [15:07:14] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 1.8%Current connections: 9 [15:07:18] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 1%Current connections: 5 [15:07:54] RECOVERY - db101 MariaDB Connections on db101 is OK: OK connection usage: 1%Current connections: 5 [15:22:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f55297257a79...53afe07bf0eb [15:22:43] [02miraheze/puppet] 07paladox 0353afe07 - Update packages-microsoft-prod.deb [15:24:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/53afe07bf0eb...b5888e6dfd3e [15:24:14] [02miraheze/puppet] 07paladox 03b5888e6 - ircecho: add support for python 3.11 [15:27:28] RECOVERY - mon141 IRCEcho on mon141 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [15:27:31] RECOVERY - mon141 php-fpm on mon141 is OK: PROCS OK: 3 processes with command name 'php-fpm8.2' [15:27:32] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 43411 bytes in 0.074 second response time [15:27:38] PROBLEM - mon141 HTTPS on mon141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:27:45] PROBLEM - mon141 Puppet on mon141 is UNKNOWN: NRPE: Unable to read output [15:33:51] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:34:57] RECOVERY - mon141 icinga.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 302 Found - 272 bytes in 0.057 second response time [15:36:41] RECOVERY - mem141 NTP time on mem141 is OK: NTP OK: Offset -0.02597075701 secs [15:38:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b5888e6dfd3e...77335f45d994 [15:38:25] [02miraheze/puppet] 07paladox 0377335f4 - monitoring: pin mariadb dist to bullseye temporarily [15:42:17] PROBLEM - mon141 HTTPS on mon141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.089 second response time [15:43:21] !log test [15:43:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:44:15] PROBLEM - mon141 HTTPS on mon141 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:51:36] paladox: phab php is too new [15:51:46] 8.1 and 8.2 aren't fully supported yet [15:51:58] Phabricator isn't supported either [15:52:30] phorge has been actively hammering out fixes though, so we should consider switching [15:54:25] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17683 bytes in 0.103 second response time [15:58:17] I can do the switch [16:00:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b4258606a6cc [16:00:16] [02miraheze/puppet] 07paladox 03b425860 - Switch Phabricator to Phorge [16:00:17] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:00:23] [02puppet] 07paladox opened pull request 03#3457: Switch Phabricator to Phorge - 13https://github.com/miraheze/puppet/pull/3457 [16:01:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4258606a6cc...bf933ed30fcb [16:01:33] [02miraheze/puppet] 07paladox 03bf933ed - Update config.yaml [16:01:34] [02puppet] 07paladox synchronize pull request 03#3457: Switch Phabricator to Phorge - 13https://github.com/miraheze/puppet/pull/3457 [16:02:16] [02puppet] 07paladox closed pull request 03#3457: Switch Phabricator to Phorge - 13https://github.com/miraheze/puppet/pull/3457 [16:02:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/77335f45d994...48b8389c41fe [16:02:19] [02miraheze/puppet] 07paladox 0348b8389 - Switch Phabricator to Phorge (#3457) [16:02:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:02:26] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:06:26] MacFan4000: done [16:21:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1bcc98c8d552 [16:21:44] [02miraheze/puppet] 07paladox 031bcc98c - Phabricator: use phorge directory [16:21:47] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:21:49] [02puppet] 07paladox opened pull request 03#3458: Phabricator: use phorge directory - 13https://github.com/miraheze/puppet/pull/3458 [16:22:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1bcc98c8d552...7f303b5f40e9 [16:22:16] [02miraheze/puppet] 07paladox 037f303b5 - Update phab.miraheze.wiki.conf [16:22:17] [02puppet] 07paladox synchronize pull request 03#3458: Phabricator: use phorge directory - 13https://github.com/miraheze/puppet/pull/3458 [16:22:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7f303b5f40e9...dabf14a1d6a1 [16:22:31] [02miraheze/puppet] 07paladox 03dabf14a - Update phabricator.miraheze.org.conf [16:22:32] [02puppet] 07paladox synchronize pull request 03#3458: Phabricator: use phorge directory - 13https://github.com/miraheze/puppet/pull/3458 [16:22:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dabf14a1d6a1...6c82aca1c1fd [16:22:51] [02miraheze/puppet] 07paladox 036c82aca - Update phd.systemd.erb [16:22:53] [02puppet] 07paladox synchronize pull request 03#3458: Phabricator: use phorge directory - 13https://github.com/miraheze/puppet/pull/3458 [16:25:58] [02puppet] 07paladox closed pull request 03#3458: Phabricator: use phorge directory - 13https://github.com/miraheze/puppet/pull/3458 [16:26:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/puppet/compare/48b8389c41fe...d22d1755e341 [16:26:03] [02miraheze/puppet] 07paladox 03d22d175 - Phabricator: use phorge directory (#3458) [16:26:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:26:07] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [16:27:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d22d1755e341...eef3b5ba6b13 [16:27:37] [02miraheze/puppet] 07paladox 03eef3b5b - Fix [16:28:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eef3b5ba6b13...b2653198bff0 [16:28:09] [02miraheze/puppet] 07paladox 03b265319 - Fix [16:29:09] PROBLEM - phab121 phd on phab121 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [16:31:07] RECOVERY - phab121 phd on phab121 is OK: PROCS OK: 1 process with args 'phd' [17:26:25]