[00:05:59] PROBLEM - mon181 CVT Bot on mon181 is CRITICAL: PROCS CRITICAL: 1 process with args 'cvtbot' [00:11:59] RECOVERY - mon181 CVT Bot on mon181 is OK: PROCS OK: 2 processes with args 'cvtbot' [00:15:59] PROBLEM - mon181 CVT Bot on mon181 is CRITICAL: PROCS CRITICAL: 1 process with args 'cvtbot' [00:32:42] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [00:52:52] PROBLEM - zhacg.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zhacg.wiki could not be found [01:01:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:13:59] RECOVERY - mon181 CVT Bot on mon181 is OK: PROCS OK: 2 processes with args 'cvtbot' [01:17:59] PROBLEM - mon181 CVT Bot on mon181 is CRITICAL: PROCS CRITICAL: 0 processes with args 'cvtbot' [01:19:21] [02miraheze/landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/landing/compare/80a77b629020...ec60b4b370bb [01:19:24] [02miraheze/landing] 07AgentIsai 03ec60b4b - Add logo and change main page gradient [01:20:06] miraheze/landing - AgentIsai the build passed. [01:31:36] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:32:54] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [01:34:41] [02miraheze/landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/landing/compare/ec60b4b370bb...77ff59643f8f [01:34:43] [02miraheze/landing] 07AgentIsai 0377ff596 - Fix font and text colors [01:35:27] miraheze/landing - AgentIsai the build passed. [01:46:53] [02miraheze/landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/landing/compare/77ff59643f8f...0d4c11be4f8b [01:46:56] [02miraheze/landing] 07AgentIsai 030d4c11b - Fix CSS [01:47:43] miraheze/landing - AgentIsai the build passed. [01:51:21] [02miraheze/landing] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/0d4c11be4f8b...7c3d4113e9ea [01:51:22] [02miraheze/landing] 07AgentIsai 037c3d411 - Fix light mode [01:52:08] miraheze/landing - AgentIsai the build passed. [02:31:09] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [02:41:59] RECOVERY - mon181 CVT Bot on mon181 is OK: PROCS OK: 2 processes with args 'cvtbot' [02:46:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e474b487d4f...89706803cd2d [02:46:53] [02miraheze/puppet] 07Universal-Omega 038970680 - Add http proxy [02:46:55] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [02:47:59] PROBLEM - mon181 CVT Bot on mon181 is CRITICAL: PROCS CRITICAL: 3 processes with args 'cvtbot' [02:48:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/89706803cd2d...4e9d9c37bbfa [02:48:58] [02miraheze/puppet] 07Universal-Omega 034e9d9c3 - - [02:49:01] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [02:49:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4e9d9c37bbfa...3a846b4f3459 [02:49:42] [02miraheze/puppet] 07Universal-Omega 033a846b4 - IPv6 [02:49:44] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [02:50:22] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[cvtbot] [02:52:22] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:52:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3a846b4f3459...b3d111af49e7 [02:52:43] [02miraheze/puppet] 07Universal-Omega 03b3d111a - Use IPv6 [02:52:45] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [02:54:21] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b68b12e0a518...509d6249370e [02:54:22] [02miraheze/dns] 07AgentIsai 03509d624 - Add db162 and os162 [02:59:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3d111af49e7...3eea616c5d39 [02:59:34] [02miraheze/puppet] 07Universal-Omega 033eea616 - - [02:59:36] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [03:00:42] PROBLEM - db181 Backups SQL on db181 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209640 seconds old and 93 bytes [03:02:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:41:18] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/509d6249370e...ec1b69c463cb [03:41:19] [02miraheze/dns] 07AgentIsai 03ec1b69c - Add os162-private [03:41:32] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/6aaa1e2788b4...38bfd683b406 [03:41:33] [02miraheze/puppet] 07AgentIsai 0338bfd68 - Add db162 and os162 [03:47:27] PROBLEM - os162 Puppet on os162 is WARNING: Could not resolve hostname : Name or service not known [03:47:31] PROBLEM - db162 Disk Space on db162 is WARNING: Could not resolve hostname : Name or service not known [03:47:34] PROBLEM - db162 Puppet on db162 is WARNING: Could not resolve hostname : Name or service not known [03:47:34] PROBLEM - os162 APT on os162 is WARNING: Could not resolve hostname : Name or service not known [03:47:34] PROBLEM - db162 ferm_active on db162 is WARNING: Could not resolve hostname : Name or service not known [03:47:34] PROBLEM - Host db162 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [03:47:38] PROBLEM - os162 conntrack_table_size on os162 is WARNING: Could not resolve hostname : Name or service not known [03:47:45] PROBLEM - os162 Current Load on os162 is WARNING: Could not resolve hostname : Name or service not known [03:47:51] PROBLEM - os162 PowerDNS Recursor on os162 is WARNING: Could not resolve hostname : Name or service not known [03:48:03] PROBLEM - os162 SSH on os162 is UNKNOWN: Usage:check_ssh [-4|-6] [-t ] [-r ] [-p ] [03:48:06] PROBLEM - os162 Disk Space on os162 is WARNING: Could not resolve hostname : Name or service not known [03:48:10] PROBLEM - Host os162 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [03:56:01] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/38bfd683b406...dd5b0a18df2b [03:56:02] [02miraheze/puppet] 07AgentIsai 03dd5b0a1 - Add os162 as api host [04:01:39] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:17:29] RECOVERY - Host db162 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [04:17:32] PROBLEM - db162 MariaDB Connections on db162 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Host '2602:294:0:b12::110' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db162.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} t [04:17:32] in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: Host '2602:294:0:b12::110' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db162.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {ma [04:17:32] hrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [04:17:35] RECOVERY - db162 Puppet on db162 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [04:17:46] RECOVERY - Host os162 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [04:18:17] PROBLEM - db162 Backups SQL on db162 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [04:18:27] PROBLEM - db162 Backups SQL mhglobal on db162 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-mhglobal-backup-weekly.log [04:18:27] RECOVERY - os162 SSH on os162 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [04:18:52] PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Opensearch_template[graylog-internal] [04:19:02] PROBLEM - db162 MariaDB on db162 is CRITICAL: Host '2602:294:0:b12::110' is not allowed to connect to this MariaDB server [04:19:07] RECOVERY - db162 Disk Space on db162 is OK: DISK OK - free space: / 37705MiB (87% inode=97%); [04:19:07] RECOVERY - os162 Disk Space on os162 is OK: DISK OK - free space: / 437886MiB (98% inode=99%); [04:19:07] RECOVERY - db162 ferm_active on db162 is OK: OK ferm input default policy is set [04:19:07] RECOVERY - os162 APT on os162 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [04:19:22] RECOVERY - os162 conntrack_table_size on os162 is OK: OK: nf_conntrack is 0 % full [04:19:22] RECOVERY - os162 PowerDNS Recursor on os162 is OK: DNS OK: 0.032 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [04:19:27] RECOVERY - os162 Current Load on os162 is OK: LOAD OK - total load average: 0.01, 0.10, 0.16 [04:32:50] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:38:59] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/160380086bf8...e4a1818710a3 [04:39:00] [02miraheze/mw-config] 07AgentIsai 03e4a1818 - T11743: Install CirrusSearch in ManageWikiExtensions [04:39:56] miraheze/mw-config - AgentIsai the build passed. [04:40:12] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e4a1818710a3...e9d51e727181 [04:40:15] [02miraheze/mw-config] 07AgentIsai 03e9d51e7 - T11743: Add CirrusSearch cluster [04:40:41] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [04:40:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:50] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [04:40:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:41:13] miraheze/mw-config - AgentIsai the build passed. [04:53:11] !log [@test151] starting deploy of {'config': True} to test151 [04:53:12] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [04:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:58:40] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=metawiki (END - exit=256) [04:58:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:59:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [04:59:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:00:20] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:01:23] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:21:11] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ec1b69c463cb...e1280c3dca1a [05:21:12] [02miraheze/dns] 07AgentIsai 03e1280c3 - Add opensearch-mw.wikitide.net [05:26:53] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd5b0a18df2b...b35d819570af [05:26:54] [02miraheze/puppet] 07AgentIsai 03b35d819 - Add wikitide.net cert to wildcard [05:27:16] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b35d819570af...cc5dadac969f [05:27:18] [02miraheze/puppet] 07AgentIsai 03cc5dada - Add opensearch-mw to opensearch nginx [05:28:53] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc5dadac969f...cfbf88f5d236 [05:28:54] [02miraheze/puppet] 07AgentIsai 03cfbf88f - Remove deferred [05:29:15] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [05:30:48] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e9d51e727181...03a5fd8c8eba [05:30:49] [02miraheze/mw-config] 07AgentIsai 0303a5fd8 - Update CirrusSearch host [05:31:12] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:31:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:20] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [05:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:31:54] miraheze/mw-config - AgentIsai the build passed. [05:43:30] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 6.55, 5.50 [05:45:26] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.13, 5.98, 5.42 [05:53:00] !log [@test151] starting deploy of {'config': True} to test151 [05:53:01] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [05:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:18] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=metawiki --procs 24 (START) [05:53:19] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=metawiki --procs 24 (END - exit=0) [05:53:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:14] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.74, 6.47, 5.66 [06:01:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=metawiki --disable cirrussearch (END - exit=0) [06:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:14] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.43, 6.39, 5.73 [06:09:03] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:20:56] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.28, 6.98, 6.37 [06:21:12] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=metawiki --procs 24 (START) [06:21:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=metawiki --procs 24 (END - exit=0) [06:21:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:22:52] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.14, 6.67, 6.33 [06:26:46] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.87, 7.10, 6.60 [06:28:42] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.33, 7.69, 6.86 [06:30:37] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.67, 7.02, 6.70 [06:32:33] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.88, 6.12, 6.43 [06:37:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a15a72c46d47...b664e34654a6 [06:37:51] [02miraheze/ssl] 07MirahezeSSLBot 03b664e34 - Bot: Add SSL cert for wiki.sadboyzpod.com [06:40:12] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/e1280c3dca1a...78642f2cefb2 [06:40:15] [02miraheze/dns] 07Reception123 0378642f2 - add holocron.net zone [06:57:57] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [07:07:51] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.95, 7.52, 6.75 [07:08:20] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw161 [07:08:21] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 117.01, 65.60, 30.99 [07:09:46] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.73, 6.88, 6.61 [07:10:16] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 18 backends are healthy [07:13:38] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.13, 6.70, 6.60 [07:16:30] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [07:16:54] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [07:17:12] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161 [07:18:24] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.063 second response time [07:18:50] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 18 backends are healthy [07:19:10] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 18 backends are healthy [07:23:49] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.58, 20.35, 11.08 [07:24:19] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.90, 7.72, 7.03 [07:25:46] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 9.22, 15.36, 10.31 [07:26:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.91, 8.15, 7.27 [07:27:11] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [07:28:11] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 3.37, 10.49, 23.38 [07:32:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.39, 7.88, 7.53 [07:32:12] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 3.21, 6.68, 18.88 [07:33:28] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 31.24, 25.33, 17.06 [07:35:25] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 7.41, 18.44, 15.52 [07:38:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.19, 7.33, 7.32 [07:40:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.14, 6.51, 7.02 [07:41:12] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.63, 20.61, 16.77 [07:43:09] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 32.27, 23.84, 18.32 [07:45:06] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 12.76, 21.10, 18.12 [07:45:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:46:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.49, 6.01, 6.58 [07:47:00] !log sudo -u www-data php /srv/mediawiki/1.40/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki metawiki [07:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:03] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 4.55, 15.36, 16.38 [07:52:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.55, 6.66, 6.61 [07:54:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.77, 7.17, 6.78 [07:56:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.59, 7.45, 6.94 [07:58:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.88, 8.09, 7.22 [08:00:03] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.84, 6.95, 6.89 [08:02:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.69, 5.97, 6.55 [08:12:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.19, 7.71, 7.00 [08:14:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.29, 7.55, 7.03 [08:23:17] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:24:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.84, 8.17, 7.36 [08:26:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.23, 7.83, 7.34 [08:44:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.45, 5.43, 6.46 [08:48:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.28, 6.81, 6.82 [08:50:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.20, 8.03, 7.23 [08:52:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.64, 7.19 [08:52:28] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:56:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.41, 6.18, 6.68 [09:00:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.58, 7.63, 7.21 [09:02:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.55, 7.60, 7.22 [09:04:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.47, 7.22, 7.14 [09:10:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.26, 7.39, 7.15 [09:12:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.83, 7.21, 7.10 [09:14:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.62, 7.84, 7.33 [09:16:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.28, 7.06, 7.10 [09:20:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.15, 6.07, 6.68 [09:26:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.63, 7.34, 7.13 [09:32:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.75, 8.15, 7.51 [09:34:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.46, 7.78, 7.45 [09:40:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.03, 7.48, 7.34 [09:44:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.78, 7.89, 7.56 [09:46:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.38, 8.05, 7.66 [09:48:10] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:52:02] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.64, 7.18, 7.47 [10:20:24] Agent, CosmicAlpha, MacFan4000, paladox: major outage - cloud18 [11:14:16] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.63, 8.03, 7.91 [11:14:40] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [11:16:24] PROBLEM - bast181 PowerDNS Recursor on bast181 is CRITICAL: Domain 'miraheze.org' was not found by the server [11:16:25] PROBLEM - bast161 Puppet on bast161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:25] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:26] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:26] PROBLEM - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yuanpi.eu.org' expires in 15 day(s) (Sun 18 Feb 2024 10:10:37 AM GMT +0000). [11:16:26] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:26] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet last ran 1 hour ago [11:16:27] PROBLEM - puppet181 Puppet on puppet181 is WARNING: WARNING: Puppet last ran 1 hour ago [11:16:27] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet last ran 1 hour ago [11:16:27] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:28] PROBLEM - graylog161 Puppet on graylog161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:28] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:29] PROBLEM - os151 Puppet on os151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:29] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:30] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:30] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:44] PROBLEM - reports171 Puppet on reports171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:44] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet last ran 1 hour ago [11:16:47] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:48] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:50] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:51] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:53] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:55] PROBLEM - db162 Puppet on db162 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:57] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:58] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:16:59] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:02] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:05] PROBLEM - os161 Puppet on os161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:09] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:12] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:12] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:14] PROBLEM - swiftac171 Puppet on swiftac171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:14] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:19] PROBLEM - swiftobject161 Puppet on swiftobject161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:21] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:22] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:24] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:17:25] PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet last ran 1 hour ago [11:18:14] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.24, 7.76, 7.90 [11:18:24] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:34] RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 1.82, 5.36, 2.91 [11:18:47] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:20:54] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:21:22] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 47 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_IRC-Discord-Relay] [11:22:17] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [11:22:53] RECOVERY - db162 Puppet on db162 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [11:23:09] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [11:24:26] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [11:26:21] RECOVERY - reports171 Puppet on reports171 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [11:26:43] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:26:53] RECOVERY - swiftac171 Puppet on swiftac171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:27:08] RECOVERY - swiftobject161 Puppet on swiftobject161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:28:08] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 30 seconds ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.40],Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_3d2png] [11:28:10] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:28:18] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:28:33] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 1 minute ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.40],Exec[git_pull_femiwiki-deploy-1.41],Exec[git_pull_3d2png] [11:28:53] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [11:28:56] RECOVERY - os161 Puppet on os161 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [11:29:50] RECOVERY - bast161 Puppet on bast161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:29:50] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:30:03] RECOVERY - graylog161 Puppet on graylog161 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [11:30:44] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:31:02] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:32:01] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [11:32:07] RECOVERY - swiftproxy161 Puppet on swiftproxy161 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [11:32:11] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.11, 8.30, 7.56 [11:32:20] RECOVERY - swiftproxy171 Puppet on swiftproxy171 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [11:33:52] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 1 minute ago with 4 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl],Exec[git_pull_puppet],Exec[git_pull_ssl],Exec[git_pull_mediawiki-repos] [11:34:14] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:35:00] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:36:09] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [11:36:11] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [11:36:58] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:40:10] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.93, 7.81, 7.77 [11:42:10] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.03, 8.55, 8.05 [11:43:27] PROBLEM - bast181 NTP time on bast181 is CRITICAL: connect to address 10.0.18.101 port 5666: Connection refusedconnect to host 10.0.18.101 port 5666: Connection refused [11:44:07] RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 0.289 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:44:10] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.70, 8.00, 7.92 [11:44:15] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:44:24] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [11:45:26] PROBLEM - bast181 NTP time on bast181 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o