[00:40:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ea46e23ae09f...49598ff84b0d [00:40:20] [02miraheze/puppet] 07Universal-Omega 0349598ff - mediawiki-csp: add 1.1.1.1 to connect-src (T11867) [00:41:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/49598ff84b0d...d582115f5bae [00:41:35] [02miraheze/puppet] 07Universal-Omega 03d582115 - varnish-csp: add deprecated comment [00:52:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-csp [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/00809fca0239 [00:52:07] [02miraheze/puppet] 07Universal-Omega 0300809fc - varnish: remove Content-Security-Policy header [00:52:09] [02puppet] 07Universal-Omega created branch 03varnish-csp - 13https://github.com/miraheze/puppet [00:52:11] [02puppet] 07Universal-Omega opened pull request 03#3841: varnish: remove Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3841 [00:53:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-csp [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/00809fca0239...6a3fd73793e8 [00:53:45] [02miraheze/puppet] 07Universal-Omega 036a3fd73 - Remove [00:53:48] [02puppet] 07Universal-Omega synchronize pull request 03#3841: varnish: remove Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3841 [00:53:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03varnish-csp [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/6a3fd73793e8...f0e3850336df [00:54:01] [02miraheze/puppet] 07Universal-Omega 03f0e3850 - Remove [00:54:04] [02puppet] 07Universal-Omega synchronize pull request 03#3841: varnish: remove Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3841 [00:59:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d582115f5bae...669aefd9a4f3 [00:59:24] [02miraheze/puppet] 07Universal-Omega 03669aefd - nginx: remove unused module_path [01:05:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03nginx-csp [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2e7cdc42bbae [01:05:30] [02miraheze/puppet] 07Universal-Omega 032e7cdc4 - nginx: add minimalistic default Content-Security-Policy header [01:05:32] [02puppet] 07Universal-Omega created branch 03nginx-csp - 13https://github.com/miraheze/puppet [01:05:33] [02puppet] 07Universal-Omega opened pull request 03#3842: nginx: add minimalistic default Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3842 [01:06:42] [02puppet] 07Universal-Omega closed pull request 03#3842: nginx: add minimalistic default Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3842 [01:06:46] [02miraheze/puppet] 07Universal-Omega deleted branch 03nginx-csp [01:06:48] [02puppet] 07Universal-Omega deleted branch 03nginx-csp - 13https://github.com/miraheze/puppet [01:53:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 70.79, 37.98, 21.11 [02:02:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:17:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.73, 13.89, 22.86 [02:33:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 10.43, 11.94, 19.99 [02:55:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.07, 24.03, 19.93 [02:59:18] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.48, 22.86, 20.41 [03:00:34] RECOVERY - db162 Backups SQL on db162 is OK: FILE_AGE OK: /var/log/sql-backup.log is 17 seconds old and 1217 bytes [03:01:16] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.03, 20.20, 19.75 [03:13:15] [02mw-config] 07anpang54 synchronize pull request 03#5541: Add configs for spacewiki - 13https://github.com/miraheze/mw-config/pull/5541 [03:14:51] [02mw-config] 07anpang54 synchronize pull request 03#5541: Add configs for spacewiki - 13https://github.com/miraheze/mw-config/pull/5541 [03:21:53] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 22.59, 19.62 [03:23:51] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.71, 22.43, 19.96 [03:27:47] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.32, 18.43, 19.12 [03:32:47] miraheze/mw-config - anpang54 the build passed. [03:33:07] [02mw-config] 07Universal-Omega closed pull request 03#5541: Add configs for spacewiki - 13https://github.com/miraheze/mw-config/pull/5541 [03:33:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6c93ceace236...3bffeea5babe [03:33:12] [02miraheze/mw-config] 07anpang54 033bffeea - Add configs for spacewiki (#5541) [03:33:40] [02mw-config] 07Universal-Omega closed pull request 03#5533: Add MediaSpoiler (T11868) - 13https://github.com/miraheze/mw-config/pull/5533 [03:33:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3bffeea5babe...d4a8a4d514af [03:33:46] [02miraheze/mw-config] 07pixDeVl 03d4a8a4d - Add MediaSpoiler (T11868) (#5533) [03:34:01] miraheze/mw-config - Universal-Omega the build passed. [03:34:34] miraheze/mw-config - Universal-Omega the build passed. [03:35:16] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:36:58] !log [@mwtask171] starting deploy of {'config': True} to all [03:37:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:09] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [03:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:45] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.25 ms [03:53:40] !log [@test151] starting deploy of {'config': True} to test151 [03:53:41] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:55:48] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.57 ms [04:02:31] !log [@mwtask181] starting deploy of {'config': True} to all [04:02:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:43] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s [04:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:21:52] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.10, 21.46, 15.17 [04:22:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.02 ms [04:23:47] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 11.87, 17.84, 14.58 [04:24:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.43 ms [04:49:10] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw162 [04:49:25] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw162 [04:49:27] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw162 [04:49:45] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 38.21, 21.95, 14.72 [04:49:57] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:14] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw162 [04:51:54] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.195 second response time [04:52:54] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw162 [04:53:01] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw162 [04:54:06] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [04:54:49] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [04:55:01] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [04:55:08] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [04:55:25] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [04:55:26] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [04:55:43] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 7.33, 21.75, 18.48 [04:57:43] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 5.27, 16.14, 16.81 [06:02:04] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 21.48, 16.61 [06:04:02] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.60, 21.62, 17.24 [06:07:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.63, 23.92, 19.09 [06:33:50] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:38:10] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:41:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.44, 21.60, 23.79 [06:49:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 22.12, 22.92 [06:51:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.68, 22.72, 23.09 [06:55:11] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.99, 25.15, 23.91 [07:01:50] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:06:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:08:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.30, 22.09, 23.87 [07:18:47] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.88, 24.12, 23.72 [07:26:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.86, 22.92, 23.57 [07:30:55] PROBLEM - education.gced.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'education.gced.in' expires in 15 day(s) (Wed 08 May 2024 07:23:30 AM GMT +0000). [07:31:05] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b31e3c9862fb...dd56a0f2ca8e [07:31:07] [02miraheze/ssl] 07WikiTideSSLBot 03dd56a0f - Bot: Update SSL cert for education.gced.in [07:32:21] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:38:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 10.08, 16.37, 20.30 [07:50:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.53, 21.04, 19.34 [07:50:56] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.74 ms [07:52:10] PROBLEM - wiki.chicory.pizza - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.chicory.pizza' expires in 15 day(s) (Wed 08 May 2024 07:24:55 AM GMT +0000). [07:52:21] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dd56a0f2ca8e...5219053edbd3 [07:52:23] [02miraheze/ssl] 07WikiTideSSLBot 035219053 - Bot: Update SSL cert for wiki.chicory.pizza [07:52:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.19, 22.03, 19.95 [07:54:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.95, 24.61, 21.15 [07:54:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.09 ms [07:59:49] RECOVERY - education.gced.in - LetsEncrypt on sslhost is OK: OK - Certificate 'education.gced.in' will expire on Sun 21 Jul 2024 06:31:00 AM GMT +0000. [08:00:50] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:06:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 13.10, 20.45, 22.20 [08:08:14] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [08:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:12:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.88, 16.13, 19.60 [08:13:18] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.53 ms [08:15:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.67 ms [08:21:21] RECOVERY - wiki.chicory.pizza - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.chicory.pizza' will expire on Sun 21 Jul 2024 06:52:15 AM GMT +0000. [08:36:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [08:40:13] [02puppet] 07Universal-Omega closed pull request 03#3840: Add secupgrade.sh script - 13https://github.com/miraheze/puppet/pull/3840 [08:40:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/669aefd9a4f3...cdcc6f3a9376 [08:40:15] [02miraheze/puppet] 07redbluegreenhat 03cdcc6f3 - Add secupgrade.sh script (#3840) [09:19:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d4a8a4d514af...afcb40e96115 [09:19:40] [02miraheze/mw-config] 07Universal-Omega 03afcb40e - Fix link for wgCookieWarningMoreUrl (T12066) [09:20:34] miraheze/mw-config - Universal-Omega the build passed. [09:21:34] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [09:21:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:49] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [09:21:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:58] !log [@test151] starting deploy of {'config': True} to test151 [09:23:59] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.86, 23.13, 15.63 [09:35:10] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mhe.my.id could not be found [09:37:21] !log [@mwtask171] starting deploy of {'config': True} to all [09:37:40] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 19s [09:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:53:37] PROBLEM - wiki.mhe.my.id - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [09:54:05] !log [@test151] starting deploy of {'config': True} to test151 [09:54:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:02:58] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:28:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 13.05, 18.97, 23.31 [10:31:26] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [10:44:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.80, 19.26, 20.35 [10:57:18] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.17, 21.59, 20.70 [11:01:17] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.70, 21.17, 20.77 [11:13:02] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 22.39, 21.36 [11:15:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.42, 19.92, 20.57 [11:24:51] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.12, 23.21, 21.76 [11:25:29] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:48] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:26:49] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.97, 23.29, 22.00 [11:26:59] PROBLEM - mwtask181 mathoid on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 10044: Connection refused [11:27:03] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10001 milliseconds [11:27:36] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:02] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:29] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:36] PROBLEM - mwtask181 JobRunner Service on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:49] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:29] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:32] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:35] PROBLEM - mwtask181 conntrack_table_size on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:30:41] PROBLEM - mwtask181 NTP time on mwtask181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:32:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 21.15, 21.31 [11:33:12] PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864221 seconds old and 120114 bytes [11:34:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 9.46, 16.74, 19.69 [11:41:46] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [11:41:54] RECOVERY - mwtask181 NTP time on mwtask181 is OK: NTP OK: Offset -0.002946645021 secs [11:41:57] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.088 second response time [11:41:58] RECOVERY - mwtask181 conntrack_table_size on mwtask181 is OK: OK: nf_conntrack is 0 % full [11:42:20] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [11:42:25] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.435 second response time [11:42:43] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.352 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:42:59] RECOVERY - mwtask181 mathoid on mwtask181 is OK: TCP OK - 0.003 second response time on 10.0.18.106 port 10044 [11:43:24] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 19 processes with command name 'php-fpm8.2' [11:43:26] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 40 minutes ago with 0 failures [11:43:59] RECOVERY - mwtask181 JobRunner Service on mwtask181 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [11:44:00] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 141555MiB (63% inode=94%); [11:54:06] !log [@test151] starting deploy of {'config': True} to test151 [11:54:07] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [11:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:54:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:01:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 166.79 ms [12:05:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.85 ms [12:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:12:49] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6cdd1c5b859d...e1165acc7943 [12:12:50] [02miraheze/CreateWiki] 07translatewiki 03e1165ac - Localisation updates from https://translatewiki.net. [12:12:53] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/af311ede4500...d12817e45a71 [12:12:54] [02miraheze/RemovePII] 07translatewiki 03d12817e - Localisation updates from https://translatewiki.net. [12:12:56] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/0efc32e762a8...1a7af36a1c53 [12:12:59] [02miraheze/WikiDiscover] 07translatewiki 031a7af36 - Localisation updates from https://translatewiki.net. [12:13:01] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/dbd5e0c31728...5c1b2aa35f83 [12:13:03] [02miraheze/MirahezeMagic] 07translatewiki 035c1b2aa - Localisation updates from https://translatewiki.net. [12:17:04] miraheze/MirahezeMagic - translatewiki the build has errored. [12:17:34] miraheze/WikiDiscover - translatewiki the build passed. [12:19:40] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:21:10] PROBLEM - permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:21:42] PROBLEM - www.ebcat.me - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.ebcat.me' expires in 15 day(s) (Wed 08 May 2024 12:06:52 PM GMT +0000). [12:21:47] miraheze/RemovePII - translatewiki the build passed. [12:21:53] miraheze/CreateWiki - translatewiki the build passed. [12:21:56] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5219053edbd3...af7e5fcd1015 [12:21:58] [02miraheze/ssl] 07WikiTideSSLBot 03af7e5fc - Bot: Update SSL cert for www.ebcat.me [12:23:59] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:24:00] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:10] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:11] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:20] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:21] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:27:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.65, 9.80, 22.84 [12:31:13] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.65, 9.61, 19.78 [12:32:23] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:34] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 12s [12:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:45] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:32:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:54] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:36] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:34:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:37:56] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:38:06] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:13] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:38:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:16] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:25] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:38:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:50:24] RECOVERY - www.ebcat.me - LetsEncrypt on sslhost is OK: OK - Certificate 'www.ebcat.me' will expire on Sun 21 Jul 2024 11:21:50 AM GMT +0000. [12:50:36] RECOVERY - permanentfuturelab.wiki - reverse DNS on sslhost is OK: SSL OK - permanentfuturelab.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [13:02:08] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:03:40] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:06:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:18:48] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [13:38:10] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:44:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:48:19] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: Domain 'wikitide.net' was not found by the server [13:52:18] RECOVERY - cp41 PowerDNS Recursor on cp41 is OK: DNS OK: 0.365 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1 [13:52:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:06:10] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [14:28:53] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: No response from DNS ::1 [14:30:42] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.90 ms [14:32:43] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.47 ms [14:32:52] RECOVERY - cp41 PowerDNS Recursor on cp41 is OK: DNS OK: 2.354 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1 [14:42:42] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:05:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.89 ms [15:09:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.06 ms [15:39:40] !log [alex@mwtask181] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, apache2, apache2-bin, apache2-data, apache2-utils, uuid-dev, libuuid1, uuid-runtime, libblkid-dev, libblkid1, libmount-dev, libmount1, libfdisk1, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, [15:39:41] php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, xserver-common, xserver-xorg-core, and xvfb [15:39:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:56] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [15:43:10] RECOVERY - mw181 APT on mw181 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:43:34] !log [alex@mw181] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:43:35] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:43:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:52] !log [alex@mw182] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:51:53] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:52:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:12] RECOVERY - mw182 APT on mw182 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:53:32] RECOVERY - mw171 APT on mw171 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:53:56] !log [alex@mw171] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:53:57] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:54:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:15] !log [alex@mw172] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:55:16] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:32] RECOVERY - mw172 APT on mw172 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:57:04] !log [alex@mw161] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:57:05] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:22] RECOVERY - mw161 APT on mw161 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:58:46] !log [alex@mw162] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [15:58:47] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [15:58:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:44] RECOVERY - mw162 APT on mw162 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [15:59:59] RECOVERY - test151 APT on test151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [16:00:09] !log [alex@test151] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, apache2, apache2-bin, apache2-data, apache2-utils, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, [16:00:10] php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [16:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:48] !log [alex@mw151] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [16:01:49] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [16:01:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:00] RECOVERY - mw151 APT on mw151 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [16:03:03] !log [alex@mw152] Upgraded packages bsdutils, libsmartcols1, util-linux-extra, util-linux, mount, uuid-dev, libuuid1, libblkid-dev, libblkid1, libmount-dev, libmount1, xserver-xorg-core, xserver-common, libfdisk1, uuid-runtime, fdisk, bsdextrautils, eject, linux-libc-dev, php8.2-zip, php8.2-xml, php8.2-readline, php8.2-opcache, php8.2-mysql, php8.2-mbstring, [16:03:04] php8.2-ldap, php8.2-intl, php8.2-gmp, php8.2-gd, php8.2-cli, php8.2-fpm, php8.2-dba, php8.2-curl, php8.2-bz2, php8.2-bcmath, php8.2-common, util-linux-locales, and xvfb [16:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:20] RECOVERY - mw152 APT on mw152 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [16:08:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.90 ms [16:10:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.24 ms [16:38:11] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:43:28] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/af7e5fcd1015...2d04853da3cd [16:43:31] [02miraheze/ssl] 07WikiTideSSLBot 032d04853 - Bot: Add SSL cert for isocasg.org [17:18:04] PROBLEM - isocasg.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns71.domaincontrol.com.', 'ns72.domaincontrol.com.'], 'CNAME': None} [18:06:55] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:08:18] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.64 ms [18:10:18] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.23 ms [18:18:37] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [18:47:52] PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query. [19:12:14] RECOVERY - wiki.rlab.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rlab.org.uk' will expire on Sat 13 Jul 2024 06:01:16 PM GMT +0000. [19:15:09] RECOVERY - wiki.rlab.org.uk - reverse DNS on sslhost is OK: SSL OK - wiki.rlab.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME OK [19:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:08:41] what is going on here? [20:09:32] !log [@mwtask171] starting deploy of {'config': True} to all [20:09:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:09:43] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [20:09:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:07] that's me doing some overdue APT upgrades [20:24:21] oh its a diff server lol [20:24:27] thought it was the same server each time [21:41:16] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:09:35] !log [@mwtask171] starting deploy of {'config': True} to all [22:09:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:09:46] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [22:09:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:43] [02CreateWiki] 07redbluegreenhat opened pull request 03#510: Add a config variable to disable the REST API - 13https://github.com/miraheze/CreateWiki/pull/510 [22:34:20] miraheze/CreateWiki - redbluegreenhat the build passed. [22:39:18] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:40:25] PROBLEM - polcompball.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Wed 08 May 2024 10:33:50 PM GMT +0000). [22:42:57] PROBLEM - wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Wed 08 May 2024 10:33:50 PM GMT +0000). [22:58:45] PROBLEM - issue-tracker.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 15 day(s) (Wed 08 May 2024 10:33:50 PM GMT +0000). [23:36:50] PROBLEM - ldap171 PowerDNS Recursor on ldap171 is CRITICAL: Domain 'wikitide.net' was not found by the server [23:37:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:38:42] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:38:47] RECOVERY - ldap171 PowerDNS Recursor on ldap171 is OK: DNS OK: 2.182 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206