[00:01:09] PROBLEM - coffeewiki.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [00:15:46] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [00:16:15] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:18:50] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:22:10] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:24:41] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [00:25:00] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:25:41] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [00:27:05] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:30:25] PROBLEM - coffeewiki.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'coffeewiki.net' expires in 15 day(s) (Sat 29 Jun 2024 11:08:56 PM GMT +0000). [00:30:27] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:30:34] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/53bd8499aaad...d309a60393f4 [00:30:35] [02ssl] 07WikiTideSSLBot 03d309a60 - Bot: Update SSL cert for coffeewiki.net [00:30:39] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:06] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:34:07] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d309a60393f4...3b1147e86c6a [00:34:08] [02ssl] 07WikiTideSSLBot 033b1147e - Bot: Update SSL cert for osdev.wiki [00:51:06] RECOVERY - cities.simulz.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Wed 11 Sep 2024 05:23:17 PM GMT +0000. [00:51:53] RECOVERY - wiki.strangereons.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.strangereons.com' will expire on Wed 11 Sep 2024 04:53:59 PM GMT +0000. [00:59:51] RECOVERY - coffeewiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'coffeewiki.net' will expire on Wed 11 Sep 2024 11:30:27 PM GMT +0000. [01:02:16] RECOVERY - polcompball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'polcompball.wiki' will expire on Wed 11 Sep 2024 05:04:25 PM GMT +0000. [01:02:47] RECOVERY - alpha.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'alpha.sagan4.org' will expire on Wed 11 Sep 2024 06:04:55 PM GMT +0000. [01:05:56] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [01:06:05] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:06:51] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [01:06:54] RECOVERY - richterian.com - LetsEncrypt on sslhost is OK: OK - Certificate 'richterian.com' will expire on Wed 11 Sep 2024 10:09:00 PM GMT +0000. [01:07:35] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [01:09:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:09:01] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:09:37] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:50] RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Wed 11 Sep 2024 05:49:01 PM GMT +0000. [01:17:03] RECOVERY - www.clinitheque.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'www.clinitheque.fr' will expire on Wed 11 Sep 2024 10:18:31 PM GMT +0000. [01:37:40] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:38:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:24:27] PROBLEM - wiki.themarkside.com - reverse DNS on sslhost is WARNING: NXDOMAIN: The DNS query name does not exist: wiki.themarkside.com. [03:36:11] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:27:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:16:19] PROBLEM - wiki.eversd.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [06:45:26] RECOVERY - wiki.eversd.com - reverse DNS on sslhost is OK: SSL OK - wiki.eversd.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:51:43] PROBLEM - wiki.themarkside.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [07:08:24] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [11:36:16] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:08:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:24:58] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864160 seconds old and 134193 bytes [12:29:24] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:37:11] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:18:42] PROBLEM - wiki.overwood.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.overwood.xyz' expires in 15 day(s) (Sun 30 Jun 2024 12:56:35 PM GMT +0000). [13:18:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3b1147e86c6a...1edc5974cf6e [13:18:55] [02ssl] 07WikiTideSSLBot 031edc597 - Bot: Update SSL cert for wiki.overwood.xyz [13:27:43] PROBLEM - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tmyt105.leyhp.com' expires in 15 day(s) (Sun 30 Jun 2024 01:12:47 PM GMT +0000). [13:27:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1edc5974cf6e...ce5907732814 [13:27:56] [02ssl] 07WikiTideSSLBot 03ce59077 - Bot: Update SSL cert for wiki.tmyt105.leyhp.com [13:36:14] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [13:40:07] PROBLEM - wiki.jsal.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.jsal.org' expires in 15 day(s) (Sun 30 Jun 2024 01:12:57 PM GMT +0000). [13:40:18] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce5907732814...6e5eb8c6317e [13:40:21] [02ssl] 07WikiTideSSLBot 036e5eb8c - Bot: Update SSL cert for wiki.jsal.org [13:54:25] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [13:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:26] RECOVERY - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tmyt105.leyhp.com' will expire on Thu 12 Sep 2024 12:27:49 PM GMT +0000. [13:58:44] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:58:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:00:26] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [14:00:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:24] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:09] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [14:06:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:06:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:09:06] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [14:09:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:16:42] RECOVERY - wiki.overwood.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.overwood.xyz' will expire on Thu 12 Sep 2024 12:18:48 PM GMT +0000. [14:32:17] PROBLEM - rodzinka.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rodzinka.wiki' expires in 15 day(s) (Sun 30 Jun 2024 02:06:39 PM GMT +0000). [14:32:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6e5eb8c6317e...1a2b97c69356 [14:32:32] [02ssl] 07WikiTideSSLBot 031a2b97c - Bot: Update SSL cert for rodzinka.wiki [14:36:14] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:38:20] RECOVERY - wiki.jsal.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jsal.org' will expire on Thu 12 Sep 2024 12:40:12 PM GMT +0000. [15:01:04] RECOVERY - rodzinka.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rodzinka.wiki' will expire on Thu 12 Sep 2024 01:32:24 PM GMT +0000. [15:06:13] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [15:38:03] PROBLEM - tno.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tno.wiki' expires in 15 day(s) (Sun 30 Jun 2024 03:28:22 PM GMT +0000). [15:38:16] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1a2b97c69356...8a5678974b48 [15:38:17] [02ssl] 07WikiTideSSLBot 038a56789 - Bot: Update SSL cert for tno.wiki [15:40:06] PROBLEM - removededm.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'removededm.com' expires in 15 day(s) (Sun 30 Jun 2024 03:28:06 PM GMT +0000). [15:40:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8a5678974b48...47a422dc5f04 [15:40:23] [02ssl] 07WikiTideSSLBot 0347a422d - Bot: Update SSL cert for removededm.com [16:06:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:06:35] RECOVERY - tno.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tno.wiki' will expire on Thu 12 Sep 2024 02:38:10 PM GMT +0000. [16:27:18] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Tue 27 Aug 2024 03:54:13 PM GMT +0000. [16:38:17] RECOVERY - removededm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'removededm.com' will expire on Thu 12 Sep 2024 02:40:12 PM GMT +0000. [17:48:17] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=zhtardiswiki [17:48:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:48:28] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=gengbaikewiki [17:48:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:12] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [18:48:29] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [18:48:58] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [18:49:28] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [19:08:29] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19091 bytes in 0.064 second response time [19:08:58] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.035 second response time [19:09:28] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [20:05:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [20:35:04] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:08:21] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:37:05] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:33:27] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [22:33:49] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 37.52, 19.46, 7.91 [22:35:25] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:35:51] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:38:10] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [22:38:10] RECOVERY - db181 APT on db181 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [22:39:54] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.213 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [22:51:49] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.38, 4.14, 10.93 [22:53:49] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.24, 2.83, 9.63