[00:04:32] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21425MiB (24% inode=98%); [00:05:04] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19356MiB (20% inode=98%); [00:05:08] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21399MiB (24% inode=98%); [00:27:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:28:46] Interesting, I thought Kafka jobs aren't seen by mw, is the 500 unclaimed alert from just Redis jobs? [00:29:14] Yes we went back to redis for most jobs temporarily [00:30:49] If you don't mind me askng [00:30:58] whats up with Kafka? [00:31:21] in the sense of why we moving to and why its not being read [00:34:23] Because of connection speed [00:34:29] I need to add a proxy [00:35:37] Aight so long term speed investment [00:49:38] Side note, is there a way to see how far along a image import is? Curious cause epic duel wiki has a actually scary amount of images and wondering:P assuming its not any trouble, of course! [01:56:11] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Fri 26 Apr 2024 01:50:24 AM GMT +0000). [02:03:57] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Fri 26 Apr 2024 01:50:24 AM GMT +0000). [02:04:50] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:16:56] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'orain.org' expires in 7 day(s) (Fri 26 Apr 2024 01:50:24 AM GMT +0000). [02:34:50] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:00:06] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 4 seconds old and 0 bytes [03:02:40] PROBLEM - db162 Backups SQL on db162 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864147 seconds old and 1218 bytes [03:37:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:48:31] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.01 ms [03:50:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 179.49 ms [04:03:22] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.11, 6.36, 5.19 [04:03:36] [02mw-config] 07anpang54 opened pull request 03#5541: Add configs for spacewiki - 13https://github.com/miraheze/mw-config/pull/5541 [04:05:18] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.23, 6.12, 5.23 [04:05:37] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 195.66 ms [04:11:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.00 ms [04:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:17:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.69 ms [04:19:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.89 ms [04:22:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:45:32] PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldtriggerwiki.com' expires in 15 day(s) (Sat 04 May 2024 05:18:57 AM GMT +0000). [05:45:44] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dc0bd216f5d1...c738c40df096 [05:45:47] [02miraheze/ssl] 07WikiTideSSLBot 03c738c40 - Bot: Update SSL cert for worldtriggerwiki.com [06:02:56] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:04:10] The import will probably say [06:04:18] From the screen it's running in [06:04:24] But that's for shell people only [06:14:03] RECOVERY - worldtriggerwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtriggerwiki.com' will expire on Wed 17 Jul 2024 04:45:38 AM GMT +0000. [06:18:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:22:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.12 ms [06:24:53] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.85 ms [06:30:56] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:51:52] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.38 ms [06:53:51] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.70 ms [06:57:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.60 ms [07:00:39] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query qadrishattari.xyz. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:03:52] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.80 ms [07:07:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.88 ms [07:09:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.84 ms [07:17:16] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:22:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.84, 6.53, 5.51 [07:24:33] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.01, 6.45, 5.62 [07:29:17] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:37:03] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [07:37:25] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [07:37:44] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [07:39:03] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [07:39:25] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19090 bytes in 0.059 second response time [07:39:44] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.039 second response time [07:47:09] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.50, 6.54, 6.07 [07:51:01] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.43, 6.46, 6.14 [07:57:48] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [08:06:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.50, 7.41, 6.50 [08:07:40] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:08:32] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.06, 6.38, 6.22 [08:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:16:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.56, 6.87, 6.56 [08:17:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:18:32] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.19, 6.72, 6.55 [08:22:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.57, 7.40, 6.92 [08:26:36] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:36:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.40, 7.49, 7.09 [08:40:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.78, 7.37, 7.13 [08:40:38] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=epicduelwikiwiki /home/macfan/images --search-recursively (END - exit=0) [08:40:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:44:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 14.01, 8.83, 7.65 [08:52:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.35, 7.26, 7.54 [08:54:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.20, 8.49, 7.93 [08:56:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.97, 7.86, 7.77 [08:58:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.24, 9.16, 8.25 [09:02:20] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:06:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.39, 7.86, 8.00 [09:07:20] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:07:37] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [09:08:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.76, 7.94 [09:10:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.53, 7.60, 7.85 [09:12:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.18, 8.75, 8.27 [09:13:36] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c738c40df096...30abe9d0b950 [09:13:37] [02miraheze/ssl] 07WikiTideSSLBot 0330abe9d - Bot: Add SSL cert for moddedminecraft.wiki [09:14:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.90, 7.42, 7.82 [09:14:49] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/c87437eb1d83...685b948daf0f [09:14:52] [02miraheze/dns] 07Reception123 03685b948 - Create neurologiahpm.com [09:16:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.07, 8.30, 8.11 [09:18:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.95, 7.69, 7.90 [09:21:46] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:30:43] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.97 ms [09:32:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.71 ms [09:36:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.99, 8.05, 7.55 [09:37:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:38:32] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.74, 7.35, 7.34 [09:40:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.61, 7.87, 7.51 [09:50:30] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.31, 7.70, 7.82 [09:50:33] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:54:26] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.29, 8.77, 8.23 [10:22:00] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.05, 7.17, 7.88 [10:23:58] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.57, 7.69, 7.98 [10:27:54] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.74, 7.62, 7.99 [10:29:53] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.83, 8.35, 8.20 [10:31:51] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.69, 7.82, 8.00 [10:37:45] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.79, 7.54, 7.70 [10:41:52] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=iceriawiki --deletewiki iceriawiki --delete (END - exit=0) [10:42:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:43:39] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=amokwiki --deletewiki amokwiki --delete (END - exit=0) [10:43:39] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.20, 7.49, 7.79 [10:43:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:45:38] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.97, 8.23, 8.03 [10:46:17] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=miscdumpingguideswiki --deletewiki miscdumpingguideswiki --delete (END - exit=0) [10:46:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:47:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.92, 7.80, 7.87 [10:47:57] !log DELETED and DROPPED for reset iceriawiki, amokwiki, miscdumpingguideswiki [10:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:49:35] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.80, 8.71, 8.15 [10:51:33] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.12, 7.86, 7.90 [10:53:31] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.23, 8.45, 8.09 [10:57:27] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.84, 7.57, 7.87 [10:59:25] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.42, 7.87, 7.93 [11:01:23] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.47, 7.87, 7.93 [11:03:22] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.67, 7.88, 7.91 [11:05:20] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.13, 6.56, 7.43 [11:07:18] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.40, 7.58, 7.72 [11:09:16] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.65, 7.33, 7.62 [11:10:18] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [11:11:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.95, 8.11, 7.88 [11:13:12] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.49, 7.93, 7.84 [11:15:10] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.24, 9.18, 8.28 [11:19:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.82, 7.83, 7.98 [11:21:06] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.39, 9.12, 8.41 [11:23:04] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.28, 7.47, 7.89 [11:23:48] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [11:25:02] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 12.06, 8.94, 8.35 [11:39:32] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:42:45] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.20, 7.58, 7.96 [11:46:42] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.42, 7.33, 7.74 [11:52:55] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [12:14:54] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/851cb6a310a1...1493b68dd2fa [12:14:55] [02miraheze/CreateWiki] 07translatewiki 031493b68 - Localisation updates from https://translatewiki.net. [12:14:57] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7bf21da78d8a...a285bb6bed65 [12:14:58] [02miraheze/ImportDump] 07translatewiki 03a285bb6 - Localisation updates from https://translatewiki.net. [12:15:00] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/0a8c9b2812e5...dc0a4f97edfe [12:15:01] [02miraheze/landing] 07translatewiki 03dc0a4f9 - Localisation updates from https://translatewiki.net. [12:15:03] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/0e549248d6ab...dbd5e0c31728 [12:15:05] [02miraheze/MirahezeMagic] 07translatewiki 03dbd5e0c - Localisation updates from https://translatewiki.net. [12:15:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.09 ms [12:15:55] miraheze/landing - translatewiki the build passed. [12:17:24] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.83 ms [12:18:19] miraheze/MirahezeMagic - translatewiki the build has errored. [12:22:49] miraheze/ImportDump - translatewiki the build passed. [12:23:26] miraheze/CreateWiki - translatewiki the build passed. [12:23:35] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:36] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:45] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:23:46] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:55] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:23:56] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:13] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:33:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:22] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.04 ms [12:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:32] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:33:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:42] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:23] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.53 ms [12:38:02] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:38:10] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:38:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:20] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:29] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:38:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:10:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.25, 6.77, 7.81 [13:12:40] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.50, 8.48, 8.29 [13:34:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.90, 6.52, 7.79 [13:40:40] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.02, 7.39, 7.73 [13:42:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.52, 7.62, 7.82 [13:55:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.86 ms [13:56:45] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=epicduelwikiwiki /home/macfan/wiki-www.epicduelwiki.com-20240414/www.epicduelwiki.com-20240414-history.xml --no-updates (START) [13:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:57:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.10 ms [13:58:40] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.60, 5.71, 6.68 [14:03:38] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.11 ms [14:04:40] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.09, 8.53, 7.51 [14:06:20] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/30abe9d0b950...95309b708930 [14:06:22] [02miraheze/ssl] 07WikiTideSSLBot 0395309b7 - Bot: Add SSL cert for neurologiahpm.com [14:08:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.58, 7.87, 7.46 [14:09:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.72 ms [14:11:04] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:12:40] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.47, 7.49, 7.38 [14:16:40] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.31, 7.59, 7.45 [14:28:39] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 15.46, 8.84, 7.79 [14:30:38] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.27, 7.78, 7.56 [14:32:38] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.82, 8.32, 7.81 [14:32:38] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [14:36:37] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.24, 7.61, 7.73 [14:38:37] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.41, 7.78, 7.76 [14:40:37] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.17, 7.18, 7.55 [14:42:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.83 ms [14:44:37] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.72, 7.71, 7.62 [14:48:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.16, 7.85, 7.73 [14:52:13] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.96 ms [14:54:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.29, 7.50, 7.46 [14:56:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.09, 6.87, 7.23 [14:58:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.69, 8.17, 7.65 [15:01:56] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:08:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.51, 7.52, 7.76 [15:09:34] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:09:41] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.07 ms [15:10:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.56, 8.18, 7.99 [15:13:45] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.15 ms [15:18:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.68, 7.24, 7.82 [15:24:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 8.20, 8.03 [15:26:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.40, 7.82, 7.90 [15:28:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.43, 7.75, 7.85 [15:29:46] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CreateWiki/maintenance/deleteWiki.php --wiki=retrorumblewiki --deletewiki retrorumblewiki --delete (END - exit=0) [15:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.39, 7.69, 7.82 [15:31:00] !log db181: DROP DATABASE retrorumblewiki; [15:31:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:32:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.24, 8.00, 7.90 [15:34:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.66, 7.14, 7.60 [15:38:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.25, 8.92, 8.19 [15:53:45] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [15:56:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.75 ms [15:58:31] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.33 ms [16:36:18] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.84 ms [16:40:22] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.37 ms [16:46:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.19, 7.45, 7.87 [16:50:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.23, 7.55, 7.75 [16:52:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.92 ms [16:54:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.36, 7.67, 7.77 [16:56:20] [02dns] 07MacFan4000 opened pull request 03#511: Removed unused zones - 13https://github.com/miraheze/dns/pull/511 [16:56:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.31, 8.10, 7.96 [16:56:44] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.90 ms [16:58:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.89, 7.57, 7.77 [17:01:45] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10226MiB (10% inode=98%); [17:02:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.24, 7.69, 7.72 [17:03:01] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/95309b708930...8e0422307f38 [17:03:04] [02miraheze/ssl] 07WikiTideSSLBot 038e04223 - Bot: Add SSL cert for wiki.lifesprogress.com [17:06:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.62, 7.99, 7.94 [17:08:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.86, 8.15, 8.01 [17:12:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.38, 7.36, 7.69 [17:14:36] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.62, 7.73 [17:16:36] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.53, 7.53, 7.68 [17:21:59] [02dns] 07Universal-Omega closed pull request 03#511: Removed unused zones - 13https://github.com/miraheze/dns/pull/511 [17:22:00] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-79/±0] 13https://github.com/miraheze/dns/compare/685b948daf0f...77e75c45b62e [17:22:01] [02miraheze/dns] 07MacFan4000 0377e75c4 - Removed unused zones (#511) [17:33:47] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [17:33:58] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.63, 7.52, 7.38 [17:39:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.94, 7.61, 7.47 [17:39:59] @cosmicalpha [17:40:42] Sorry pinged you because I was having trouble finding your profile (Discord is broken on mobile) but see DMs for a small issue with RemovePII [17:41:38] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.11, 8.38, 7.75 [17:43:33] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.02, 7.55, 7.52 [17:47:23] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.73, 7.73, 7.53 [17:49:18] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.60, 7.02, 7.28 [18:00:49] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.18, 6.06, 6.68 [18:02:18] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:35] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.06, 6.78, 6.84 [18:11:25] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.18, 6.25, 6.62 [18:28:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1493b68dd2fa...6ef766794d22 [18:28:14] [02miraheze/CreateWiki] 07Universal-Omega 036ef7667 - Actually add request-moredetails notification type [18:30:30] !log [reception@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [18:30:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:42] !log [reception@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 12s [18:30:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:36:14] [02CreateWiki] 07redbluegreenhat opened pull request 03#509: Cleanup RestWikiRequest - 13https://github.com/miraheze/CreateWiki/pull/509 [18:36:25] miraheze/CreateWiki - Universal-Omega the build passed. [18:41:51] miraheze/CreateWiki - redbluegreenhat the build has errored. [18:42:45] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#509: Cleanup RestWikiRequest - 13https://github.com/miraheze/CreateWiki/pull/509 [18:50:52] miraheze/CreateWiki - redbluegreenhat the build passed. [18:51:59] [02CreateWiki] 07redbluegreenhat closed pull request 03#509: Cleanup RestWikiRequest - 13https://github.com/miraheze/CreateWiki/pull/509 [18:52:01] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6ef766794d22...6cdd1c5b859d [18:52:04] [02miraheze/CreateWiki] 07redbluegreenhat 036cdd1c5 - Cleanup RestWikiRequest (#509) [18:52:51] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 [18:52:52] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [18:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:02] miraheze/CreateWiki - redbluegreenhat the build passed. [20:06:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:00:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03monitoring-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/82e629699252 [21:00:21] [02miraheze/puppet] 07Universal-Omega 0382e6296 - monitoring: use wikitide.net for SSL monitoring [21:00:22] [02puppet] 07Universal-Omega created branch 03monitoring-ssl - 13https://github.com/miraheze/puppet [21:00:26] [02puppet] 07Universal-Omega opened pull request 03#3836: monitoring: use wikitide.net for SSL monitoring - 13https://github.com/miraheze/puppet/pull/3836 [21:03:35] [02puppet] 07Universal-Omega synchronize pull request 03#3836: monitoring: use wikitide.net for SSL monitoring - 13https://github.com/miraheze/puppet/pull/3836 [21:03:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03monitoring-ssl [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/82e629699252...04cd7c0228d7 [21:03:40] [02miraheze/puppet] 07Universal-Omega 0304cd7c0 - Add more wc monitoring [21:03:57] [02puppet] 07Universal-Omega edited pull request 03#3836: monitoring: add more wildcard SSL monitoring - 13https://github.com/miraheze/puppet/pull/3836 [21:04:53] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [21:17:01] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=epicduelwikiwiki /home/macfan/wiki-www.epicduelwiki.com-20240414/www.epicduelwiki.com-20240414-history.xml --no-updates (END - exit=0) [21:17:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:02:56] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9704MiB (10% inode=98%); [22:03:22] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:04:13] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=epicduelwikiwiki (START) [22:04:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:10:17] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9708MiB (10% inode=98%); [23:16:18] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:17:31] PROBLEM - mwtask171 NTP time on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host [23:17:32] PROBLEM - mw172 Disk Space on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host [23:17:32] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:17:33] PROBLEM - phorge171 phd on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host [23:17:36] PROBLEM - cp37 Nginx Backend for mw161 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:17:36] PROBLEM - phorge171 conntrack_table_size on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host [23:17:37] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host [23:17:39] PROBLEM - ldap171 ferm_active on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host [23:17:39] PROBLEM - ldap171 NTP time on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host [23:17:39] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host [23:17:39] PROBLEM - phorge171 Disk Space on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host [23:17:45] PROBLEM - reports171 SSH on reports171 is CRITICAL: connect to address 10.0.17.139 and port 22: No route to host [23:17:46] PROBLEM - mw172 Puppet on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host [23:17:46] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw172.wikitide.net port 443 after 225 ms: Couldn't connect to server [23:17:48] PROBLEM - ldap171 conntrack_table_size on ldap171 is CRITICAL: connect to address 10.0.17.121 port 5666: No route to hostconnect to host 10.0.17.121 port 5666: No route to host [23:17:51] PROBLEM - swiftac171 conntrack_table_size on swiftac171 is CRITICAL: connect to address 10.0.17.125 port 5666: No route to hostconnect to host 10.0.17.125 port 5666: No route to host [23:17:51] PROBLEM - swiftac171 Swift Container Service on swiftac171 is CRITICAL: connect to address 10.0.17.125 and port 6001: No route to host [23:17:52] PROBLEM - bots171 PowerDNS Recursor on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host [23:17:52] PROBLEM - bots171 IRC RC Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host [23:17:52] PROBLEM - bots171 IRC Log Server Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host [23:17:54] PROBLEM - swiftac171 Backups Swift Account Container on swiftac171 is CRITICAL: connect to address 10.0.17.125 port 5666: No route to hostconnect to host 10.0.17.125 port 5666: No route to host [23:17:54] PROBLEM - swiftproxy171 NTP time on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host [23:17:54] PROBLEM - swiftproxy171 Swift Proxy on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 80: No route to host [23:17:54] PROBLEM - swiftproxy171 SSH on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 22: No route to host [23:17:55] PROBLEM - bots171 Current Load on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host [23:17:56] PROBLEM - jobchron171 SSH on jobchron171 is CRITICAL: connect to address 10.0.17.120 and port 22: No route to host [23:17:58] PROBLEM - bots171 IRC-Discord Relay Bot on bots171 is CRITICAL: connect to address 10.0.17.143 port 5666: No route to hostconnect to host 10.0.17.143 port 5666: No route to host [23:17:58] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:59] PROBLEM - swiftobject171 SSH on swiftobject171 is CRITICAL: connect to address 10.0.17.126 and port 22: No route to host [23:18:00] PROBLEM - swiftac171 PowerDNS Recursor on swiftac171 is CRITICAL: connect to address 10.0.17.125 port 5666: No route to hostconnect to host 10.0.17.125 port 5666: No route to host [23:18:00] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5627 bytes in 0.014 second response time [23:18:00] PROBLEM - mw171 Current Load on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host [23:18:00] PROBLEM - mw171 Puppet on mw171 is CRITICAL: connect to address 10.0.17.122 port 5666: No route to hostconnect to host 10.0.17.122 port 5666: No route to host [23:18:01] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: connect to address 10.0.17.122 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:18:12] PROBLEM - mw172 ferm_active on mw172 is CRITICAL: connect to address 10.0.17.123 port 5666: No route to hostconnect to host 10.0.17.123 port 5666: No route to host [23:18:12] PROBLEM - db171 SSH on db171 is CRITICAL: connect to address 10.0.17.119 and port 22: No route to host [23:18:13] PROBLEM - mwtask171 ferm_active on mwtask171 is CRITICAL: connect to address 10.0.17.144 port 5666: No route to hostconnect to host 10.0.17.144 port 5666: No route to host [23:18:13] PROBLEM - ping on swiftac171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.125) [23:18:14] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 7 backends are down. mw151 mw161 mw162 mw171 mw172 mw181 mw182 [23:18:14] PROBLEM - jobchron171 Disk Space on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host [23:18:15] PROBLEM - ns1 Puppet on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [23:18:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: connect to address 10.0.17.136 port 5666: No route to hostconnect to host 10.0.17.136 port 5666: No route to host [23:18:16] PROBLEM - swiftobject171 NTP time on swiftobject171 is CRITICAL: connect to address 10.0.17.126 port 5666: No route to hostconnect to host 10.0.17.126 port 5666: No route to host [23:18:16] PROBLEM - cp37 Nginx Backend for reports171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:17] PROBLEM - cp37 Puppet on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:17] PROBLEM - Host mwtask171 is DOWN: CRITICAL - Host Unreachable (10.0.17.144) [23:18:18] PROBLEM - Host bots171 is DOWN: CRITICAL - Host Unreachable (10.0.17.143) [23:18:18] PROBLEM - swiftproxy171 Puppet on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host [23:18:19] PROBLEM - phorge171 NTP time on phorge171 is CRITICAL: connect to address 10.0.17.124 port 5666: No route to hostconnect to host 10.0.17.124 port 5666: No route to host [23:18:19] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: connect to address 10.0.17.124 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:18:20] PROBLEM - jobchron171 PowerDNS Recursor on jobchron171 is CRITICAL: connect to address 10.0.17.120 port 5666: No route to hostconnect to host 10.0.17.120 port 5666: No route to host [23:18:20] PROBLEM - cp37 Nginx Backend for phorge171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:32] PROBLEM - Host jobchron171 is DOWN: CRITICAL - Host Unreachable (10.0.17.120) [23:18:32] PROBLEM - Host reports171 is DOWN: CRITICAL - Host Unreachable (10.0.17.139) [23:18:32] PROBLEM - ping on reports171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.139) [23:18:33] PROBLEM - Host swiftac171 is DOWN: CRITICAL - Host Unreachable (10.0.17.125) [23:18:34] PROBLEM - cp37 Nginx Backend for test151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:35] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mh142.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:18:35] PROBLEM - db171 conntrack_table_size on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [23:18:37] PROBLEM - ldap171 LDAP on ldap171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:37] PROBLEM - cp37 conntrack_table_size on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:37] PROBLEM - Host swiftobject171 is DOWN: CRITICAL - Host Unreachable (10.0.17.126) [23:18:38] PROBLEM - db171 Backups SQL mhglobal on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [23:18:39] PROBLEM - Host ns1 is DOWN: CRITICAL - Host Unreachable (10.0.17.136) [23:18:41] PROBLEM - db171 ferm_active on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [23:18:41] PROBLEM - db171 Backups SQL on db171 is CRITICAL: connect to address 10.0.17.119 port 5666: No route to hostconnect to host 10.0.17.119 port 5666: No route to host [23:18:46] PROBLEM - cp37 Nginx Backend for swiftproxy171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:46] PROBLEM - cp37 PowerDNS Recursor on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:46] PROBLEM - cp37 HTTPS on cp37 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp37.wikitide.net port 443 after 233 ms: Couldn't connect to server [23:18:48] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [23:18:48] PROBLEM - Host db171 is DOWN: CRITICAL - Host Unreachable (10.0.17.119) [23:18:49] PROBLEM - Host mw171 is DOWN: CRITICAL - Host Unreachable (10.0.17.122) [23:18:52] PROBLEM - swiftproxy171 HTTP on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 80: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:18:52] PROBLEM - cp37 NTP time on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:53] PROBLEM - Host ldap171 is DOWN: CRITICAL - Host Unreachable (10.0.17.121) [23:18:59] PROBLEM - cp37 Nginx Backend for mw172 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:59] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:18:59] PROBLEM - cp37 Nginx Backend for mwtask171 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:01] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 6 backends are down. mw151 mw152 mw171 mw172 mw181 mw182 [23:19:02] PROBLEM - cp37 Nginx Backend for mw152 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:05] PROBLEM - cp37 Nginx Backend for mon181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:12] PROBLEM - swiftproxy161 HTTP on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [23:19:14] PROBLEM - cp37 Current Load on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:14] PROBLEM - cp37 Nginx Backend for mwtask181 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:14] PROBLEM - cp37 Nginx Backend for swiftproxy161 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:16] PROBLEM - swiftproxy171 Disk Space on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host [23:19:17] PROBLEM - swiftproxy171 HTTPS on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [23:19:17] PROBLEM - swiftproxy171 memcached on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 and port 11211: No route to host [23:19:19] PROBLEM - swiftproxy171 ferm_active on swiftproxy171 is CRITICAL: connect to address 10.0.17.108 port 5666: No route to hostconnect to host 10.0.17.108 port 5666: No route to host [23:19:20] PROBLEM - cloud17 SSH on cloud17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:28] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2933 bytes in 3.258 second response time [23:19:28] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:19:29] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 4 backends are down. mw161 mw171 mw172 mw181 [23:19:30] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:30] PROBLEM - Host swiftproxy171 is DOWN: CRITICAL - Host Unreachable (10.0.17.108) [23:19:30] PROBLEM - ping on swiftproxy171 is CRITICAL: CRITICAL - Host Unreachable (10.0.17.108) [23:19:31] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:31] PROBLEM - cp37 Nginx Backend for matomo151 on cp37 is CRITICAL: connect to address 10.0.17.138 port 5666: No route to hostconnect to host 10.0.17.138 port 5666: No route to host [23:19:31] PROBLEM - Host cp37 is DOWN: CRITICAL - Host Unreachable (10.0.17.138) [23:19:38] PROBLEM - mh142.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mh142.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:19:44] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 5 backends are down. mw161 mw162 mw171 mw172 mw181 [23:19:45] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:46] PROBLEM - cloud17 conntrack_table_size on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:19:50] PROBLEM - cloud17 NTP time on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:19:50] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 6 backends are down. mw161 mw162 mw171 mw172 mw181 mw182 [23:19:53] PROBLEM - cloud17 PowerDNS Recursor on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:20:10] PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100% [23:20:22] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [23:20:42] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:22:41] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3782 bytes in 1.081 second response time [23:23:27] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:25:22] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:27:25] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3779 bytes in 0.901 second response time [23:27:25] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3783 bytes in 4.203 second response time [23:30:39] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3781 bytes in 1.238 second response time [23:31:41] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [23:32:03] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:33:26] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:36:33] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3783 bytes in 1.523 second response time [23:37:01] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:45:01] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3759 bytes in 6.798 second response time [23:45:10] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [23:45:23] RECOVERY - Host cp37 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [23:45:36] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 9% [23:45:44] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 47 minutes ago with 0 failures [23:45:57] RECOVERY - cp37 ferm_active on cp37 is OK: OK ferm input default policy is set [23:46:01] RECOVERY - cp37 SSH on cp37 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [23:46:15] RECOVERY - Host mwtask171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [23:46:22] RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [23:46:23] RECOVERY - ping on cp37 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms [23:46:32] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9009MiB (10% inode=98%); [23:46:39] RECOVERY - cp37 conntrack_table_size on cp37 is OK: OK: nf_conntrack is 0 % full [23:46:47] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2929 bytes in 0.644 second response time [23:47:02] PROBLEM - cp37 NTP time on cp37 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o