[01:12:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:45:33] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.89 ms [01:47:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.82 ms [01:53:45] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.35 ms [01:55:46] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.43 ms [02:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:18:28] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:56:45] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.65 ms [04:00:49] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.66 ms [04:34:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.73 ms [04:36:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.47 ms [04:56:45] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wizardia.wiki' expires in 15 day(s) (Sun 26 May 2024 04:46:48 AM GMT +0000). [04:56:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d2ea40b7d959...b4923559014e [04:56:59] [02ssl] 07WikiTideSSLBot 03b492355 - Bot: Update SSL cert for wizardia.wiki [05:23:09] PROBLEM - wiki.yunachannel.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yunachannel.com' expires in 15 day(s) (Sun 26 May 2024 05:04:42 AM GMT +0000). [05:23:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b4923559014e...0c322da1ce2d [05:23:22] [02ssl] 07WikiTideSSLBot 030c322da - Bot: Update SSL cert for wiki.yunachannel.com [05:23:51] PROBLEM - yoshipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yoshipedia.com' expires in 15 day(s) (Sun 26 May 2024 04:54:13 AM GMT +0000). [05:25:47] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Thu 08 Aug 2024 03:56:51 AM GMT +0000. [05:45:29] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:51:48] RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Thu 08 Aug 2024 04:23:15 AM GMT +0000. [06:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:46:15] PROBLEM - wiki.kirbygang.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kirbygang.com' expires in 15 day(s) (Sun 26 May 2024 06:28:33 AM GMT +0000). [06:46:27] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0c322da1ce2d...7e5e3fe176b5 [06:46:28] [02ssl] 07WikiTideSSLBot 037e5e3fe - Bot: Update SSL cert for wiki.kirbygang.com [07:13:08] PROBLEM - wiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.coolstation.space' expires in 15 day(s) (Sun 26 May 2024 07:00:51 AM GMT +0000). [07:13:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7e5e3fe176b5...c84cfac906c6 [07:13:22] [02ssl] 07WikiTideSSLBot 03c84cfac - Bot: Update SSL cert for wiki.coolstation.space [07:15:30] RECOVERY - wiki.kirbygang.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kirbygang.com' will expire on Thu 08 Aug 2024 05:46:20 AM GMT +0000. [07:33:15] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:48:39] PROBLEM - internalwiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'internalwiki.coolstation.space' expires in 15 day(s) (Sun 26 May 2024 07:17:44 AM GMT +0000). [07:48:50] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c84cfac906c6...5432b495aaf6 [07:48:53] [02ssl] 07WikiTideSSLBot 035432b49 - Bot: Update SSL cert for internalwiki.coolstation.space [08:01:15] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [08:11:23] RECOVERY - wiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.coolstation.space' will expire on Thu 08 Aug 2024 06:13:14 AM GMT +0000. [08:21:21] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 180.16 ms [08:25:54] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:31:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.74 ms [08:37:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 182.63 ms [08:39:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.47 ms [08:46:39] RECOVERY - internalwiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Thu 08 Aug 2024 06:48:44 AM GMT +0000. [08:55:06] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [09:06:49] PROBLEM - db161 APT on db161 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:08:51] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 63 packages available for upgrade (30 critical updates). [09:24:51] PROBLEM - db151 Backups SQL on db151 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209668 seconds old and 121912 bytes [10:24:48] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [10:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:53:58] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [11:02:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:54:19] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:23:31] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [13:45:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:53:04] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [14:14:36] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:22:14] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [14:43:58] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:30:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.89 ms [15:32:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.89 ms [16:40:35] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.67 ms [16:50:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.56 ms [17:03:23] [02puppet] 07Universal-Omega pushed 031 commit to 03rename-shell [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cbbe7563763a [17:03:25] [02puppet] 07Universal-Omega 03cbbe756 - Rename shell user [17:03:28] [02puppet] 07Universal-Omega created branch 03rename-shell - 13https://github.com/miraheze/puppet [17:03:29] [02puppet] 07Universal-Omega opened pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852 [17:05:16] [02puppet] 07Universal-Omega pushed 031 commit to 03rename-shell [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cbbe7563763a...803d150bcecc [17:05:19] [02puppet] 07Universal-Omega 03803d150 - Rename [17:05:21] [02puppet] 07Universal-Omega synchronize pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852 [18:12:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.66 ms [18:16:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.43 ms [18:24:40] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.73 ms [18:32:50] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.48 ms [19:07:40] [02puppet] 07clarfonthey commented on pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852#issuecomment-2105132661 [19:09:48] [02puppet] 07RhinosF1 commented on pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852#issuecomment-2105135091 [19:10:17] [02puppet] 07RhinosF1 commented on pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852#issuecomment-2105135850 [19:14:57] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [19:32:39] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=darkangelwiki --search-recursively /home/universalomega/darkangelwiki (START) [19:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:20] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=darkangelwiki --search-recursively /home/universalomega/darkangelwiki (END - exit=0) [19:36:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:11] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=ghostmachinewiki --search-recursively /home/universalomega/ghostmachinewiki (START) [19:37:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:18] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=ghostmachinewiki --search-recursively /home/universalomega/ghostmachinewiki (END - exit=0) [19:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:13:45] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:16:20] [02puppet] 07Universal-Omega commented on pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852#issuecomment-2105215696 [20:38:49] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.50 ms [20:39:12] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.401 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:39:12] PROBLEM - bushcraftwiki.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bushcraftwiki.com reverse DNS resolves to any-in-2001-4860-4802-34--15.1e100.net [20:41:44] PROBLEM - bushcraftwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808BE8B69D140000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:303 [20:42:53] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.21 ms [20:43:08] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:04:15] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.14 ms [21:09:08] PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query. [21:10:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.93 ms [21:41:40] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=dcmultiversewiki --no-updates /home/universalomega/dcmultiversewiki_xml_5dc56b3d424835c081e4.xml (START) [21:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:53:12] [02dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3aef00981a79...e9a77febccca [21:53:14] [02dns] 07Universal-Omega 03e9a77fe - Add dcmultiversewiki.com zone [21:59:46] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/5432b495aaf6...c279f8a9209e [21:59:48] [02ssl] 07WikiTideSSLBot 03c279f8a - Bot: Add SSL cert for dcmultiversewiki.com [22:07:16] [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c279f8a9209e...d4ae3de901a2 [22:07:18] [02ssl] 07Universal-Omega 03d4ae3de - Readd dcmultiversewiki.com redirects [22:14:17] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:18:33] PROBLEM - dcmultiverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dcmultiverse.miraheze.org could not be found [22:18:38] PROBLEM - dc.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc.miraheze.org could not be found [22:27:39] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d51725875aaa...2915ead8be83 [22:27:42] [02mw-config] 07Universal-Omega 032915ead - Add some wiki configs [22:28:33] miraheze/mw-config - Universal-Omega the build passed. [22:32:09] !log [@mwtask181] starting deploy of {'config': True} to all [22:32:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:19] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [22:32:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:36] !log [@mwtask171] starting deploy of {'config': True} to all [22:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:37:45] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [22:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:35] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2915ead8be83...eb722ed3e703 [22:44:36] [02mw-config] 07Universal-Omega 03eb722ed - Remove wgCompressRevisions [22:45:33] miraheze/mw-config - Universal-Omega the build passed. [22:48:00] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eb722ed3e703...36c4e6c0580d [22:48:03] [02mw-config] 07Universal-Omega 0336c4e6c - Re-add ReplaceText (restricted for now) [22:48:57] miraheze/mw-config - Universal-Omega the build passed. [22:53:18] !log [@test151] starting deploy of {'config': True} to test151 [22:53:19] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [22:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:04] !log [@mwtask181] starting deploy of {'config': True} to all [23:02:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:14] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [23:02:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:45] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lgbtqia.wiki' expires in 15 day(s) (Sun 26 May 2024 10:58:04 PM GMT +0000). [23:07:34] !log [@mwtask171] starting deploy of {'config': True} to all [23:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:42] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [23:07:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:09] PROBLEM - lgbtqia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lgbtqia.wiki' expires in 15 day(s) (Sun 26 May 2024 10:58:04 PM GMT +0000). [23:24:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d4ae3de901a2...2385eb3e491d [23:24:21] [02ssl] 07WikiTideSSLBot 032385eb3 - Bot: Update SSL cert for lgbtqia.wiki [23:53:29] RECOVERY - lgbtqia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'lgbtqia.wiki' will expire on Thu 08 Aug 2024 10:24:14 PM GMT +0000.