[00:02:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:13:15] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.barengreza.my.id All nameservers failed to answer the query. [00:38:06] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mhe.my.id All nameservers failed to answer the query. [00:41:46] PROBLEM - wiki.barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.barengreza.my.id could not be found [01:06:49] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mhe.my.id could not be found [01:16:30] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:35:32] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mhe.my.id All nameservers failed to answer the query. [01:56:59] PROBLEM - wikislamica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikislamica.org' expires in 15 day(s) (Sun 19 May 2024 01:33:02 AM GMT +0000). [01:57:11] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9cae2fc68ade...712d594b22a8 [01:57:12] [02miraheze/ssl] 07WikiTideSSLBot 03712d594 - Bot: Update SSL cert for wikislamica.org [02:00:49] PROBLEM - wiki.ivy.cm - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ivy.cm' expires in 15 day(s) (Sun 19 May 2024 01:45:49 AM GMT +0000). [02:01:03] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/712d594b22a8...0abb71c606a1 [02:01:04] [02miraheze/ssl] 07WikiTideSSLBot 030abb71c - Bot: Update SSL cert for wiki.ivy.cm [02:04:14] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mhe.my.id could not be found [02:14:07] PROBLEM - wiki.mcjones.gay - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mcjones.gay' expires in 15 day(s) (Sun 19 May 2024 01:48:17 AM GMT +0000). [02:14:21] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0abb71c606a1...a97eecb5a3a8 [02:14:22] [02miraheze/ssl] 07WikiTideSSLBot 03a97eecb - Bot: Update SSL cert for wiki.mcjones.gay [02:24:48] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.09 ms [02:26:26] RECOVERY - wikislamica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikislamica.org' will expire on Thu 01 Aug 2024 12:57:05 AM GMT +0000. [02:30:08] RECOVERY - wiki.ivy.cm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ivy.cm' will expire on Thu 01 Aug 2024 01:00:58 AM GMT +0000. [02:32:57] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mhe.my.id All nameservers failed to answer the query. [02:33:02] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.96 ms [02:33:59] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [02:42:44] RECOVERY - wiki.mcjones.gay - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcjones.gay' will expire on Thu 01 Aug 2024 01:14:15 AM GMT +0000. [03:01:40] PROBLEM - wiki.mhe.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mhe.my.id could not be found [03:01:58] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:07:42] PROBLEM - db181 Backups SQL on db181 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209626 seconds old and 6541 bytes [03:14:15] PROBLEM - db161 Backups SQL on db161 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209715 seconds old and 7830 bytes [03:45:54] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:55:34] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.74 ms [03:57:37] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.13 ms [04:14:56] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [04:18:23] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 170.30 ms [04:20:26] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.62 ms [04:44:01] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:00:47] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.87 ms [05:06:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.29 ms [05:15:18] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.35 ms [05:18:32] PROBLEM - wiki.drycleaninglobby.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.drycleaninglobby.net' expires in 15 day(s) (Sun 19 May 2024 05:03:54 AM GMT +0000). [05:19:25] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.33 ms [05:24:35] yes [05:34:40] this is indicated at special:listgrouprights? [05:39:13] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.74 ms [05:53:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.89 ms [06:06:57] yes [06:07:09] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.55 ms [06:09:11] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.16 ms [06:16:21] hmm not sure if custom domains are migrated to cloudflare yet... [06:26:43] Custom domains using a CNAME can use cf-lb.miraheze.org according to @cosmicalpha [06:26:51] A few moments ago on phab [06:27:16] We can only have up to 100 domains on it right now until CF replies and ups this though [06:28:10] And I just said cf-lb to make it cleaner tbh, it could be on any *.miraheze.org except mw-lb to move it to cloudflare. [06:28:37] @rhinosf1 ^ [06:28:56] @cosmicalpha cool [06:29:33] Does that mean any of the wikis that CNAME to theirwiki.miraheze.org have auto moved ? [06:30:35] It means they don't work, I've manually made the ones that I know did that work by adding to SaaS but domains that move have to be added there first also. [06:31:03] @cosmicalpha can you add the wiki from the task? [06:31:10] I already did [06:31:15] Perfect [06:31:25] I will make an API script to auto add there though similar to the current ssl-certificate also [06:31:29] Didn't we stop moaning at people who weren't pointing at mw-lb [06:31:40] Yeah but I changed it back again lol [06:31:43] Cause I had that as a rule in icinga for a while [06:31:51] For basically this exact reason [06:31:53] Ah good [06:42:19] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:21:36] PROBLEM - wiki.themanaworld.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-11-a.gandi.net.', 'ns-196-c.gandi.net.', 'ns-27-b.gandi.net.'], 'CNAME': 'cf-lb.miraheze.org.'} [07:40:24] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:44:14] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.61 ms [07:48:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.06 ms [08:33:49] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.72 ms [08:35:51] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.42 ms [09:06:33] PROBLEM - wiki.themanaworld.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806B179CC6140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [10:34:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.48 ms [10:36:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.62 ms [10:37:28] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.57 ms [10:39:31] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 137.01 ms [10:40:53] [02CreateWiki] 07redbluegreenhat opened pull request 03#518: Make the `visibility` property private - 13https://github.com/miraheze/CreateWiki/pull/518 [10:44:50] miraheze/CreateWiki - redbluegreenhat the build has errored. [10:51:02] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 302.34 ms [10:51:20] [02mw-config] 07redbluegreenhat opened pull request 03#5551: T12012: Disable CreateWiki's special pages outside of the global wikis - 13https://github.com/miraheze/mw-config/pull/5551 [10:52:18] miraheze/mw-config - redbluegreenhat the build passed. [10:53:03] [02mw-config] 07redbluegreenhat edited pull request 03#5551: T12102: Disable CreateWiki's special pages outside of the global wikis - 13https://github.com/miraheze/mw-config/pull/5551 [10:53:05] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 297.43 ms [10:53:35] [02mw-config] 07redbluegreenhat closed pull request 03#5551: T12102: Disable CreateWiki's special pages outside of the global wikis - 13https://github.com/miraheze/mw-config/pull/5551 [10:53:38] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/36ab4e200aaf...e5664995fbb8 [10:53:38] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [10:53:39] [02miraheze/mw-config] 07redbluegreenhat 03e566499 - T12102: Disable CreateWiki's special pages outside of the global wikis (#5551) [10:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:48] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [10:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:53:59] !log [@test151] starting deploy of {'config': True} to test151 [10:54:00] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [10:54:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:33] miraheze/mw-config - redbluegreenhat the build passed. [11:04:33] RECOVERY - wiki.themanaworld.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.themanaworld.org' will expire on Thu 01 Aug 2024 09:55:34 AM GMT +0000. [11:07:22] !log [@mwtask171] starting deploy of {'config': True} to all [11:07:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:07:30] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [11:07:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:08:44] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 172.25 ms [11:10:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.40 ms [11:12:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:17:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:04:36] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.83 ms [12:06:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.01 ms [12:13:09] PROBLEM - opendatascot.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'opendatascot.org' expires in 15 day(s) (Sun 19 May 2024 12:03:50 PM GMT +0000). [12:13:21] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a97eecb5a3a8...042e7f406e41 [12:13:23] [02miraheze/ssl] 07WikiTideSSLBot 03042e7f4 - Bot: Update SSL cert for opendatascot.org [12:24:17] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.94 ms [12:26:20] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.46 ms [12:34:00] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:42:18] RECOVERY - opendatascot.org - LetsEncrypt on sslhost is OK: OK - Certificate 'opendatascot.org' will expire on Thu 01 Aug 2024 11:13:15 AM GMT +0000. [13:02:20] PROBLEM - graphite151 NTP time on graphite151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o