[00:03:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.22, 20.27, 18.57 [00:05:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.37, 18.70, 18.19 [00:09:59] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 186.61 ms [00:16:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.43, 21.61, 23.42 [00:20:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 22.78, 23.50 [00:22:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.61, 21.66, 22.97 [00:23:14] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 182.72 ms [00:25:17] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 181.65 ms [00:25:44] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [00:26:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 24.19, 23.70 [00:29:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.32, 21.77, 18.34 [00:31:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.73, 20.77, 18.43 [00:33:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.99, 18.68, 17.92 [00:42:25] [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:47:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:02:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.77, 22.64, 23.88 [01:04:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.82, 24.98, 24.59 [01:08:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.29, 18.59, 14.30 [01:08:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.14, 17.91, 14.66 [01:08:33] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 18.79, 14.28 [01:08:34] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 18.42, 14.06 [01:08:52] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw171 mw181 [01:09:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 21.87, 18.47 [01:09:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 18.54, 13.18 [01:09:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.98, 19.15, 14.77 [01:10:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.92, 21.86, 16.51 [01:11:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 21.60, 16.15 [01:11:44] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw172 [01:12:31] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.81, 21.88, 16.55 [01:12:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [01:12:33] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.69, 21.38, 16.33 [01:13:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.49, 21.37, 16.80 [01:13:45] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [01:14:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.53, 21.69, 17.28 [01:14:31] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.24, 22.70, 17.51 [01:14:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.42, 23.17, 23.93 [01:15:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.56, 21.47, 16.10 [01:15:39] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [01:16:30] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.74, 23.08, 18.35 [01:16:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 22.82, 18.05 [01:16:32] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [01:16:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.09, 24.14, 24.19 [01:16:51] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [01:17:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 22.84, 17.23 [01:17:42] 502 and icinga flaring [01:17:45] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [01:18:09] back maybe [01:18:32] back but slow [01:19:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.70, 18.83 [01:20:08] fuck openAI [01:20:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 23.88, 19.66 [01:20:49] [1/2] doesn't look good [01:20:49] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1278887116138479656/lij5vmI.png?ex=66d26ff1&is=66d11e71&hm=b01ea89643e185647e3084922ff9814184778a215066c7a80e61b56dfcd76f57& [01:21:20] jesus fuck [01:22:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.14, 23.18, 19.98 [01:22:25] [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:22:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.78, 22.59, 19.83 [01:23:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.02, 21.19, 18.42 [01:23:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.95, 23.06, 20.08 [01:24:29] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.80, 20.38, 19.24 [01:25:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.33, 18.49, 17.77 [01:25:25] over 1k unclaimed jobs [01:26:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.38, 19.57, 19.15 [01:26:25] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.39, 20.13, 19.37 [01:26:33] recovering says icinga [01:27:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.92, 23.40, 22.39 [01:29:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.35, 25.74, 23.38 [01:29:05] oh god [01:29:09] almost 2k jobs [01:29:21] 182 is going crit [01:29:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.10, 20.31, 19.70 [01:30:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.99, 19.41, 19.46 [01:31:00] @elliethearchiver which panel is that again [01:36:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.63, 20.77, 19.96 [01:37:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:38:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 22.27, 20.61 [01:41:10] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 22.14, 20.08 [01:41:22] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 22.10, 20.01 [01:42:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.65, 22.92, 20.57 [01:43:06] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.92, 22.53, 20.47 [01:43:21] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.46, 20.80, 19.82 [01:44:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.19, 20.53, 20.01 [01:44:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 14.67, 21.00, 20.93 [01:45:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 11.82, 17.49, 18.71 [01:46:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 10.17, 17.10, 18.83 [01:46:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.70, 18.60, 20.03 [01:46:56] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.64, 17.90, 19.10 [01:47:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.41, 22.43, 23.84 [01:55:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.54, 15.49, 19.94 [01:55:36] late reply but https://grafana.wikitide.net/d/GtxbP1Xnk/mediawiki?orgId=1&viewPanel=24 [01:55:42] oh wait it all looks normal again [01:56:03] actually the least loaded i've seen those [01:57:24] huh [01:57:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:57:56] https://cdn.discordapp.com/attachments/808001911868489748/1278896457520386168/uQQQ6OG.png?ex=66d278a4&is=66d12724&hm=fbf64a38b18acc39eb0f5e0e48bf6234f1265d36a6ad760b83e0057ffc11fcc5& [02:21:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 22.93, 20.06 [02:22:25] [Grafana] FIRING: The mediawiki job queue has more than 1000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:23:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.63, 19.44, 19.14 [02:31:54] [02mw-config] 07hurohukidaikon commented on pull request 03#5643: Update UploadWizard settings for kagagawiki - 13https://github.com/miraheze/mw-config/pull/5643#issuecomment-2319720278 [02:42:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:44:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.63, 22.08, 23.75 [02:50:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.59, 23.25 [02:56:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.85, 23.32, 23.60 [03:00:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 24.23, 23.77 [03:17:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.92, 22.51, 19.14 [03:17:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.60, 16.28, 12.41 [03:17:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.18, 19.24, 14.78 [03:18:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 21.10, 15.89 [03:18:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.73, 21.44, 16.31 [03:18:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.63, 20.28, 15.91 [03:18:32] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [03:18:39] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [03:18:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw181 mw182 [03:18:47] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 21.10, 16.07 [03:18:52] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 5 backends are down. mw152 mw161 mw172 mw181 mw182 [03:19:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.72, 17.93, 13.54 [03:20:32] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:20:39] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [03:20:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [03:21:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.52, 21.17, 16.74 [03:22:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.96, 23.08, 18.18 [03:23:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 22.83, 17.92 [03:23:51] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [03:24:07] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181 [03:24:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.27, 23.26, 18.59 [03:24:52] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:26:07] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [03:26:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.60, 24.81, 19.88 [03:27:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 30.03, 24.92, 18.40 [03:27:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.32, 23.54, 19.39 [03:28:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.56, 23.34, 19.90 [03:30:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.54, 25.41, 20.93 [03:30:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.58, 24.94, 20.91 [03:31:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.52, 21.93, 18.91 [03:32:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.90, 23.74, 20.88 [03:32:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.16, 22.52, 20.52 [03:32:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.10, 21.87, 20.22 [03:32:42] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.68, 21.72, 20.53 [03:33:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.49, 18.42, 18.00 [03:33:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.34, 20.13, 19.34 [03:34:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.99, 20.36, 19.98 [03:34:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.90, 20.31, 19.85 [03:34:42] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.26, 19.92, 20.01 [03:36:12] This is normal with 10k wikis tbh, it isn't normal that the same jobs stay queued and never run, the real metric should be how long the backlog is IE how long it takes a new job to actually run, rather than just 1k queued jobs, which can be normal with 10k wikis, some high traffic with a lot of job usage in like SMW wikis. [03:37:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:38:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.80, 18.96, 19.78 [03:39:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.24, 21.66, 22.74 [03:42:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:43:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 22.73, 22.80 [03:45:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 20.82, 19.27 [03:47:37] perhaps temporarily use test151 as an overflow [03:47:51] or is that not properly equipped to handle stuff rn [03:49:09] No, we are trying to completely seperate beta (including db and everything else) from prod, so we won't be using test151 for anything in prod. [03:50:03] ah [03:50:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.88, 22.09, 20.48 [03:50:23] figured as much, just suggested it on the off chance it could be used as an overflow temporarily [03:50:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.69, 20.04, 19.18 [03:50:40] If that became needed we would add more servers we have the resources for many more, but it isn't a capacity issue it is something else. [03:50:50] ah [03:51:26] is it possible to look at the logs of an individual “stuck” job because that might help determine the issue [03:51:40] sorry if I’m just suggesting stuff you’ve already tried 17 times [03:51:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.76, 22.13, 20.51 [03:52:05] Jobs don't have very good logging capabilities [03:52:14] ah [03:52:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.47, 20.58, 20.11 [03:52:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.26, 18.81, 18.80 [03:52:32] They are very hard to debug sometimes. [03:52:44] is there anything preventing them from having proper logs or are they just not set up to have proper logs [03:53:11] Adding another jobrunner actually theoretically may help. Maybe I should consider that or at least look into if it will likely help [03:53:32] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [03:53:34] Just aren't setup that way. [03:53:38] ah [03:53:48] The way they work with redis and MediaWiki [03:54:02] and I’m assuming it’s just outright impossible to attempt to kill and restart jobs if they take too long [03:54:04] Probably more redis [03:54:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.33, 18.05, 19.28 [03:54:29] Individual ones I think can't be done yeah. [03:54:40] or just not something that should be done for one reason or another [03:54:53] They aren't designed to be managed individually but rather in batches [03:55:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.94, 23.06, 23.95 [03:55:05] makes sense [03:55:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.03, 17.79, 19.21 [03:56:16] and you can’t just like cherry pick jobs if they become unresponsive, put them into an “unresponsive” batch, and then kill that whole batch [03:56:29] We did actually use test131 as an overflow buffer when we were still on the ServerChoice VMs because it got so bad and we had no more resources but now we won't again and we have 4x the resources we used to have. [03:57:20] Its not easy, by the time you try they eventually just go through or get abandoned. [03:57:27] ah [03:57:48] I’m going to blame it on SMW because 9/10 times it’s SMW’s fault [03:58:00] so that is the most likely option [03:58:02] It very well could be SMW lol [03:58:07] SMW has a lot of jobs [03:58:46] Maybe we should launch a jobqueue only for SMW so only SMW jobs run on it and other servers per queue. I wonder if that will help [03:59:04] for every weird smw bug no one can figure out there’s one job in the queue [03:59:30] that’s the new metric, I’ve decided [04:05:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.06, 22.60, 22.80 [04:07:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.49, 22.83, 22.90 [04:07:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:13:35] The alert bar was originally 500 jobs, but I raised that as it was just too low for 9k wikis [04:19:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.35, 23.97, 22.68 [04:21:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 20.30, 17.81 [04:22:05] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 21.36, 17.98 [04:22:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.52, 21.83, 19.29 [04:23:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.69, 23.74, 23.05 [04:23:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.66, 18.66, 17.52 [04:24:04] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.72, 20.83, 18.27 [04:24:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.82, 19.36, 18.68 [04:26:03] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.75, 17.91, 17.54 [04:31:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 10.21, 15.30, 19.39 [04:44:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.15, 22.39, 23.67 [04:50:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.82, 21.27, 22.50 [04:52:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.32, 20.33, 21.95 [04:54:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.01, 21.61, 22.17 [04:56:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.18, 22.43, 22.43 [05:04:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.67, 16.84, 19.99 [05:10:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.29, 23.79, 21.86 [05:12:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:12:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.58, 23.98, 22.21 [05:14:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.27, 25.81, 23.08 [05:25:52] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.21, 21.85, 18.99 [05:26:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.29, 20.25, 16.05 [05:27:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 18.78, 13.41 [05:27:35] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.96, 20.29, 15.77 [05:27:47] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.47, 24.17, 20.16 [05:28:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.05, 21.03, 15.84 [05:28:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [05:28:52] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw162 mw181 [05:29:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.53, 21.72, 16.83 [05:29:59] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 21.40, 16.30 [05:30:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.83, 23.75, 18.47 [05:30:32] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw161 mw182 [05:30:39] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.11, 23.56, 18.03 [05:30:51] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [05:31:37] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw151 mw152 mw181 [05:31:39] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.50, 22.58, 20.46 [05:32:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 23.66, 23.39, 18.98 [05:32:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [05:34:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.77, 24.83, 20.04 [05:34:32] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [05:35:31] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 23.24, 21.17 [05:35:31] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [05:36:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.31, 23.29, 19.21 [05:37:00] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 46% [05:37:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.99, 23.45, 21.51 [05:38:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.65, 23.20, 23.93 [05:39:23] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.44, 24.56, 22.15 [05:40:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.54, 24.98, 20.77 [05:40:57] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 36% [05:41:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 14.99, 21.75, 19.23 [05:41:28] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.69, 21.83, 19.95 [05:41:55] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.35, 21.98, 20.09 [05:42:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.00, 23.45, 21.45 [05:42:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.95, 23.09, 20.58 [05:42:34] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.41, 23.80, 21.71 [05:42:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.14, 23.79, 23.89 [05:43:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.06, 18.93, 18.49 [05:43:55] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.96, 20.02, 19.61 [05:44:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.25, 20.25, 19.88 [05:45:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.16, 22.62, 22.14 [05:45:27] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.20, 17.30, 18.61 [05:46:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 13.65, 19.16, 20.25 [05:46:32] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 12.09, 18.15, 19.95 [05:46:49] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.44, 22.96, 23.72 [05:49:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.40, 17.37, 20.06 [05:50:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.70, 22.30, 23.16 [05:54:07] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [05:54:14] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [05:55:10] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [05:55:12] bad phorge >:( [05:56:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.27, 21.60, 22.89 [06:01:07] BlankEclair: why don't we auto restart php [06:01:15] i believe we do, right? [06:01:32] BlankEclair: not on phorge [06:01:36] Puppet will do it [06:01:36] oh :/ [06:01:41] But why not systemd [06:01:49] It'd then be able to self recover quicker [06:02:02] i'm already trying to read dynamicpagelist3's code :( [06:04:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.72, 16.94, 20.07 [06:08:07] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [06:08:14] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.044 second response time [06:09:10] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19644 bytes in 0.065 second response time [06:18:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 20.75, 19.12 [06:20:08] This is the worst extension I maintain lol [06:20:17] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 19.36, 15.12 [06:20:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 17.73, 13.96 [06:20:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.31, 20.15, 15.08 [06:20:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.35, 20.41, 19.16 [06:21:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.34, 21.82, 17.12 [06:21:10] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.43, 18.86, 14.36 [06:21:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.63, 18.31, 13.38 [06:21:41] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.66, 20.43, 15.33 [06:22:16] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.35, 20.47, 16.12 [06:22:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.67, 18.42, 14.74 [06:22:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.21, 19.75, 15.64 [06:23:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.02, 21.24, 17.50 [06:23:09] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.23, 17.31, 14.38 [06:23:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.46, 16.47, 13.33 [06:23:40] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.50, 18.19, 15.14 [06:24:15] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 11.53, 17.12, 15.42 [06:24:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.12, 24.48, 21.10 [06:25:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.48, 18.68, 17.02 [06:26:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.72, 22.83, 20.93 [06:28:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.09, 20.14, 20.18 [06:38:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.48, 22.14, 20.48 [06:40:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.76, 22.14, 20.76 [06:42:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.54, 24.76, 21.89 [06:43:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.46, 22.16, 18.06 [06:43:05] [02dns] 07RhinosF1 opened pull request 03#544: Update 9b9t.wiki - 13https://github.com/miraheze/dns/pull/544 [06:43:49] [02dns] 07Universal-Omega closed pull request 03#544: Update 9b9t.wiki - 13https://github.com/miraheze/dns/pull/544 [06:43:50] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b0a975c6927f...640f2127eeb4 [06:43:53] [02dns] 07RhinosF1 03640f212 - Update 9b9t.wiki (#544) [06:46:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.76, 18.75, 15.85 [06:48:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.41, 19.51, 16.52 [06:55:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.86, 21.65, 21.30 [06:55:06] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/640f2127eeb4...cac0e12f2de3 [06:55:09] [02dns] 07Universal-Omega 03cac0e12 - Fix [06:57:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.99, 19.10, 20.39 [07:01:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.54, 21.48, 20.92 [07:09:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.54, 21.39, 21.77 [07:11:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 23.47, 22.45 [07:13:52] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 20.60, 17.64 [07:14:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 20.83, 17.87 [07:16:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.33, 23.11, 18.91 [07:16:27] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 21.42, 17.95 [07:16:42] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.23, 21.93, 18.21 [07:18:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.77, 23.02, 19.61 [07:20:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 16.67, 23.02, 20.09 [07:20:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.86, 24.82, 20.70 [07:20:41] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.68, 22.70, 19.49 [07:22:25] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 14.11, 21.03, 19.41 [07:22:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 15.40, 21.45, 19.98 [07:22:40] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.15, 19.61, 18.79 [07:23:48] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.01, 20.23, 19.90 [07:24:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.75, 19.22, 19.24 [07:24:25] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.26, 18.22, 18.57 [07:24:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 12.64, 18.42, 19.05 [07:25:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.53, 21.75, 23.20 [07:27:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.60, 22.80, 23.40 [07:28:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:29:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.68, 21.07, 22.72 [07:35:42] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 21.34, 19.54 [07:35:59] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.68, 18.51, 15.40 [07:36:20] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 19.97, 17.91 [07:36:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.49, 21.06, 19.15 [07:36:33] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 20.86, 18.25 [07:37:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.51, 23.39, 22.84 [07:39:11] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.46, 22.00, 19.72 [07:40:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.18, 22.66, 20.28 [07:40:31] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.83, 21.49, 19.09 [07:40:39] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [07:41:05] [02dns] 07RhinosF1 opened pull request 03#545: Update 9b9t.wiki - 13https://github.com/miraheze/dns/pull/545 [07:41:06] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.05, 23.23, 20.45 [07:41:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.25, 23.19, 21.08 [07:41:53] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 23.12, 18.43 [07:42:07] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw181 [07:42:37] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [07:43:51] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.88, 22.43, 18.77 [07:44:29] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.02, 24.27, 20.75 [07:44:34] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [07:46:25] [02dns] 07RhinosF1 synchronize pull request 03#545: Update 9b9t.wiki - 13https://github.com/miraheze/dns/pull/545 [07:47:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 24.33, 22.27 [07:48:07] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [07:48:16] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.90, 22.19, 20.90 [07:48:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.11, 22.24, 20.74 [07:48:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 23.13, 21.40 [07:48:39] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [07:48:46] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.89, 23.78, 21.96 [07:49:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.41, 21.90, 21.62 [07:49:46] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.98, 19.64, 18.82 [07:50:15] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.48, 19.78, 20.18 [07:50:27] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 22.89, 21.15 [07:50:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.83, 22.08, 21.24 [07:51:33] [02dns] 07RhinosF1 closed pull request 03#545: Update 9b9t.wiki - 13https://github.com/miraheze/dns/pull/545 [07:51:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 22.83, 21.97 [07:52:26] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.35, 22.92, 21.39 [07:54:13] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.14, 20.48, 20.47 [07:55:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.54, 22.30, 22.05 [07:56:12] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 18.99, 19.51, 20.08 [07:58:18] it's more important the number of jobs is stable or going down btw [07:58:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.19, 18.94, 20.22 [07:58:34] having a high number of jobs isn't a bad thing if they are all recent [07:59:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.71, 22.05, 21.82 [08:02:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 27.46, 22.69, 21.63 [08:02:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.87, 22.06, 21.20 [08:02:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 22.75, 21.71 [08:03:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.31, 23.25, 22.45 [08:04:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.57, 22.02, 21.49 [08:04:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.57, 21.65, 21.40 [08:08:06] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.87, 21.19, 20.56 [08:08:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 22.42, 21.75 [08:08:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.52, 23.72, 22.20 [08:12:05] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 23.82, 21.71 [08:12:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 23.05, 21.49 [08:14:04] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.14, 23.81, 22.00 [08:14:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.85, 22.77, 21.58 [08:14:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.29, 23.52, 22.71 [08:16:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.91, 23.05, 22.77 [08:17:16] <.labster> Translation is the worst extension you don't maintain. [08:23:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:24:01] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.73, 18.14, 20.20 [08:25:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 10.05, 15.63, 19.31 [08:26:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.28, 15.97, 19.60 [08:26:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.68, 18.14, 20.04 [08:26:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.08, 16.64, 19.78 [08:31:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.15, 19.46, 23.45 [08:32:28] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:47:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.22, 18.62, 20.40 [08:47:06] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [08:51:16] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 41% [08:53:14] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 26% [09:02:32] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:38:47] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is WARNING: WARNING - NGINX Error Rate is 42% [09:39:16] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 21.74, 19.11 [09:39:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.07, 19.00, 13.97 [09:39:34] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.61, 19.34, 15.47 [09:39:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 21.00, 16.62 [09:40:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.81, 19.32, 15.43 [09:40:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.97, 19.89, 16.61 [09:40:40] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [09:42:47] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 38% [09:43:08] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.51, 23.43, 20.44 [09:43:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.23, 20.84, 15.91 [09:43:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.95, 22.76, 18.35 [09:43:43] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:43:44] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw162 [09:44:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.20, 24.17, 18.70 [09:44:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 22.44, 17.60 [09:44:39] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [09:45:04] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.07, 24.12, 21.04 [09:45:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.76, 23.49, 17.48 [09:45:32] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 22.78, 18.21 [09:46:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.58, 22.37, 18.15 [09:46:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.22, 23.88, 19.44 [09:47:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.58, 23.52, 21.21 [09:47:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.82, 22.27, 17.82 [09:48:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.94, 23.61, 19.89 [09:49:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.48, 24.95, 22.02 [09:49:30] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.51, 22.93, 19.29 [09:49:36] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [09:50:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.80, 23.56, 20.30 [09:50:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 24.51, 20.70 [09:51:11] Yes this is what I meant. [09:51:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 13.35, 18.71, 17.53 [09:52:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.34, 23.23, 20.69 [09:55:29] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.14, 20.38, 19.25 [09:56:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.09, 19.34, 19.67 [09:57:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.45, 20.34, 20.00 [09:59:30] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.48, 21.93, 20.13 [10:00:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 24.31, 21.97 [10:00:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 21.70, 19.98 [10:00:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 22.25, 20.76 [10:01:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:01:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.70, 20.98, 20.42 [10:02:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.11, 22.96, 21.75 [10:02:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.68, 21.46, 20.15 [10:02:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.70, 21.43, 20.68 [10:03:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 19.76, 20.40, 20.26 [10:04:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 22.92, 20.82 [10:06:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.10, 22.51, 20.90 [10:06:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 22.30, 21.17 [10:07:29] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.50, 24.33, 21.87 [10:07:41] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.34, 23.73, 21.60 [10:08:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.66, 24.95, 22.81 [10:08:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 24.77, 21.92 [10:09:22] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.58, 21.26, 18.38 [10:10:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.85, 23.14, 21.71 [10:10:37] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 43% [10:12:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.21, 23.84, 23.09 [10:12:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 25.21, 22.66 [10:12:35] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 34% [10:13:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.25, 22.67, 19.75 [10:15:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.88, 19.94, 19.09 [10:16:09] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:16:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 16.96, 22.90, 22.50 [10:17:29] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 14.52, 20.89, 22.17 [10:17:41] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 15.04, 21.42, 22.37 [10:18:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 16.61, 21.11, 22.31 [10:20:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 11.11, 15.93, 19.88 [10:20:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.48, 17.63, 20.40 [10:23:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.72, 21.07, 23.64 [10:23:29] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 15.48, 17.53, 20.24 [10:23:41] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 12.12, 15.92, 19.66 [10:24:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.21, 16.48, 19.84 [10:26:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:37:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.00, 16.53, 19.72 [10:44:19] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [11:09:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.46, 21.64, 18.84 [11:15:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 23.90, 20.65 [11:15:10] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:25:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.00, 22.46, 22.13 [11:31:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.24, 17.13, 19.88 [11:43:34] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [11:53:11] [02dns] 07RhinosF1 opened pull request 03#546: Delete zones/petrawiki.org - 13https://github.com/miraheze/dns/pull/546 [11:53:48] [02dns] 07RhinosF1 opened pull request 03#547: Delete zones/9b9t.wiki - 13https://github.com/miraheze/dns/pull/547 [11:54:13] [02dns] 07RhinosF1 opened pull request 03#548: Delete zones/adadevelopersacademy.wiki - 13https://github.com/miraheze/dns/pull/548 [11:54:22] aw, I can't action those PRs? What stuff do I have access to? [11:55:00] dns is restricted to only Infrastructure Specialists [11:55:06] [02dns] 07Universal-Omega closed pull request 03#546: Delete zones/petrawiki.org - 13https://github.com/miraheze/dns/pull/546 [11:55:07] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/cac0e12f2de3...8f9167269334 [11:55:08] [02dns] 07RhinosF1 038f91672 - Delete zones/petrawiki.org (#546) [11:55:16] [02dns] 07Universal-Omega closed pull request 03#548: Delete zones/adadevelopersacademy.wiki - 13https://github.com/miraheze/dns/pull/548 [11:55:17] you have access to mediawiki-repos, mw-config and all of our mediawiki extensions [11:55:17] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/8f9167269334...f2e23318a149 [11:55:19] [02dns] 07RhinosF1 03f2e2331 - Delete zones/adadevelopersacademy.wiki (#548) [11:55:32] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/f2e23318a149...2fb152da5d75 [11:55:35] [02dns] 07RhinosF1 032fb152d - Delete zones/9b9t.wiki (#547) [11:55:36] [02dns] 07Universal-Omega closed pull request 03#547: Delete zones/9b9t.wiki - 13https://github.com/miraheze/dns/pull/547 [11:55:41] Ah I see! [11:56:16] [02dns] 07RhinosF1 opened pull request 03#549: Delete zones/miraheze.org - 13https://github.com/miraheze/dns/pull/549 [11:57:30] you may ask for extended access tho [11:59:20] [02dns] 07Universal-Omega closed pull request 03#549: Delete zones/miraheze.org - 13https://github.com/miraheze/dns/pull/549 [11:59:23] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/2fb152da5d75...e3c5fd0c0580 [11:59:26] [02dns] 07RhinosF1 03e3c5fd0 - Delete zones/miraheze.org (#549) [11:59:39] [02dns] 07redbluegreenhat commented on pull request 03#549: Delete zones/miraheze.org - 13https://github.com/miraheze/dns/pull/549#issuecomment-2321005410 [11:59:58] [02dns] 07redbluegreenhat edited a comment on pull request 03#549: Delete zones/miraheze.org - 13https://github.com/miraheze/dns/pull/549#issuecomment-2321005410 [12:05:53] TheresNoTime: you got shell? [12:06:08] to run a quick sql for me [12:06:47] get all wiki_url from cw_wikis where it isnt like %.miraheze.org [12:06:53] PROBLEM - cg.songngu.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cg.songngu.xyz could not be found [12:07:01] RhinosF1: one moment.. [12:07:27] icinga-miraheze: expected? [12:07:32] icinga-miraheze: dumbo [12:08:10] RhinosF1: what did it say? [12:08:42] CosmicAlpha: it's the rDNS check for a domain I just moved to CF [12:08:49] ah [12:08:54] that check can be deleted now [12:09:27] FYI https://issue-tracker.miraheze.org/T12525 [12:09:36] Don't worry about failing checks. I'd like to just deal with them all at once rather than hundreds of individual PRs. We can remove all the remaining at once. [12:10:11] bluemoon0332: we'll just delete most of it eventually [12:10:49] What would we even monitor for CF I guess replace with some check to see if cname is pointed so we know if we can remove from CF? [12:11:17] we can do a new is pointed check i guess [12:11:27] to see if it cnames to us [12:11:53] RhinosF1: I have that data you wanted [12:12:03] I'll post to phorge, one sec [12:12:06] RhinosF1: Yeah I'd like that just so we don't have unneeded domains in CF we could even hook it into a CF API script to auto remove if it is not maybe [12:12:06] ok [12:12:14] CosmicAlpha: sure [12:12:59] Anyway I am gonna call it a night for the evening I think. I may be back on a bit later we will see. [12:13:10] see ya [12:15:56] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.88, 22.97, 19.68 [12:16:06] RhinosF1: Here you go: https://issue-tracker.miraheze.org/P509 [12:17:19] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lophocmatngu.wiki could not be found [12:18:14] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.31, 21.18, 16.14 [12:18:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.10, 22.06, 17.00 [12:20:13] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.97, 21.27, 16.82 [12:20:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.82, 19.71, 16.13 [12:21:03] 516!? That's a lot lol [12:21:31] Y’all having fun [12:21:41] i'm here like :3< [12:21:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.88, 22.54, 20.92 [12:22:12] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 12.12, 17.99, 16.17 [12:22:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.90, 20.33, 17.65 [12:22:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.34, 17.25, 15.67 [12:22:27] Just trying to get custom domains settled on Cloudflare lol [12:22:40] I just woke up lol [12:22:46] Awesome! [12:23:05] Some of them are already working from what rhinos said? [12:23:09] PROBLEM - test2.specialchat.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'test2.specialchat.org' expires in 15 day(s) (Sun 15 Sep 2024 12:08:14 PM GMT +0000). [12:23:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/087b0b1b2e76...be5fb8d0afe5 [12:23:24] [02ssl] 07WikiTideSSLBot 03be5fb8d - Bot: Update SSL cert for test2.specialchat.org [12:24:06] A few but over 500 left [12:24:31] The ones that use subdomains? [12:24:42] yep [12:24:53] Yeah [12:25:21] Gotcha [12:25:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.04, 19.37, 20.01 [12:25:41] does this mean that varnish will no longer be used after the custom domain migration? [12:25:49] Hopefully [12:25:56] gotta update my docs [12:25:58] It is one less thing to maintain [12:26:13] also, how long will mw-lb.miraheze.org be able to be hit? [12:26:15] Considering last I saw it was what 6 times slowed [12:26:27] wdym? [12:26:55] like, how long can we touch miraheze directly after the migration? [12:27:19] mw-lb will never go away, just it itself will just be aliased to cf-lb so everything will be automatic, new ones should use cf-lb directly though. [12:27:48] There may be a minor bit of downtime for custom domains when that is switched over [12:28:40] oh okay [12:28:53] will we be able to touch mh directly after this or not tho [12:29:25] @rhinosf1 may be able to answer. I'm not fully understanding your question lol [12:30:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:30:17] will there be a way to bypass cloudflare and connect to miraheze servers directly? [12:31:45] Yes, with an access key to avoid direct DDOSing [12:32:02] Unless there is another way around it [12:32:56] as... as in like https [12:33:51] idk lol I'm just confused. Probably cofusing myself multitasking here and IRL [12:35:25] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:36:15] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.20, 20.81, 20.10 [12:36:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.11, 18.47, 16.27 [12:36:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 19.62, 16.86 [12:36:41] PROBLEM - petrawiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:36:52] PROBLEM - petrawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - petrawiki.org All nameservers failed to answer the query. [12:37:00] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.31, 18.93, 15.97 [12:37:18] PROBLEM - hr.petrawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - hr.petrawiki.org All nameservers failed to answer the query. [12:37:53] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:38:05] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 21.32, 17.73 [12:38:11] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 22.91, 20.94 [12:38:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182 [12:39:38] PROBLEM - adadevelopersacademy.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - adadevelopersacademy.wiki All nameservers failed to answer the query. [12:40:17] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.68, 21.90, 16.05 [12:40:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.74, 23.01, 18.62 [12:40:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [12:40:59] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.96, 21.30, 17.77 [12:41:42] PROBLEM - en.petrawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.petrawiki.org All nameservers failed to answer the query. [12:42:15] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.18, 20.78, 16.43 [12:42:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.72, 22.37, 18.92 [12:42:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.08, 22.48, 19.17 [12:44:13] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 23.30, 17.86 [12:44:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.85, 22.77, 19.75 [12:44:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.10, 24.86, 20.44 [12:44:57] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 23.44, 19.48 [12:46:00] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:46:04] PROBLEM - files.petrawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - files.petrawiki.org All nameservers failed to answer the query. [12:46:11] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.75, 22.75, 18.31 [12:46:38] PROBLEM - en.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:46:57] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.83, 23.69, 20.08 [12:48:09] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.24, 20.01, 17.84 [12:48:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.12, 22.74, 20.72 [12:48:45] PROBLEM - translate.petrawiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:49:26] PROBLEM - translate.petrawiki.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - translate.petrawiki.org All nameservers failed to answer the query. [12:50:00] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.39, 23.30, 21.13 [12:50:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.39, 20.22, 19.38 [12:50:34] TheresNoTime: I can't get the CF API to work [12:51:00] https://www.irccloud.com/pastebin/T9UGv1jr/ [12:51:14] TheresNoTime: what is wrong with that? [12:51:47] RECOVERY - test2.specialchat.org - LetsEncrypt on sslhost is OK: OK - Certificate 'test2.specialchat.org' will expire on Thu 28 Nov 2024 11:24:45 AM GMT +0000. [12:51:59] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.90, 23.16, 21.30 [12:52:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.24, 23.83, 21.82 [12:52:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.04, 20.40, 20.18 [12:52:55] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.63, 20.23, 19.58 [12:53:28] Can't get the CF API to work [12:53:36] If it would, I'd move all mw-lb [12:53:47] RhinosF1: why's it doing GET requests? [12:53:48] Did you drink enough coffee [12:53:58] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.05, 21.85, 21.07 [12:54:15] also, jeez, the code is weird [12:54:20] BlankEclair: token/verify is a GET [12:54:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 22.39, 20.46 [12:54:24] borderline looks like it was ai-generated [12:54:44] I changed it when I went to just the test endpoint [12:54:59] BlankEclair: it's from check rDNS [12:55:12] what error are you getting? [12:56:07] {"success":false,"errors":[{"code":6003,"message":"Invalid request headers","error_chain":[{"code":6111,"message":"Invalid format for Authorization header"}]}],"messages":[],"result":null} [12:56:11] BlankEclair: [12:56:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.20, 23.75, 22.12 [12:56:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 22.44, 20.98 [12:56:52] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.78, 22.10, 20.43 [12:56:55] just doing a sanity check: i assume that you've redacted the token in the headers, and you haven't actually just sent "Bearer TOKEN" to the server, right? [12:57:11] BlankEclair: yes [12:57:56] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 24.67, 22.39 [12:58:08] Try rewriting all of it and see what happens [12:58:30] that... could work i guess [12:58:37] i love mitm proxies though [12:59:12] Or I look through the code I don’t understand and you explain it to be until something clicks? [12:59:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.99, 20.71, 18.75 [13:00:51] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.84, 23.62, 21.49 [13:01:03] welp no idea then [13:01:54] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.44, 19.46, 18.49 [13:02:50] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.05, 23.98, 21.88 [13:03:54] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.24, 23.93, 23.14 [13:04:50] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.52, 23.69, 22.05 [13:05:53] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 24.69, 23.48 [13:06:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.30, 23.01, 22.36 [13:08:23] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.76, 24.55, 22.98 [13:08:48] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 28.01, 24.76, 22.73 [13:11:42] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 21.49, 19.52 [13:13:34] So this file is all that stand between us custom domains and cloud? [13:13:40] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.25, 19.38, 18.99 [13:13:49] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 17.46, 22.87, 23.58 [13:14:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.76, 22.63, 23.59 [13:14:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.64, 23.92, 23.45 [13:14:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.95, 23.75, 23.66 [13:14:46] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 17.30, 23.61, 23.26 [13:15:18] yes [13:15:29] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:02] So you wanna try rewriting or explaining to a child to see if something clicks:ThinkerMH: [13:20:03] mre: curl -v https://api.cloudflare.com/client/v4/user/tokens/verify -H 'Authorization: Bearer TOKEN' [13:20:19] Its python thats whats wrong [13:20:44] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 11.78, 16.54, 20.31 [13:20:53] Alright then go rewrite this in PHP [13:20:54] Use nodejs its a breeze working with apis [13:21:08] Eazy guzzle is your best friend [13:21:51] But the call of duty beta starts tonight so no php for me this weekend 🌞🌞🌞 [13:22:03] oa i literally posted an example curl command that reproduces the error [13:22:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.28, 16.58, 20.20 [13:22:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.76, 16.67, 20.25 [13:22:29] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 13.39, 16.84, 20.37 [13:23:45] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.68, 16.84, 19.98 [13:24:16] It works [13:24:25] Oh? [13:24:40] most shit api ever [13:24:56] Life [13:24:56] how?? [13:25:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:25:13] was it vague because the token was invalid or some shit [13:25:30] BlankEclair: it's just super unclear docs [13:26:28] Welp good news is good news [13:27:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.63, 20.26, 23.69 [13:29:17] it keeps crashing [13:29:22] but we're slowly doing it [13:30:05] Progress [13:31:02] fixed my bug i think [13:32:04] we're at 85 uploaded [13:34:21] Wooo [13:37:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.18, 16.22, 19.81 [13:37:14] Do all wikis moving go down for a bit or could it just fail [13:38:13] it's doing w now [13:38:16] Also how do we tell if a wiki is moved [13:38:20] for a few minutes they'll go down [13:38:24] all will move at once [13:39:11] So it’s uploading then Flick the switch [13:39:26] How fast is it moving [13:39:43] nearly done [13:40:12] Fire in the hole [13:41:30] I’m imagining you as a wizard in a cloak in front of some flashy core casting some big spells to move the floating islands of custom domains wikis behind the shield of firely clouds [13:43:48] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:43:59] !log move mw-lb to behind cloudflare [13:44:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:44:08] @pixldev all done [13:44:18] Ready to flick the switch? [13:45:01] @pixldev flicked [13:45:15] Well then [13:45:21] https://tenor.com/view/cloud-cloudflare-cloudfare-michelle-zatlyn-gif-26876259 [13:45:53] Huh [13:45:57] I can access phighting wiki [13:46:03] PROBLEM - wiki.sadboyzpod.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sadboyzpod.com could not be found [13:46:06] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [13:46:13] What’s a good way to tell if it’s behind CR [13:46:14] CF [13:46:16] PROBLEM - tech.exputra.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tech.exputra.com could not be found [13:46:18] PROBLEM - wiki.b3313.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.b3313.org could not be found [13:46:28] PROBLEM - kb.nena.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kb.nena.org could not be found [13:46:29] PROBLEM - wiki.starshipsailing.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starshipsailing.net could not be found [13:46:31] PROBLEM - wiki.simplifiedcoding.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.simplifiedcoding.org could not be found [13:46:32] PROBLEM - grk.repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806BDE9769140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:46:33] PROBLEM - wiki.aoemods.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aoemods.com could not be found [13:46:34] PROBLEM - lab.wikimimo.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lab.wikimimo.com could not be found [13:46:39] PROBLEM - wiki.knowledgerevolution.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.knowledgerevolution.eu could not be found [13:46:47] @pixldev domain? [13:46:49] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805BAFFC2A150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:46:56] https://phighting.wiki [13:47:00] PROBLEM - wiki.ivy.cm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ivy.cm could not be found [13:47:04] PROBLEM - wiki.iurevar.world - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.iurevar.world could not be found [13:47:08] PROBLEM - wiki.zergmafia.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zergmafia.top could not be found [13:47:10] PROBLEM - wiki.ricochetuniverse.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ricochetuniverse.com could not be found [13:47:10] PROBLEM - en.omniversalis.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.omniversalis.org could not be found [13:47:12] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80CB09659B140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:47:13] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for data.nonbinary.wiki could not be found [13:47:14] PROBLEM - wiki.vinesh.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vinesh.eu.org could not be found [13:47:19] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mobilityengineer.com could not be found [13:47:19] PROBLEM - wikislamica.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikislamica.org could not be found [13:47:19] PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.beergeeks.co.il could not be found [13:47:25] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.artmechanicum.com could not be found [13:47:25] PROBLEM - www.kinitopedia.lol - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.kinitopedia.lol could not be found [13:47:30] works for me [13:47:31] PROBLEM - grk.repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grk.repository.archiopedia.org could not be found [13:47:34] what do you get? [13:47:36] PROBLEM - wiki.seamly.io - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.io could not be found [13:47:38] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FB02A0D7140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:47:39] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineland.eu could not be found [13:47:45] PROBLEM - wiki.mineplay.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mineplay.nl could not be found [13:47:55] Same [13:47:58] PROBLEM - o.kaoyu666.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for o.kaoyu666.cn could not be found [13:48:01] PROBLEM - wiki.rotary9220.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rotary9220.net could not be found [13:48:03] PROBLEM - minescape.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for minescape.wiki could not be found [13:48:04] PROBLEM - ru-teirailway.f5.si - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ru-teirailway.f5.si could not be found [13:48:05] I can visit if fine [13:48:08] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for with.cpt-ra.bid could not be found [13:48:14] So can discord [13:48:17] phighting is legacy still [13:48:17] PROBLEM - wiki.susqu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.susqu.org could not be found [13:48:20] Ah [13:48:23] PROBLEM - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rosestulipsandliberty.com could not be found [13:48:31] PROBLEM - gufengcheng.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80CB50279C140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:48:36] Thought they would all go down and up [13:48:43] PROBLEM - internalwiki.coolstation.space - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807BF183D7140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:48:43] PROBLEM - crustypedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.801B324C7F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:48:45] PROBLEM - wiki.elysiamc.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.elysiamc.xyz could not be found [13:48:46] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [13:48:46] PROBLEM - wiki.potatotransit.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.potatotransit.xyz could not be found [13:48:47] PROBLEM - wiki.leapit.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.leapit.no could not be found [13:48:47] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pubwiki.lab.co.pl could not be found [13:48:49] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [13:48:49] PROBLEM - wiki.betaonline.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.betaonline.org could not be found [13:48:51] 100 custom domains are down at the moment [13:48:57] PROBLEM - www.sekaipedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sekaipedia.org could not be found [13:48:58] PROBLEM - largedu.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for largedu.eu.org could not be found [13:48:58] out of the 300 moved [13:49:02] PROBLEM - equestripedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for equestripedia.org could not be found [13:49:06] PROBLEM - wiki.3point0.science - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.3point0.science could not be found [13:49:07] PROBLEM - wiki.lesunk.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lesunk.com could not be found [13:49:11] PROBLEM - repository.archiopedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805BF44D3D150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:49:17] PROBLEM - wiki.maribelhearn.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.maribelhearn.com could not be found [13:49:20] 300 moved? [13:49:22] PROBLEM - wiki.dragonsmp.pro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dragonsmp.pro could not be found [13:49:25] Thought it was 500 [13:49:26] PROBLEM - www.ngnl.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.ngnl.wiki could not be found [13:49:26] Oh [13:49:29] PROBLEM - wiki.ate42.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ate42.ru could not be found [13:49:30] 299 subdomains? [13:49:31] PROBLEM - unhingedwiki.paralleldv.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for unhingedwiki.paralleldv.org could not be found [13:49:35] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gab.pt.eu.org could not be found [13:49:39] PROBLEM - wiki.gtnhdemocracy.site - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gtnhdemocracy.site could not be found [13:49:44] PROBLEM - wiki.10key.plus - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.10key.plus could not be found [13:49:50] 340 domains are in cloudflare [13:49:58] so the other 160 must be using our dns [13:50:12] Huh [13:50:13] PROBLEM - wiki.irisstation.lol - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.irisstation.lol could not be found [13:50:13] What [13:50:23] PROBLEM - wiki.secondrenaissance.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.secondrenaissance.net could not be found [13:50:23] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [13:50:23] PROBLEM - wiki.pixandbit.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pixandbit.gay could not be found [13:50:23] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80BBAB940E150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:50:36] > [30/08/2024 23:46] What’s a good way to tell if it’s behind CR [13:50:38] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80AB794265140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:50:40] PROBLEM - wiki.showtapeselection.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.showtapeselection.com could not be found [13:50:40] PROBLEM - wiki.msnld.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.msnld.com could not be found [13:50:40] PROBLEM - elsterclopedie.marithaime.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for elsterclopedie.marithaime.nl could not be found [13:50:41] PROBLEM - sona.pona.la - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sona.pona.la could not be found [13:50:41] PROBLEM - wiki.hitscan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hitscan.org could not be found [13:50:42] server or cf-ray http response header [13:50:45] only CNAME'd domains have moved @pixldev [13:50:46] PROBLEM - cities.simulz.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800BF8DD99140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:50:48] PROBLEM - correypedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for correypedia.org could not be found [13:50:48] PROBLEM - wiki.yonicstudios.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yonicstudios.com could not be found [13:50:50] 86 are still moving [13:50:52] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [13:50:53] did i... not paste? [13:50:57] > [30/08/2024 23:46] What’s a good way to tell if it’s behind CR [13:51:07] oh nvm i did paste, i just didn't see it [13:51:13] PROBLEM - publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for publictestwiki.com could not be found [13:51:18] PROBLEM - meijinversus.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meijinversus.com could not be found [13:51:26] PROBLEM - dutki.kro.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DB333AB3140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:51:27] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80ABAF374A150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:51:33] PROBLEM - wiki.aoemods.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FBA2D866140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:51:34] PROBLEM - alternatewiki.tombricks.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for alternatewiki.tombricks.com could not be found [13:51:34] PROBLEM - ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ethene.wiki could not be found [13:51:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for uk.religiononfire.mar.in.ua could not be found [13:51:38] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.andreijiroh.uk.eu.org could not be found [13:51:40] PROBLEM - wiki.consid.vn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.consid.vn could not be found [13:51:40] PROBLEM - evil.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for evil.wiki could not be found [13:51:41] PROBLEM - wiki.cdntennis.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cdntennis.ca could not be found [13:51:44] PROBLEM - wiki.funkey-project.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.funkey-project.com could not be found [13:51:44] PROBLEM - wiki.scvo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.scvo.org could not be found [13:51:58] Amend- on mobile [13:51:58] PROBLEM - wiki.strangereons.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.strangereons.com could not be found [13:52:13] PROBLEM - wiki.eversd.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.eversd.com could not be found [13:52:21] get a terminal emulator /hj [13:52:23] PROBLEM - wiki.cor.fyi - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cor.fyi could not be found [13:52:29] PROBLEM - threedomwiki.pcast.site - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for threedomwiki.pcast.site could not be found [13:52:30] PROBLEM - wiki.sequoiafabrica.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sequoiafabrica.org could not be found [13:52:31] PROBLEM - talumn.neptune.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for talumn.neptune.wiki could not be found [13:52:33] PROBLEM - airlineinsider.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for airlineinsider.org could not be found [13:52:48] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [13:52:55] PROBLEM - de.berlinwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.berlinwiki.org could not be found [13:53:09] PROBLEM - archives.nsgov.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archives.nsgov.org could not be found [13:53:10] PROBLEM - moddedminecraft.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for moddedminecraft.wiki could not be found [13:53:14] PROBLEM - wiki.k-language.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.k-language.dev could not be found [13:53:14] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [13:53:21] PROBLEM - wiki.globasa.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.globasa.net could not be found [13:53:22] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DBB090DA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:53:22] PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minigamemadness.net could not be found [13:53:22] PROBLEM - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.drydraytonvillagehall.org.uk could not be found [13:53:23] I do [13:53:30] I forgot lol [13:53:33] PROBLEM - wiki.limaru.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.limaru.net could not be found [13:53:33] PROBLEM - wiki.climatechange.ai - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.climatechange.ai could not be found [13:53:35] PROBLEM - wiki.hwfib.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.hwfib.com could not be found [13:53:36] PROBLEM - www.bluepageswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluepageswiki.org could not be found [13:53:37] PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.809B68CE04150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:53:38] PROBLEM - wiki.col6.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.col6.de could not be found [13:53:40] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [13:53:42] BlankEclair: you can see the progress by the number of icinga errors [13:53:42] PROBLEM - wiki.enkro.co - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.enkro.co could not be found [13:53:46] PROBLEM - wiki.chicory.pizza - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.chicory.pizza could not be found [13:53:48] LMAO [13:53:52] ah yes [13:54:08] https://fox.nexus/@quips/113047877935968427 [13:54:13] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.thesimswiki.com could not be found [13:54:14] PROBLEM - wiki.ouro.one - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ouro.one could not be found [13:54:17] PROBLEM - wiki.worlduniversityandschool.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worlduniversityandschool.org could not be found [13:54:19] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lewisgill.com could not be found [13:54:47] PROBLEM - wiki.mc.marochuru.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808BD56114150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:54:48] PROBLEM - hcw.tomat.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hcw.tomat.dev could not be found [13:54:49] PROBLEM - wiki.site97.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.site97.fr could not be found [13:54:55] PROBLEM - wiki.potabi.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.potabi.com could not be found [13:55:06] PROBLEM - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gesamtschule-nordkirchen.de could not be found [13:55:06] PROBLEM - wiki.kkstudio.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kkstudio.eu.org could not be found [13:55:06] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [13:55:06] PROBLEM - wiki.mc.marochuru.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mc.marochuru.net could not be found [13:55:09] PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for robloxapi.wiki could not be found [13:55:14] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [13:55:22] woah there's a roblox api wiki? [13:55:32] BlankEclair: there is all sorts [13:55:33] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [13:55:35] 11 still moving [13:55:39] > CNAME Cross-User Banned [13:55:41] okay then [13:55:43] PROBLEM - wiki.sofurry.games - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sofurry.games could not be found [13:55:49] PROBLEM - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807B57E178140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:55:50] PROBLEM - small.polandballwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for small.polandballwiki.com could not be found [13:56:10] PROBLEM - wiki.meeusen.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meeusen.net could not be found [13:56:12] PROBLEM - eng.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eng.archiopedia.org could not be found [13:56:13] PROBLEM - wiki.kachow.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80ABF687CC140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:56:25] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805B498314150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:56:30] PROBLEM - zh.tardis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zh.tardis.wiki could not be found [13:56:30] PROBLEM - atlas.starworld.zone - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for atlas.starworld.zone could not be found [13:56:31] PROBLEM - wiki.cube-conflict.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cube-conflict.com could not be found [13:56:34] PROBLEM - wiki.aridia.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aridia.space could not be found [13:56:44] PROBLEM - bluearchive.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bluearchive.wiki could not be found [13:56:44] PROBLEM - wiki.stag.lol - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.stag.lol could not be found [13:56:44] PROBLEM - wiki.veloren.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.veloren.net could not be found [13:56:49] PROBLEM - wiki.yumeka.icu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yumeka.icu could not be found [13:57:01] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [13:57:31] PROBLEM - wiki.zahira.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zahira.me could not be found [13:57:31] PROBLEM - wiki.rlab.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rlab.org.uk could not be found [13:57:31] PROBLEM - wiki.whentheycry.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.whentheycry.org could not be found [13:57:37] PROBLEM - wiki.gestaltonline.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gestaltonline.ca could not be found [13:57:41] PROBLEM - iewiki.auroville.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807B59027F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:57:42] PROBLEM - wiki.newnormal.fans - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.newnormal.fans could not be found [13:57:50] PROBLEM - wiki.mega-earth.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mega-earth.com could not be found [13:57:50] PROBLEM - wiki.solarconquest.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.solarconquest.net could not be found [13:57:51] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.809BE97A61140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:57:56] PROBLEM - wiki.esnmilanostatale.it - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.esnmilanostatale.it could not be found [13:57:57] PROBLEM - wiki.drycleaninglobby.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.drycleaninglobby.net could not be found [13:58:00] PROBLEM - wiki.meregos.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.meregos.com could not be found [13:58:00] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mason.sagan4.org could not be found [13:58:05] PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mostly.vet could not be found [13:58:05] PROBLEM - wiki.geoparkcorumbatai.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.geoparkcorumbatai.com.br could not be found [13:58:06] PROBLEM - pickaxe.showierdata.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pickaxe.showierdata.xyz could not be found [13:58:06] PROBLEM - fanwiki.takagi3.cn - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fanwiki.takagi3.cn could not be found [13:58:21] PROBLEM - wiki.jill-jimmy.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jill-jimmy.com could not be found [13:58:29] PROBLEM - wiki.jagtek.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jagtek.org could not be found [13:58:31] PROBLEM - wiki.myehs.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.myehs.eu could not be found [13:58:31] PROBLEM - wiki.evscope.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.evscope.org could not be found [13:58:31] PROBLEM - wiki.lefrenchmelee.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lefrenchmelee.fr could not be found [13:58:35] PROBLEM - wiki.insideearth.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.insideearth.info could not be found [13:58:36] PROBLEM - wiki.worldbuilding.network - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldbuilding.network could not be found [13:58:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.14, 21.72, 23.73 [13:58:52] PROBLEM - crustypedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for crustypedia.org could not be found [13:58:55] Shit really! [13:58:57] ? [13:58:58] PROBLEM - alpha.sagan4.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for alpha.sagan4.org could not be found [13:59:01] PROBLEM - wiki.autocountsoft.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.autocountsoft.com could not be found [13:59:02] PROBLEM - progressbar95.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for progressbar95.wiki could not be found [13:59:07] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.801B8BEE4C150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:59:11] PROBLEM - legacygt.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.801B9C0C67140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:59:11] > [30/08/2024 23:55] PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for robloxapi.wiki could not be found [13:59:17] PROBLEM - wiki.candelabrem.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.candelabrem.com could not be found [13:59:17] PROBLEM - wiki.routedoodle.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.routedoodle.com could not be found [13:59:18] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80BBF02192140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:59:22] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for beta.sagan4.org could not be found [13:59:23] PROBLEM - wiki.elgeis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.elgeis.com could not be found [13:59:25] PROBLEM - wiki.cogsci.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cogsci.live could not be found [13:59:37] PROBLEM - ndgkb.nena.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ndgkb.nena.org could not be found [13:59:41] PROBLEM - capixabapedia.vipnunes.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for capixabapedia.vipnunes.com could not be found [13:59:42] PROBLEM - glitchcity.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808B718FCA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [13:59:46] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [13:59:50] PROBLEM - wiki.puucraft.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.puucraft.net could not be found [13:59:55] PROBLEM - evade.robtospro.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for evade.robtospro.com could not be found [13:59:59] PROBLEM - test2.specialchat.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for test2.specialchat.org could not be found [14:00:00] PROBLEM - www.istpcomputing.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.istpcomputing.com could not be found [14:00:07] How do you see headers from curl response [14:00:07] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80AB54B1FA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:00:17] PROBLEM - wiki.gestaltonline.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.803B434E11150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:00:24] PROBLEM - wiki.harlonserver.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.harlonserver.net could not be found [14:00:26] PROBLEM - wiki.chevrine.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.chevrine.com could not be found [14:00:37] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lgbtqia.wiki could not be found [14:00:37] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [14:00:39] PROBLEM - itora.bunnies1230.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for itora.bunnies1230.com could not be found [14:00:45] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DB213664140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:00:46] PROBLEM - wiki.starship.digital - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.starship.digital could not be found [14:00:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 22.98, 23.92 [14:00:54] Uh [14:00:54] PROBLEM - polcompball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for polcompball.wiki could not be found [14:00:54] PROBLEM - www.ethene.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.ethene.wiki could not be found [14:00:55] PROBLEM - abeyancewiki.bumblebee-studios.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for abeyancewiki.bumblebee-studios.se could not be found [14:00:57] PROBLEM - airlineinsider.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DB93A813150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:00:58] PROBLEM - wiki.cyberfurs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberfurs.org could not be found [14:01:03] PROBLEM - drachronicles.voxel.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for drachronicles.voxel.gay could not be found [14:01:10] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for elephants.ggreig.com could not be found [14:01:11] @rhinosf1 heeeey should robloxapi.wiki redirect to the mana world [14:01:11] PROBLEM - wiki.gamergeeked.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gamergeeked.us could not be found [14:01:18] PROBLEM - wiki.manos.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.manos.pt could not be found [14:01:23] PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found [14:01:26] PROBLEM - education.gced.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for education.gced.in could not be found [14:01:32] PROBLEM - internalwiki.coolstation.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for internalwiki.coolstation.space could not be found [14:01:36] PROBLEM - wiki.kirbygang.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kirbygang.com could not be found [14:01:36] PROBLEM - wiki.thelastdimension.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thelastdimension.net could not be found [14:01:53] PROBLEM - gogigantic.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gogigantic.wiki could not be found [14:01:55] PROBLEM - wiki.grottocenter.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grottocenter.org could not be found [14:01:56] PROBLEM - bluearchive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80AB5B0F73140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:02:04] PROBLEM - wiki.jeremyboyd.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jeremyboyd.org could not be found [14:02:05] PROBLEM - wiki.blockate.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.blockate.com could not be found [14:02:05] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [14:02:05] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mahdiruiz.line.pm could not be found [14:02:06] purged [14:02:08] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [14:02:11] PROBLEM - wiki.closai.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.809BE8ED95140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:02:12] PROBLEM - wiki.case-clicker.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.case-clicker.com could not be found [14:02:12] PROBLEM - celeste.ink - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for celeste.ink could not be found [14:02:12] PROBLEM - wiki.k2cho.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.k2cho.me could not be found [14:02:16] Yup [14:02:20] PROBLEM - en.clockup.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.clockup.wiki could not be found [14:02:21] Definitely a wizard [14:02:22] PROBLEM - aman.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for aman.awiki.org could not be found [14:02:29] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.stellurgists.wiki could not be found [14:02:34] PROBLEM - wiki.ff6worldscollide.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ff6worldscollide.com could not be found [14:02:40] PROBLEM - www.chinafake.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.chinafake.wiki could not be found [14:02:40] PROBLEM - chobots.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chobots.wiki could not be found [14:02:40] PROBLEM - www.dovearchives.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.dovearchives.wiki could not be found [14:02:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.56, 22.20, 23.56 [14:03:01] PROBLEM - wiki.wubbygame.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wubbygame.com could not be found [14:03:01] PROBLEM - wiki.kachow.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kachow.org could not be found [14:03:03] My mental picture of you purging cf stuff is literally casting a fire blast and burning a cloud away and pulling in a new one [14:03:03] PROBLEM - wiki.sletat.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sletat.tech could not be found [14:03:13] PROBLEM - guia.esporo.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.esporo.net could not be found [14:03:15] PROBLEM - wiki.widwwa.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.widwwa.co.uk could not be found [14:03:17] God damn it [14:03:19] PROBLEM - wiki.zamnhacking.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zamnhacking.net could not be found [14:03:21] PROBLEM - wiki.junkstore.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.junkstore.xyz could not be found [14:03:26] The wiki doesn’t exist anymore [14:03:32] PROBLEM - wiki.simorgh.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.simorgh.me could not be found [14:03:36] PROBLEM - cloud.neptune.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cloud.neptune.wiki could not be found [14:03:36] PROBLEM - wiki.closai.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.closai.com could not be found [14:03:39] BlankEclair https://robloxapi.wiki [14:03:42] PROBLEM - wiki.misslessfighters.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.misslessfighters.club could not be found [14:03:43] PROBLEM - wiki.orvyn.ca - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.803B75829D140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:03:44] PROBLEM - glitchcity.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for glitchcity.wiki could not be found [14:03:47] PROBLEM - wiki.18t.rip - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.18t.rip could not be found [14:03:50] PROBLEM - www.streamerlore.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.streamerlore.com could not be found [14:03:57] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found [14:04:09] PROBLEM - wiki.eggsdstudios.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.eggsdstudios.com could not be found [14:04:11] PROBLEM - www.winenjoy.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805BE4402E150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:04:12] PROBLEM - wiki.namegames.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.namegames.dev could not be found [14:04:12] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ao90.pinho.org could not be found [14:04:14] PROBLEM - arceniawiki.doughhearts.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for arceniawiki.doughhearts.com could not be found [14:04:14] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.danvs.net could not be found [14:04:37] PROBLEM - wiki.warframestat.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.warframestat.us could not be found [14:04:37] PROBLEM - wiki.gtnewhorizons.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gtnewhorizons.com could not be found [14:04:52] PROBLEM - wiki.s23.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.s23.org could not be found [14:04:54] PROBLEM - slymods.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for slymods.info could not be found [14:04:56] PROBLEM - wiki.cubestudios.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cubestudios.xyz could not be found [14:04:57] PROBLEM - wiki.apico.buzz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.apico.buzz could not be found [14:04:57] PROBLEM - wiki.omegabuild.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.omegabuild.uk could not be found [14:05:16] PROBLEM - wiki.orivium.io - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.orivium.io could not be found [14:05:16] PROBLEM - wiki.nevillepedia.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nevillepedia.eu could not be found [14:05:18] PROBLEM - wiki.kingmingshrimp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kingmingshrimp.com could not be found [14:05:22] PROBLEM - wiki.valkyrienskies.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.valkyrienskies.org could not be found [14:05:25] PROBLEM - iewiki.purnamcommunity.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iewiki.purnamcommunity.in could not be found [14:05:29] PROBLEM - www.fossilfighterswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fossilfighterswiki.com could not be found [14:05:42] PROBLEM - wiki.pulsus.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pulsus.cc could not be found [14:05:42] PROBLEM - tfuk.app - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tfuk.app could not be found [14:05:59] PROBLEM - rockrevolution.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rockrevolution.wiki could not be found [14:06:00] PROBLEM - wiki.yahia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yahia.xyz could not be found [14:06:41] PROBLEM - acgn.sfdev.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for acgn.sfdev.eu.org could not be found [14:06:41] PROBLEM - sonicpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sonicpedia.org could not be found [14:06:42] PROBLEM - www.documentopublico.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.documentopublico.com.br could not be found [14:06:44] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.803BB654D7140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:06:49] PROBLEM - fid.koymi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fid.koymi.net could not be found [14:06:52] PROBLEM - resources.africanvision.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for resources.africanvision.org could not be found [14:06:52] PROBLEM - wiki.amplicity.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.amplicity.fr could not be found [14:06:55] PROBLEM - grk.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grk.archiopedia.org could not be found [14:06:59] PROBLEM - repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for repository.archiopedia.org could not be found [14:06:59] PROBLEM - wiki.arsrobotics.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.arsrobotics.org could not be found [14:07:01] PROBLEM - wiki.coolstation.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.coolstation.space could not be found [14:07:01] PROBLEM - puritwiki.p-e.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for puritwiki.p-e.kr could not be found [14:07:07] PROBLEM - wiki.digitalcandela.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitalcandela.com could not be found [14:07:07] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [14:07:08] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [14:07:08] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thunis.eu could not be found [14:07:11] PROBLEM - chinafake.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for chinafake.wiki could not be found [14:07:11] PROBLEM - swests.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for swests.org could not be found [14:07:11] PROBLEM - es.countryhumans.polandball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for es.countryhumans.polandball.wiki could not be found [14:07:11] PROBLEM - nl.xliving.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nl.xliving.org could not be found [14:07:12] PROBLEM - wiki.giantesselsie.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.giantesselsie.net could not be found [14:07:12] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FB4B92B4140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:07:13] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.religiononfire.mar.in.ua could not be found [14:07:14] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vise.dayid.org could not be found [14:07:16] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [14:07:17] PROBLEM - wiki.ooer.ooo - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ooer.ooo could not be found [14:07:17] PROBLEM - wiki.queenscourt.games - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.queenscourt.games could not be found [14:07:17] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80CB949782140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:07:18] PROBLEM - wiki.theluxuryelevator.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.theluxuryelevator.com could not be found [14:07:22] PROBLEM - wiki.wildport.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wildport.com could not be found [14:07:29] PROBLEM - zomvilles.pixiemeta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for zomvilles.pixiemeta.com could not be found [14:07:38] PROBLEM - mhdh.pj568.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mhdh.pj568.eu.org could not be found [14:07:38] PROBLEM - wiki.bluevertigo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bluevertigo.org could not be found [14:07:39] PROBLEM - wiki.neonai.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.neonai.dev could not be found [14:08:00] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found [14:08:15] PROBLEM - wiki.blackorchestra.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.blackorchestra.net could not be found [14:08:15] PROBLEM - wiki.dwarvesrp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dwarvesrp.com could not be found [14:08:16] PROBLEM - wiki.dfonline.dev - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dfonline.dev could not be found [14:08:17] PROBLEM - wiki.corgicam.tv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.corgicam.tv could not be found [14:08:19] PROBLEM - wiki.counterculturelabs.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.counterculturelabs.org could not be found [14:08:20] PROBLEM - wiki.milkywayidle.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.milkywayidle.com could not be found [14:08:20] PROBLEM - 100acg.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 100acg.wiki could not be found [14:08:20] PROBLEM - marvelcinematicuniverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marvelcinematicuniverse.wiki could not be found [14:08:22] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.qadrishattari.xyz could not be found [14:08:31] PROBLEM - www.hebammenwiki.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hebammenwiki.de could not be found [14:08:34] PROBLEM - wiki.infomedia.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.infomedia.co.uk could not be found [14:08:35] PROBLEM - islegoblin.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for islegoblin.wiki could not be found [14:08:38] PROBLEM - revi.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for revi.wiki could not be found [14:08:38] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [14:08:41] PROBLEM - wiki.penguinmod.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.penguinmod.com could not be found [14:08:41] PROBLEM - wiki.denby.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.denby.tech could not be found [14:08:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.07, 21.69, 22.80 [14:08:50] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [14:08:52] PROBLEM - dlfm-wiki.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80CB1764FA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:09:09] PROBLEM - wiki.pokrew.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pokrew.net could not be found [14:09:11] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mxlinuxusers.de could not be found [14:09:16] PROBLEM - wiki.luemir.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.luemir.xyz could not be found [14:09:20] PROBLEM - wiki.walkscape.app - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.walkscape.app could not be found [14:09:21] PROBLEM - wiki.controlistes.fr - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DBD1236F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:09:28] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [14:09:31] PROBLEM - wiki.pixlies.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.pixlies.net could not be found [14:09:31] PROBLEM - dlfm-wiki.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dlfm-wiki.top could not be found [14:09:31] PROBLEM - reviwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviwiki.info could not be found [14:09:34] PROBLEM - wiki.make717.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.make717.org could not be found [14:09:35] PROBLEM - resources.africanvision.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804BB15AF9140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:09:35] PROBLEM - tno.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806B5594DA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:09:35] PROBLEM - wiki.esufranchise.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.esufranchise.com could not be found [14:09:35] PROBLEM - msmhacks.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for msmhacks.uk could not be found [14:09:37] PROBLEM - wiki.lifesprogress.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.lifesprogress.com could not be found [14:09:44] PROBLEM - wiki.mcjones.gay - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcjones.gay could not be found [14:09:55] PROBLEM - wiki.sheepservermc.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sheepservermc.net could not be found [14:09:57] PROBLEM - strinova.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for strinova.org could not be found [14:10:09] PROBLEM - wiki.zahira.me - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.801BA26245150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:10:11] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tep.wiki could not be found [14:10:11] PROBLEM - wiki.virtualjet.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.virtualjet.net could not be found [14:10:13] PROBLEM - wiki.cutefame.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cutefame.net could not be found [14:10:15] PROBLEM - wiki.buryland.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.buryland.net could not be found [14:10:16] PROBLEM - wiki.tgtdrblx.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tgtdrblx.com could not be found [14:10:19] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800BC91C08150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:10:22] PROBLEM - wiki.projectdiablo2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.projectdiablo2.com could not be found [14:10:26] PROBLEM - wiki.openhatch.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.openhatch.org could not be found [14:10:36] PROBLEM - wiki.greatphermesia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.greatphermesia.com could not be found [14:10:46] PROBLEM - 321nails.crpteam.club - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for 321nails.crpteam.club could not be found [14:10:50] PROBLEM - www.cgradegames.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.cgradegames.net could not be found [14:11:03] PROBLEM - legacygt.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for legacygt.wiki could not be found [14:11:04] PROBLEM - grc.repository.archiopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for grc.repository.archiopedia.org could not be found [14:11:05] PROBLEM - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DB8456AB140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:11:06] PROBLEM - tdr.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tdr.wiki could not be found [14:11:24] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tmyt105.leyhp.com could not be found [14:11:34] PROBLEM - wiki.taes.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.taes.com.br could not be found [14:11:53] PROBLEM - www.samafia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.samafia.org could not be found [14:12:06] PROBLEM - wiki.bluevertigo.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807BB4F320150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:12:10] PROBLEM - www.winenjoy.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.winenjoy.net could not be found [14:12:10] PROBLEM - tno.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tno.wiki could not be found [14:12:19] PROBLEM - wiki.fbpml.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fbpml.org could not be found [14:12:23] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [14:12:32] PROBLEM - wiki.meeusen.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80EBC076F1140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:12:33] PROBLEM - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.hoolehistoryheritagesociety.org.uk could not be found [14:12:34] PROBLEM - wiki.controlistes.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.controlistes.fr could not be found [14:12:38] PROBLEM - wiki.sukotuze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.sukotuze.com could not be found [14:12:38] PROBLEM - wiki.gowashtenaw.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.gowashtenaw.org could not be found [14:12:38] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for index.pdcommunity.ir could not be found [14:12:43] PROBLEM - reviews.thejoshmeister.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviews.thejoshmeister.com could not be found [14:12:53] PROBLEM - 100acg.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806BC8C659140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:12:54] PROBLEM - wiki.oasismp.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.oasismp.com could not be found [14:12:55] PROBLEM - wiki.aclevo.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aclevo.com could not be found [14:12:55] PROBLEM - wiki.tdrweb.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tdrweb.top could not be found [14:13:09] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.albrusmusic.ru could not be found [14:13:23] PROBLEM - wiki.kscucf.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kscucf.org could not be found [14:13:25] PROBLEM - lpfteam.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lpfteam.nl could not be found [14:13:27] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.lab612.at could not be found [14:13:30] PROBLEM - eros.locksteel.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for eros.locksteel.net could not be found [14:13:30] PROBLEM - wiki.otir.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.otir.nl could not be found [14:13:34] PROBLEM - wiki.orvyn.ca - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.orvyn.ca could not be found [14:13:35] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.joust.ro could not be found [14:13:45] PROBLEM - wiki.sukotuze.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800BFCE852150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:13:45] PROBLEM - wiki.cowgl.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cowgl.xyz could not be found [14:13:47] PROBLEM - wiki.jeremyboyd.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80EBE0FC01150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:13:51] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.zymonic.com could not be found [14:13:54] PROBLEM - wiki.thesimswiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thesimswiki.com could not be found [14:14:01] PROBLEM - thepolicyhub.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thepolicyhub.org.uk could not be found [14:14:02] PROBLEM - www.glitchcity.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.glitchcity.wiki could not be found [14:14:02] PROBLEM - steem.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for steem.wiki could not be found [14:14:02] PROBLEM - wiki.metaiitgn.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.metaiitgn.org could not be found [14:14:08] PROBLEM - wiki.fatcatfablab.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fatcatfablab.org could not be found [14:14:11] PROBLEM - mabi.myomi.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mabi.myomi.net could not be found [14:14:13] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804BD02954150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:14:13] PROBLEM - en.noblework.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.noblework.org could not be found [14:14:14] PROBLEM - meta.sagan4.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.sagan4.org could not be found [14:14:18] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.bikinibottomdiscord.org could not be found [14:14:18] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800B025F1A150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:14:18] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FB484F8F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:14:19] PROBLEM - ie.unigon.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ie.unigon.net could not be found [14:14:19] PROBLEM - wiki.ventistudio.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ventistudio.fr could not be found [14:14:19] PROBLEM - www.publictestwiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.publictestwiki.com could not be found [14:14:26] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [14:14:28] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.wikimedia.cat could not be found [14:14:37] PROBLEM - siam.info.gf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for siam.info.gf could not be found [14:14:46] PROBLEM - ru.countryhumans.polandball.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ru.countryhumans.polandball.wiki could not be found [14:14:46] PROBLEM - dutki.kro.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dutki.kro.kr could not be found [14:14:47] PROBLEM - forecasting.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for forecasting.wiki could not be found [14:14:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.53, 22.90, 23.31 [14:14:50] PROBLEM - gufengcheng.top - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gufengcheng.top could not be found [14:14:53] PROBLEM - wiki.grab-tools.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.grab-tools.live could not be found [14:14:54] PROBLEM - wmworld.sktz.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wmworld.sktz.live could not be found [14:15:01] PROBLEM - wiki.eastsidegamers.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.eastsidegamers.no could not be found [14:15:01] PROBLEM - lgbtqia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lgbtqia.wiki could not be found [14:15:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.18, 18.60, 16.45 [14:15:02] PROBLEM - wiki.seredrau.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seredrau.xyz could not be found [14:15:04] PROBLEM - history.estill.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.estill.org could not be found [14:15:18] PROBLEM - wiki.ssij.se - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssij.se could not be found [14:15:20] PROBLEM - gogigantic.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80EB8A5026150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:15:32] PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.macc.nyc could not be found [14:15:32] PROBLEM - cities.simulz.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cities.simulz.kr could not be found [14:15:54] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Thu 28 Nov 2024 01:43:26 PM GMT +0000. [14:16:01] PROBLEM - amps.wiki.gd - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for amps.wiki.gd could not be found [14:16:04] PROBLEM - valsora.tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for valsora.tep.wiki could not be found [14:16:06] PROBLEM - wiki.creaturathegame.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.creaturathegame.com could not be found [14:16:16] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Thu 28 Nov 2024 01:31:55 PM GMT +0000. [14:16:45] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.809BE1246B140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:16:46] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Thu 28 Nov 2024 01:53:07 PM GMT +0000. [14:16:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 25.15, 24.10 [14:17:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.04, 18.17, 16.55 [14:17:07] PROBLEM - www.glitchcity.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80FBEB887F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:17:16] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806BC9E8A4140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:17:26] RECOVERY - internalwiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Thu 28 Nov 2024 01:31:25 PM GMT +0000. [14:17:26] RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Thu 28 Nov 2024 01:25:18 PM GMT +0000. [14:17:48] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DBAC05DF140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:19:16] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Thu 28 Nov 2024 01:24:52 PM GMT +0000. [14:19:47] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Thu 28 Nov 2024 01:28:08 PM GMT +0000. [14:20:02] RECOVERY - cities.simulz.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Thu 28 Nov 2024 01:25:13 PM GMT +0000. [14:21:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.71, 20.76, 17.97 [14:21:21] RECOVERY - dutki.kro.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'dutki.kro.kr' will expire on Thu 28 Nov 2024 01:26:21 PM GMT +0000. [14:23:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.10, 18.66, 17.55 [14:28:44] [02puppet] 07redbluegreenhat created branch 03csr-p256 - 13https://github.com/miraheze/puppet [14:28:45] [02puppet] 07redbluegreenhat pushed 031 commit to 03csr-p256 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e3ef7735a8a0 [14:28:46] [02puppet] 07redbluegreenhat 03e3ef773 - SSL: Use the P-256 algorithm for private keys in CSR generation [14:29:04] [02puppet] 07redbluegreenhat opened pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905 [14:29:18] [02puppet] 07coderabbitai[bot] commented on pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905#issuecomment-2321442332 [14:30:05] [02puppet] 07redbluegreenhat commented on pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905#issuecomment-2321444587 [14:30:10] [02puppet] 07coderabbitai[bot] commented on pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905#issuecomment-2321444841 [14:30:11] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905#issuecomment-2321442332 [14:30:28] miraheze/puppet - redbluegreenhat the build passed. [14:30:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.13, 23.36, 23.98 [14:31:16] [02puppet] 07coderabbitai[bot] edited pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905 [14:31:38] RECOVERY - bluearchive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bluearchive.wiki' will expire on Thu 28 Nov 2024 01:59:07 PM GMT +0000. [14:31:49] [02puppet] 07coderabbitai[bot] reviewed pull request 03#3905 commit - 13https://github.com/miraheze/puppet/pull/3905#discussion_r1738794207 [14:31:50] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905#issuecomment-2321442332 [14:33:55] [02puppet] 07redbluegreenhat edited pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905 [14:35:23] [02puppet] 07redbluegreenhat closed pull request 03#3905: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3905 [14:35:25] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23f4f9a618bf...99952fc414ab [14:35:27] [02puppet] 07redbluegreenhat 0399952fc - SSL: Use the P-256 algorithm for private keys in CSR generation (#3905) [14:35:30] [02puppet] 07redbluegreenhat deleted branch 03csr-p256 - 13https://github.com/miraheze/puppet [14:35:32] [02puppet] 07redbluegreenhat deleted branch 03csr-p256 [14:36:56] miraheze/puppet - redbluegreenhat the build passed. [14:38:15] RECOVERY - dlfm-wiki.top - LetsEncrypt on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Thu 28 Nov 2024 02:15:47 PM GMT +0000. [14:38:57] RECOVERY - tno.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tno.wiki' will expire on Thu 28 Nov 2024 02:03:20 PM GMT +0000. [14:41:05] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [14:41:46] RECOVERY - 100acg.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '100acg.wiki' will expire on Thu 28 Nov 2024 02:29:50 PM GMT +0000. [14:44:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 13.45, 15.99, 19.22 [14:47:10] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:49:25] PROBLEM - housing.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808B490BDB140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [14:52:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:57:11] RECOVERY - legacygt.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Thu 28 Nov 2024 02:13:17 PM GMT +0000. [14:59:38] RECOVERY - glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'glitchcity.wiki' will expire on Thu 28 Nov 2024 02:19:32 PM GMT +0000. [15:13:34] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [15:14:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.72, 22.10, 18.10 [15:16:53] Weird [15:18:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.78, 21.28, 18.91 [15:20:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.09, 19.34, 18.48 [15:24:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.25, 23.75, 20.40 [15:26:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.27, 22.71, 20.39 [15:27:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.44, 19.49, 15.75 [15:28:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 24.48, 21.33 [15:29:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.00, 19.32, 16.12 [15:37:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 23.83, 19.34 [15:37:55] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.35, 18.40, 14.99 [15:39:55] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 14.79, 16.85, 14.83 [15:41:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.71, 23.48, 20.20 [15:49:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.88, 23.30, 21.35 [15:53:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.55, 23.33, 21.97 [16:01:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 22.34, 21.60 [16:02:15] !log changed rule 1 cache key rules to use original host [16:02:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:24] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 19.38, 17.59 [16:04:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.12, 22.06, 17.95 [16:04:33] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.12, 21.26, 17.69 [16:05:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.12, 23.88, 22.61 [16:05:22] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.24, 17.21, 13.42 [16:05:24] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 29.19, 24.20, 19.64 [16:05:45] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 21.77, 17.93 [16:06:36] @rhinosf1 what do you think CF defines as a "Resolved host" on the cache key rules? [16:07:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 24.67, 23.02 [16:07:22] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.13, 17.19, 13.87 [16:07:23] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.73, 22.44, 19.40 [16:07:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:07:44] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 21.86, 21.11, 18.12 [16:09:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.19, 23.28, 22.70 [16:09:44] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 23.21, 19.29 [16:10:05] I could find nothing on their docs regarding what that means [16:10:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.86, 23.64, 20.07 [16:10:54] I do think that may have been the cause for the bizarre HTTP requests to mw* though [16:11:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.50, 24.45, 23.20 [16:11:43] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.01, 22.97, 19.69 [16:12:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 26.37, 24.79, 20.93 [16:12:39] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [16:12:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw161 mw181 [16:13:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.04, 23.18, 22.86 [16:13:17] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.34, 23.90, 21.60 [16:13:20] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 28.45, 25.37, 21.60 [16:13:42] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.33, 24.29, 20.58 [16:14:01] PROBLEM - comprehensibleinputwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'comprehensibleinputwiki.org' expires in 15 day(s) (Sun 15 Sep 2024 03:46:04 PM GMT +0000). [16:14:13] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/be5fb8d0afe5...3c8ea1432af0 [16:14:14] [02ssl] 07WikiTideSSLBot 033c8ea14 - Bot: Update SSL cert for comprehensibleinputwiki.org [16:14:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw182 [16:14:39] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [16:14:49] Weird [16:14:51] Cool [16:15:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 23.90, 23.17 [16:15:45] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [16:16:15] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.92, 23.29, 18.53 [16:16:33] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:16:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [16:17:18] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.33, 23.57, 21.67 [16:18:27] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.68, 21.82, 20.56 [16:19:12] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 23.89, 22.25 [16:19:23] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [16:19:40] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.17, 23.69, 21.43 [16:19:45] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [16:20:11] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.11, 23.25, 19.56 [16:20:26] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.67, 22.80, 21.00 [16:21:22] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3821 bytes in 0.082 second response time [16:21:40] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.99, 25.15, 22.25 [16:22:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.58, 23.84, 22.38 [16:23:09] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.45, 23.30, 22.41 [16:24:07] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.27, 23.80, 20.73 [16:24:19] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 28.43, 25.73, 23.25 [16:25:08] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 28.02, 24.92, 23.10 [16:26:19] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 17.53, 23.08, 22.63 [16:26:23] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 18.50, 22.11, 21.28 [16:27:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.00, 23.66, 23.52 [16:27:06] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 16.22, 21.39, 22.01 [16:27:37] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 11.59, 19.98, 21.35 [16:28:03] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 9.37, 17.26, 18.88 [16:28:23] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.28, 19.20, 20.33 [16:28:43] [02dns] 07RhinosF1 opened pull request 03#550: Update johanloopmans.nl - 13https://github.com/miraheze/dns/pull/550 [16:29:37] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.56, 17.37, 20.21 [16:29:51] [02dns] 07RhinosF1 opened pull request 03#551: Update tintinpedia.fr - 13https://github.com/miraheze/dns/pull/551 [16:30:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 12.09, 16.91, 20.18 [16:30:57] [02dns] 07RhinosF1 opened pull request 03#552: Update project-patterns.com - 13https://github.com/miraheze/dns/pull/552 [16:31:03] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.11, 17.66, 20.23 [16:31:12] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 11.72, 16.20, 19.49 [16:32:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:32:37] [02dns] 07RhinosF1 opened pull request 03#553: Update langrisser.wiki - 13https://github.com/miraheze/dns/pull/553 [16:33:36] [02dns] 07RhinosF1 opened pull request 03#554: Update programming.red - 13https://github.com/miraheze/dns/pull/554 [16:34:37] [02dns] 07RhinosF1 opened pull request 03#555: Update mh142.com - 13https://github.com/miraheze/dns/pull/555 [16:35:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.23, 16.37, 20.19 [16:35:37] [02dns] 07RhinosF1 opened pull request 03#556: Update clinitheque.fr - 13https://github.com/miraheze/dns/pull/556 [16:35:45] [02dns] 07Universal-Omega closed pull request 03#550: Update johanloopmans.nl - 13https://github.com/miraheze/dns/pull/550 [16:35:48] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e3c5fd0c0580...065c6f1b0234 [16:35:49] [02dns] 07RhinosF1 03065c6f1 - Update johanloopmans.nl (#550) [16:36:42] [02dns] 07RhinosF1 opened pull request 03#557: Update theenglishlakes.uk - 13https://github.com/miraheze/dns/pull/557 [16:38:30] [02dns] 07RhinosF1 opened pull request 03#558: Update rippaversewiki.com - 13https://github.com/miraheze/dns/pull/558 [16:39:31] [02dns] 07RhinosF1 opened pull request 03#559: Update ebcat.me - 13https://github.com/miraheze/dns/pull/559 [16:40:28] [02dns] 07RhinosF1 opened pull request 03#560: Update iceria.org - 13https://github.com/miraheze/dns/pull/560 [16:40:30] [02dns] 07Universal-Omega closed pull request 03#551: Update tintinpedia.fr - 13https://github.com/miraheze/dns/pull/551 [16:40:33] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/065c6f1b0234...e9023c547b62 [16:40:36] [02dns] 07RhinosF1 03e9023c5 - Update tintinpedia.fr (#551) [16:40:42] [02dns] 07Universal-Omega closed pull request 03#552: Update project-patterns.com - 13https://github.com/miraheze/dns/pull/552 [16:40:44] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e9023c547b62...d523b0c7914f [16:40:47] [02dns] 07RhinosF1 03d523b0c - Update project-patterns.com (#552) [16:40:53] [02dns] 07Universal-Omega closed pull request 03#553: Update langrisser.wiki - 13https://github.com/miraheze/dns/pull/553 [16:40:55] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d523b0c7914f...ea0a00df8d8f [16:40:56] [02dns] 07RhinosF1 03ea0a00d - Update langrisser.wiki (#553) [16:41:04] [02dns] 07Universal-Omega closed pull request 03#554: Update programming.red - 13https://github.com/miraheze/dns/pull/554 [16:41:07] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ea0a00df8d8f...6ed16ff7cb9e [16:41:10] [02dns] 07RhinosF1 036ed16ff - Update programming.red (#554) [16:41:16] [02dns] 07Universal-Omega closed pull request 03#555: Update mh142.com - 13https://github.com/miraheze/dns/pull/555 [16:41:17] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6ed16ff7cb9e...6a25f5b65662 [16:41:19] [02dns] 07RhinosF1 036a25f5b - Update mh142.com (#555) [16:41:26] [02dns] 07RhinosF1 opened pull request 03#561: Update eerstelijnszones.be - 13https://github.com/miraheze/dns/pull/561 [16:41:29] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6a25f5b65662...0bfe5f03b7a0 [16:41:31] [02dns] 07RhinosF1 030bfe5f0 - Update clinitheque.fr (#556) [16:41:33] [02dns] 07Universal-Omega closed pull request 03#556: Update clinitheque.fr - 13https://github.com/miraheze/dns/pull/556 [16:41:39] [02dns] 07Universal-Omega closed pull request 03#557: Update theenglishlakes.uk - 13https://github.com/miraheze/dns/pull/557 [16:41:42] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0bfe5f03b7a0...f1f76402864d [16:41:45] [02dns] 07RhinosF1 03f1f7640 - Update theenglishlakes.uk (#557) [16:41:49] [02dns] 07Universal-Omega closed pull request 03#558: Update rippaversewiki.com - 13https://github.com/miraheze/dns/pull/558 [16:41:52] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f1f76402864d...1432995dfe65 [16:41:53] [02dns] 07RhinosF1 031432995 - Update rippaversewiki.com (#558) [16:41:58] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1432995dfe65...8529584d54bb [16:42:01] [02dns] 07RhinosF1 038529584 - Update ebcat.me (#559) [16:42:04] [02dns] 07Universal-Omega closed pull request 03#559: Update ebcat.me - 13https://github.com/miraheze/dns/pull/559 [16:42:08] [02dns] 07Universal-Omega closed pull request 03#560: Update iceria.org - 13https://github.com/miraheze/dns/pull/560 [16:42:09] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8529584d54bb...62e57d88b313 [16:42:12] [02dns] 07RhinosF1 0362e57d8 - Update iceria.org (#560) [16:42:19] [02dns] 07Universal-Omega closed pull request 03#561: Update eerstelijnszones.be - 13https://github.com/miraheze/dns/pull/561 [16:42:20] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/62e57d88b313...cabc3a7a3368 [16:42:23] [02dns] 07RhinosF1 03cabc3a7 - Update eerstelijnszones.be (#561) [16:42:24] [02dns] 07RhinosF1 opened pull request 03#562: Update pyramidgames.wiki - 13https://github.com/miraheze/dns/pull/562 [16:42:57] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cabc3a7a3368...169c4f474f01 [16:42:59] [02dns] 07RhinosF1 03169c4f4 - Update pyramidgames.wiki (#562) [16:43:01] [02dns] 07Universal-Omega closed pull request 03#562: Update pyramidgames.wiki - 13https://github.com/miraheze/dns/pull/562 [16:43:16] [02dns] 07RhinosF1 opened pull request 03#563: Update puzzles.wiki - 13https://github.com/miraheze/dns/pull/563 [16:43:48] [02dns] 07Universal-Omega closed pull request 03#563: Update puzzles.wiki - 13https://github.com/miraheze/dns/pull/563 [16:43:51] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/169c4f474f01...2e31cf84bedf [16:43:52] [02dns] 07RhinosF1 032e31cf8 - Update puzzles.wiki (#563) [16:44:06] [02dns] 07RhinosF1 opened pull request 03#564: Update baharna.org - 13https://github.com/miraheze/dns/pull/564 [16:44:50] [02dns] 07Universal-Omega closed pull request 03#564: Update baharna.org - 13https://github.com/miraheze/dns/pull/564 [16:44:53] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2e31cf84bedf...c5d99370e7f9 [16:44:56] [02dns] 07RhinosF1 03c5d9937 - Update baharna.org (#564) [16:44:59] [02dns] 07RhinosF1 opened pull request 03#565: Update streamerlore.com - 13https://github.com/miraheze/dns/pull/565 [16:45:14] RECOVERY - gogigantic.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gogigantic.wiki' will expire on Thu 28 Nov 2024 04:16:05 PM GMT +0000. [16:45:31] [02dns] 07Universal-Omega closed pull request 03#565: Update streamerlore.com - 13https://github.com/miraheze/dns/pull/565 [16:45:32] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c5d99370e7f9...0eb66d520751 [16:45:34] [02dns] 07RhinosF1 030eb66d5 - Update streamerlore.com (#565) [16:45:58] [02dns] 07RhinosF1 opened pull request 03#566: Update rothwell-leeds.co.uk - 13https://github.com/miraheze/dns/pull/566 [16:46:25] [02dns] 07Universal-Omega closed pull request 03#566: Update rothwell-leeds.co.uk - 13https://github.com/miraheze/dns/pull/566 [16:46:26] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0eb66d520751...445fa12ffdb2 [16:46:27] [02dns] 07RhinosF1 03445fa12 - Update rothwell-leeds.co.uk (#566) [16:46:58] [02dns] 07RhinosF1 opened pull request 03#567: Update civwiki.org - 13https://github.com/miraheze/dns/pull/567 [16:47:26] [02dns] 07Universal-Omega closed pull request 03#567: Update civwiki.org - 13https://github.com/miraheze/dns/pull/567 [16:47:27] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/445fa12ffdb2...26359684b836 [16:47:30] [02dns] 07RhinosF1 032635968 - Update civwiki.org (#567) [16:47:45] [02dns] 07RhinosF1 opened pull request 03#568: Update widedata.org - 13https://github.com/miraheze/dns/pull/568 [16:48:16] [02dns] 07Universal-Omega closed pull request 03#568: Update widedata.org - 13https://github.com/miraheze/dns/pull/568 [16:48:17] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/26359684b836...6c2ea1bf1d60 [16:48:18] [02dns] 07RhinosF1 036c2ea1b - Update widedata.org (#568) [16:48:41] [02dns] 07RhinosF1 opened pull request 03#569: Update dariawiki.org - 13https://github.com/miraheze/dns/pull/569 [16:49:08] [02dns] 07Universal-Omega closed pull request 03#569: Update dariawiki.org - 13https://github.com/miraheze/dns/pull/569 [16:49:10] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6c2ea1bf1d60...4c94934a7a6d [16:49:12] [02dns] 07RhinosF1 034c94934 - Update dariawiki.org (#569) [16:49:44] [02dns] 07RhinosF1 opened pull request 03#570: Update lebork.info.pl - 13https://github.com/miraheze/dns/pull/570 [16:50:12] PROBLEM - www.ebcat.me - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.ebcat.me could not be found [16:50:41] [02dns] 07RhinosF1 opened pull request 03#571: Update cpmusicwiki.com - 13https://github.com/miraheze/dns/pull/571 [16:51:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.22, 21.12, 19.78 [16:51:17] [02dns] 07Universal-Omega closed pull request 03#570: Update lebork.info.pl - 13https://github.com/miraheze/dns/pull/570 [16:51:20] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4c94934a7a6d...893c43bde2f1 [16:51:22] [02dns] 07RhinosF1 03893c43b - Update lebork.info.pl (#570) [16:51:27] [02dns] 07Universal-Omega closed pull request 03#571: Update cpmusicwiki.com - 13https://github.com/miraheze/dns/pull/571 [16:51:28] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/893c43bde2f1...55ec0d1ead14 [16:51:30] [02dns] 07RhinosF1 0355ec0d1 - Update cpmusicwiki.com (#571) [16:51:32] [02dns] 07RhinosF1 opened pull request 03#572: Update burnout.wiki - 13https://github.com/miraheze/dns/pull/572 [16:51:39] PROBLEM - www.programming.red - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.programming.red could not be found [16:52:29] [02dns] 07Universal-Omega closed pull request 03#572: Update burnout.wiki - 13https://github.com/miraheze/dns/pull/572 [16:52:31] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/55ec0d1ead14...91c6dc82ddfa [16:52:34] [02dns] 07RhinosF1 0391c6dc8 - Update burnout.wiki (#572) [16:52:37] [02dns] 07RhinosF1 opened pull request 03#573: Update lostidols.wiki - 13https://github.com/miraheze/dns/pull/573 [16:52:43] [02puppet] 07redbluegreenhat created branch 03csr-p256 - 13https://github.com/miraheze/puppet [16:52:46] [02puppet] 07redbluegreenhat pushed 031 commit to 03csr-p256 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e3ef7735a8a0 [16:52:47] [02puppet] 07redbluegreenhat 03e3ef773 - SSL: Use the P-256 algorithm for private keys in CSR generation [16:53:05] [02puppet] 07redbluegreenhat opened pull request 03#3906: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3906 [16:53:09] [02puppet] 07redbluegreenhat closed pull request 03#3906: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3906 [16:53:13] [02puppet] 07coderabbitai[bot] commented on pull request 03#3906: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3906#issuecomment-2321966466 [16:53:21] [02puppet] 07redbluegreenhat deleted branch 03csr-p256 - 13https://github.com/miraheze/puppet [16:53:22] [02puppet] 07redbluegreenhat deleted branch 03csr-p256 [16:53:27] [02dns] 07Universal-Omega closed pull request 03#573: Update lostidols.wiki - 13https://github.com/miraheze/dns/pull/573 [16:53:29] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/91c6dc82ddfa...dd1693ea827c [16:53:31] [02dns] 07RhinosF1 03dd1693e - Update lostidols.wiki (#573) [16:53:33] [02puppet] 07redbluegreenhat commented on pull request 03#3906: SSL: Use the P-256 algorithm for private keys in CSR generation - 13https://github.com/miraheze/puppet/pull/3906#issuecomment-2321967334 [16:54:44] miraheze/puppet - redbluegreenhat the build passed. [16:55:07] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80AB880CEC140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [16:55:16] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/3c8ea1432af0...23a47dba1ebe [16:55:18] [02ssl] 07redbluegreenhat 0323a47db - Add miraheze.org Cloudflare Origin Certificate [16:56:32] PROBLEM - www.widedata.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.widedata.org could not be found [16:57:48] [02dns] 07RhinosF1 opened pull request 03#574: Update allthetropes.org - 13https://github.com/miraheze/dns/pull/574 [16:59:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.00, 24.31, 21.43 [16:59:25] [02dns] 07Universal-Omega closed pull request 03#574: Update allthetropes.org - 13https://github.com/miraheze/dns/pull/574 [16:59:26] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/dd1693ea827c...afab244a91b5 [16:59:27] [02dns] 07RhinosF1 03afab244 - Update allthetropes.org (#574) [16:59:44] PROBLEM - www.clinitheque.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.clinitheque.fr could not be found [17:00:43] PROBLEM - www.johanloopmans.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.johanloopmans.nl could not be found [17:00:58] PROBLEM - www.rippaversewiki.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.rippaversewiki.com could not be found [17:01:02] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.85, 23.43, 21.48 [17:02:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:04:11] PROBLEM - www.tintinpedia.fr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.tintinpedia.fr could not be found [17:05:29] [02puppet] 07RhinosF1 opened pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:05:35] [02puppet] 07coderabbitai[bot] commented on pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907#issuecomment-2321994255 [17:06:06] [02puppet] 07RhinosF1 synchronize pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:06:11] PROBLEM - www.iceria.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.iceria.org could not be found [17:06:46] miraheze/puppet - RhinosF1 the build passed. [17:07:02] [02puppet] 07RhinosF1 synchronize pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:07:18] miraheze/puppet - RhinosF1 the build passed. [17:07:28] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.808B8B3ED6140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [17:07:28] PROBLEM - www.langrisser.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.langrisser.wiki could not be found [17:07:31] [02puppet] 07RhinosF1 synchronize pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:08:42] [02puppet] 07RhinosF1 commented on pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907#issuecomment-2321999623 [17:08:51] [02puppet] 07coderabbitai[bot] commented on pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907#issuecomment-2321999821 [17:08:52] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907#issuecomment-2321994255 [17:09:02] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.68, 19.29, 20.19 [17:10:06] miraheze/puppet - RhinosF1 the build passed. [17:10:09] [02puppet] 07coderabbitai[bot] edited pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:10:14] PROBLEM - www.project-patterns.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.project-patterns.com could not be found [17:10:18] miraheze/puppet - RhinosF1 the build passed. [17:10:57] PROBLEM - www.mh142.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mh142.com could not be found [17:11:23] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907#issuecomment-2321994255 [17:12:01] RECOVERY - comprehensibleinputwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'comprehensibleinputwiki.org' will expire on Thu 28 Nov 2024 03:15:37 PM GMT +0000. [17:13:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.79, 23.47, 21.59 [17:13:18] PROBLEM - www.rippaversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80BB8838B3140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [17:15:31] PROBLEM - www.puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800B19ADF9140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [17:15:54] [02puppet] 07Universal-Omega closed pull request 03#3907: Delete check_reverse_dns - 13https://github.com/miraheze/puppet/pull/3907 [17:15:55] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±3] 13https://github.com/miraheze/puppet/compare/99952fc414ab...12ae66f0ff13 [17:15:56] [02puppet] 07RhinosF1 0312ae66f - Delete check_reverse_dns (#3907) [17:17:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:17:09] miraheze/puppet - Universal-Omega the build passed. [17:18:51] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.40, 21.55, 17.88 [17:19:38] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:20:29] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.37, 22.20, 18.71 [17:20:52] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw172 [17:21:09] PROBLEM - archives.nsgov.org - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:10] PROBLEM - moddedminecraft.wiki - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 18.63, 21.24, 17.97 [17:21:12] PROBLEM - www.thesimswiki.com - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:13] PROBLEM - wiki.k-language.dev - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:13] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:19] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:20] PROBLEM - de.berlinwiki.org - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:21] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.29, 23.47, 19.88 [17:21:21] PROBLEM - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:21:22] PROBLEM - wiki.minigamemadness.net - reverse DNS on sslhost is UNKNOWN: execvpe(/usr/lib/nagios/plugins/check_reverse_dns.py) failed: No such file or directory [17:22:29] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.08, 21.63, 18.98 [17:22:49] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.61, 21.83, 18.75 [17:24:10] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.22, 21.67, 18.41 [17:24:27] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.53, 23.55, 19.99 [17:24:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw162 mw181 [17:24:37] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 3 backends are down. mw162 mw181 mw182 [17:24:56] [02puppet] 07redbluegreenhat created branch 03phorge-cf-ca-cert - 13https://github.com/miraheze/puppet [17:24:58] [02puppet] 07redbluegreenhat pushed 031 commit to 03phorge-cf-ca-cert [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ce73934f7249 [17:25:00] [02puppet] 07redbluegreenhat 03ce73934 - Phorge: Use the Miraheze Cloudflare origin certificate [17:25:03] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 23.59, 19.68 [17:25:15] [02puppet] 07redbluegreenhat opened pull request 03#3908: Phorge: Use the Miraheze Cloudflare origin certificate - 13https://github.com/miraheze/puppet/pull/3908 [17:25:19] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.42, 23.77, 20.83 [17:25:21] [02puppet] 07coderabbitai[bot] commented on pull request 03#3908: Phorge: Use the Miraheze Cloudflare origin certificate - 13https://github.com/miraheze/puppet/pull/3908#issuecomment-2322024456 [17:26:26] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 23.18, 23.46, 20.40 [17:26:41] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [17:27:01] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.31, 23.93, 20.34 [17:27:18] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.02, 23.34, 20.96 [17:27:28] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw181 [17:28:10] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 23.52, 19.92 [17:28:28] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [17:28:34] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [17:28:44] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 27.46, 24.37, 20.81 [17:28:58] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 24.71, 21.06 [17:29:16] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.39, 22.82, 21.14 [17:32:09] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.92, 23.63, 20.74 [17:32:23] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 23.35, 21.16 [17:32:54] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.01, 23.24, 21.36 [17:33:14] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 23.88, 21.99 [17:33:21] [02puppet] 07redbluegreenhat pushed 031 commit to 03phorge-cf-ca-cert [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ce73934f7249...a851509e594e [17:33:22] [02puppet] 07redbluegreenhat 03a851509 - Add the CF origin cert to wildcard [17:33:23] [02puppet] 07redbluegreenhat synchronize pull request 03#3908: Phorge: Use the Miraheze Cloudflare origin certificate - 13https://github.com/miraheze/puppet/pull/3908 [17:34:08] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.77, 24.41, 21.26 [17:34:20] [02puppet] 07redbluegreenhat pushed 031 commit to 03phorge-cf-ca-cert [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a851509e594e...a9c6566df2a6 [17:34:22] [02puppet] 07redbluegreenhat 03a9c6566 - oops [17:34:23] [02puppet] 07redbluegreenhat synchronize pull request 03#3908: Phorge: Use the Miraheze Cloudflare origin certificate - 13https://github.com/miraheze/puppet/pull/3908 [17:34:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.45, 23.39, 23.84 [17:34:51] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 23.89, 21.83 [17:35:13] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.88, 23.46, 22.10 [17:35:25] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [17:36:08] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 16.05, 22.35, 20.99 [17:36:22] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.68, 5.06, 2.59 [17:36:25] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 22.96, 24.16, 24.11 [17:36:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.28, 22.12, 23.79 [17:36:39] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 23.03, 19.67 [17:36:42] [02puppet] 07redbluegreenhat closed pull request 03#3908: Phorge: Use the Miraheze Cloudflare origin certificate - 13https://github.com/miraheze/puppet/pull/3908 [17:36:43] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/12ae66f0ff13...2345b974081b [17:36:46] [02puppet] 07redbluegreenhat 032345b97 - Phorge: Use the Miraheze Cloudflare origin certificate (#3908) [17:36:47] [02puppet] 07redbluegreenhat deleted branch 03phorge-cf-ca-cert [17:36:48] [02puppet] 07redbluegreenhat deleted branch 03phorge-cf-ca-cert - 13https://github.com/miraheze/puppet [17:36:55] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dariawiki.org' will expire on Thu 28 Nov 2024 04:21:50 PM GMT +0000. [17:37:12] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 24.05, 22.49 [17:38:17] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 3.35, 4.95, 2.89 [17:38:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.20, 23.61, 24.16 [17:39:11] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.64, 23.92, 22.70 [17:41:00] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:41:38] [02puppet] 07redbluegreenhat pushed 031 commit to 03wildcard-fix [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5db123e7670a [17:41:41] [02puppet] 07redbluegreenhat 035db123e - SSL: Fix wildcard [17:41:43] [02puppet] 07redbluegreenhat created branch 03wildcard-fix - 13https://github.com/miraheze/puppet [17:41:43] PROBLEM - os162 Puppet on os162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:41:47] [02puppet] 07redbluegreenhat opened pull request 03#3909: SSL: Fix wildcard - 13https://github.com/miraheze/puppet/pull/3909 [17:41:52] [02puppet] 07coderabbitai[bot] commented on pull request 03#3909: SSL: Fix wildcard - 13https://github.com/miraheze/puppet/pull/3909#issuecomment-2322046685 [17:42:07] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 22.75, 21.60 [17:42:24] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:42:58] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:43:09] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.31, 23.86, 22.86 [17:43:13] [02puppet] 07redbluegreenhat closed pull request 03#3909: SSL: Fix wildcard - 13https://github.com/miraheze/puppet/pull/3909 [17:43:15] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2345b974081b...0964a9e8d9cf [17:43:17] [02puppet] 07redbluegreenhat 030964a9e - SSL: Fix wildcard (#3909) [17:43:18] RECOVERY - www.rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rippaversewiki.com' will expire on Thu 28 Nov 2024 04:33:43 PM GMT +0000. [17:43:20] [02puppet] 07redbluegreenhat deleted branch 03wildcard-fix [17:43:21] [02puppet] 07redbluegreenhat deleted branch 03wildcard-fix - 13https://github.com/miraheze/puppet [17:44:03] RECOVERY - www.puzzles.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.puzzles.wiki' will expire on Thu 28 Nov 2024 04:27:57 PM GMT +0000. [17:44:12] PROBLEM - kafka181 Puppet on kafka181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:44:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.56, 21.60, 23.23 [17:45:01] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [17:45:08] !log [alex@phorge171] restarted nginx [17:45:30] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:46:04] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:46:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:46:06] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.99, 22.59, 21.76 [17:46:33] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/miraheze-origin-cert.key] [17:46:50] Phorge is now serving the cloudflare origin certificate [17:47:50] PROBLEM - www.lostidols.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.802BF9275A140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40 [17:48:05] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 29.23, 25.03, 22.75 [17:48:06] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [17:50:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.39, 22.73, 23.64 [17:52:00] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:52:00] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.57, 23.78, 23.16 [17:52:08] So ditched tunnels? [17:52:16] yes [17:52:19] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.32, 25.28, 24.47 [17:52:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.05, 25.34, 24.24 [17:53:05] RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pyramidgames.wiki' will expire on Thu 28 Nov 2024 04:29:09 PM GMT +0000. [17:53:50] woo hoo [17:54:04] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.96, 23.96, 23.14 [17:56:04] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 24.32, 23.36 [17:58:03] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 15.97, 22.09, 22.72 [17:58:07] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 18.92, 23.14, 23.66 [17:58:19] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 11.88, 19.74, 22.43 [17:58:23] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 12.48, 21.27, 23.37 [17:59:00] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.11, 21.32, 23.44 [17:59:40] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 10.52, 16.13, 20.01 [18:00:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.96, 21.64, 23.62 [18:00:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.18, 22.13, 23.63 [18:02:00] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:02:03] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 12.09, 16.18, 20.10 [18:02:15] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 11.43, 14.93, 19.83 [18:02:19] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 11.09, 15.25, 20.38 [18:04:04] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 10.46, 14.93, 19.76 [18:04:57] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 9.52, 13.67, 19.28 [18:06:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.57, 15.60, 20.20 [18:07:00] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:08:11] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:08:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.85, 22.48, 23.05 [18:09:18] RECOVERY - os162 Puppet on os162 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:10:07] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:12:03] RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:14:34] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:15:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:15:26] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:33:39] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.42, 17.29, 15.26 [18:34:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.80, 21.40, 18.83 [18:35:37] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.96, 16.99, 15.39 [18:38:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.95, 20.34, 18.96 [18:47:25] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:14:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.89, 20.43, 18.93 [19:16:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.28, 20.39, 19.10 [19:20:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.52, 21.88, 20.07 [19:22:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.60, 18.92, 19.19 [19:31:03] Nice [19:31:27] We don't need them [19:32:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.48, 19.81, 19.01 [19:34:19] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.22, 20.04 [19:36:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.11, 19.44, 19.26 [19:44:01] So my "CNAME Cross-User Banned" error will go away on its own maybe? :) [19:47:04] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:17] I think it will [19:49:55] by the way, I don't want to rant or anything, but whoever push that breaking change to UploadWizard is an utter idioto [19:52:53] hence why #5643 complaining about making it more complicated [19:54:48] I made a task for our non-wiki HTTP services: https://issue-tracker.miraheze.org/T12527 [19:55:20] actually only part of Phorge is going through CF, phorge-static. still goes though CP [20:03:25] !log Added DNS record on miraheze.org in preparation for moving Matomo [20:03:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:32] Phorge-static will go when wikitide.net goes [20:06:42] Which will be soonish [20:06:45] @rhinosf1 do you wanna ammend the statment in minor announcement to let people know exactly whats on CF and whats not [20:07:20] If you're using a CNAME, you are on cloudflare [20:07:39] Yeah just so people know and don’t ask [20:07:45] If you're using our dns and have a subdomain of your domain, most are moved [20:07:58] If you're not any of them, you're probably cloudflare [20:08:03] You can copy that [20:08:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.21, 19.50, 17.77 [20:08:40] There is no confirmed date for when anyone else will move [20:08:46] got it [20:09:49] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/afab244a91b5...1c254b56b6d0 [20:09:50] [02dns] 07redbluegreenhat 031c254b5 - Move analytics.wikitide.net to Cloudflare [20:10:30] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1c254b56b6d0...52cbc73cf50c [20:10:31] [02dns] 07redbluegreenhat 0352cbc73 - rm extra space [20:11:58] @bluemoon0332 fyi we're going to move the whole of wikitide.net soon [20:12:06] You don't need to move individual services [20:12:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.55, 19.92, 18.42 [20:12:20] Well no-one told me [20:12:28] Moved [20:12:29] I love secret plans that people keep to themselves [20:12:34] Everything is moving to cloudflare [20:12:40] Eventually [20:12:49] I believe this is just a communication fwap [20:13:03] although secret plans are more fun [20:13:15] We just need to set it all up [20:14:01] cp* will go eventually so anything that uses that will go first [20:14:20] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/52cbc73cf50c...1cbb3812a583 [20:14:21] [02dns] 07redbluegreenhat 031cbb381 - revert [20:14:29] who is we [20:14:36] probably cosmic alpha [20:15:54] well dumb me for trying to organize stuff with phorge tasks, I didn't know the entire migration was already planned [20:18:58] We need a move wikitide.net task tbh [20:19:16] Beta should probably go very soon [20:20:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.23, 21.56, 19.68 [20:20:23] how much server space will that free up [20:20:42] Loosing cp won't save any space on cloud [20:20:48] It'll save us a bit of money though [20:21:14] should probably make sure everyone on the tech team is on the same page about migration as well if there’s confusion [20:21:18] how so? [20:21:31] i mean processing power not storage [20:22:02] Because they aren't on cloud* [20:22:09] oh? [20:22:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.54, 19.53, 19.18 [20:22:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:22:28] MacFan has updated the phab task with a list of all our owned domains that need to move [20:22:39] It will need to be mostly done by someone with registrar access [20:22:48] I forgot we own Miraheze.com and wiki [20:22:54] well wiki I remembered more [20:22:57] We own a lot [20:23:12] Wiki.surf is fun [20:23:37] I remember asking if we could expand the url shortener to all miraheze.org urls [20:23:39] How can they be? Some of them aren't even in the same country as cloud* [20:23:46] Yes [20:23:48] ooooooooh right [20:23:52] thats the point of a cache [20:24:04] yes its possible? [20:24:15] Ye just do it [20:24:23] All domains controlled by us can in url shorter [20:24:30] Is it regex based? [20:24:54] No idea [20:25:02] Read the docs, make a PR [20:25:30] Task time [20:25:33] soon™️ [20:29:47] If the extension isn’t already global should i make it? [20:30:07] I don't see why it couldn't be global [20:30:10] It shouldn't cause issues [20:31:31] is there precedent for needing that approved? [20:32:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.92, 20.81, 19.56 [20:38:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.88, 18.75, 19.24 [20:41:39] miraheze.wiki isn't actually used for anything anymore, just the root redirects to us [20:42:41] I prefer having it as an option for default domain [20:43:47] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [20:53:48] [02mw-config] 07Universal-Omega created branch 03replace-wfParseUrl - 13https://github.com/miraheze/mw-config [20:53:49] [02mw-config] 07Universal-Omega pushed 031 commit to 03replace-wfParseUrl [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/2739dd86e34e [20:53:50] [02mw-config] 07Universal-Omega 032739dd8 - Replace deprecated wfParseUrl [20:53:52] [02mw-config] 07Universal-Omega opened pull request 03#5647: Replace deprecated wfParseUrl - 13https://github.com/miraheze/mw-config/pull/5647 [20:54:47] miraheze/mw-config - Universal-Omega the build passed. [20:57:46] [02mw-config] 07Universal-Omega edited pull request 03#5647: Replace deprecated wfParseUrl - 13https://github.com/miraheze/mw-config/pull/5647 [20:58:33] btw the rest of the domains would be on the basic plan, unless we want to ask support to put them on a higher plan, since plans are per domain [20:58:53] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.51, 22.10, 20.12 [21:02:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.63, 23.28, 21.13 [21:04:03] Ye we are fine with that for all but wikitide.org cause that's used for wikis [21:04:05] I think [21:04:37] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.39, 24.65, 21.87 [21:04:39] Cause only thing on wikitide.net that needs to be cached is phorge-static and that'll be a custom hostname [21:04:57] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0964a9e8d9cf...071b66f40301 [21:04:58] [02puppet] 07Universal-Omega 03071b66f - Use namespaced RequestContext [21:05:02] But I will need @cosmicalpha to move the firewall rules to org wide actually [21:05:10] I just thought of that for the ip bans [21:05:32] They are single domain only cause I only have perms on miraheze.org not the entire account [21:05:43] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/071b66f40301...b176c79dffa0 [21:05:46] [02puppet] 07Universal-Omega 03b176c79 - Use namespaced RequestContext [21:08:24] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b176c79dffa0...4bb36d25f4b7 [21:08:26] [02puppet] 07Universal-Omega 034bb36d2 - Use namespaced RequestContext [21:10:10] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.00, 19.04, 17.18 [21:12:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:12:09] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.77, 19.25, 17.48 [21:13:48] [02mw-config] 07Universal-Omega closed pull request 03#5647: Replace deprecated wfParseUrl - 13https://github.com/miraheze/mw-config/pull/5647 [21:13:50] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ddb376f2ee28...727ef53fccdd [21:13:51] [02mw-config] 07Universal-Omega 03727ef53 - Replace deprecated wfParseUrl (#5647) [21:13:53] [02mw-config] 07Universal-Omega deleted branch 03replace-wfParseUrl [21:13:56] [02mw-config] 07Universal-Omega deleted branch 03replace-wfParseUrl - 13https://github.com/miraheze/mw-config [21:14:09] assuming you are referring to IP access rules, it seems like we would have to recreate them as account-wide and delete the single domain ones [21:14:47] miraheze/mw-config - Universal-Omega the build passed. [21:16:56] [02puppet] 07The-Voidwalker reviewed pull request 03#3885 commit - 13https://github.com/miraheze/puppet/pull/3885#discussion_r1739419155 [21:19:14] MacFan4000: there's more than just IP, might be some ranges and ASNs in there [21:19:17] But yes [21:19:45] We should do that before we serve any content from a non mh.o domain [21:19:54] Which I think is just the wikitide.org wikis [21:20:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.74, 23.35, 23.54 [21:23:30] What do I need to do? [21:24:22] Move the access rules stuff to being account wide and make sure we all can do account wide firewall before we move wikitide.org to cf [21:30:22] [02mw-config] 07Universal-Omega closed pull request 03#5598: T12214: Avoid redirects if action=raw is set - 13https://github.com/miraheze/mw-config/pull/5598 [21:30:24] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/727ef53fccdd...601834a3d0ed [21:30:27] [02mw-config] 07BlankEclair 03601834a - T12214: Avoid redirects if action=raw is set (#5598) [21:31:14] miraheze/mw-config - Universal-Omega the build passed. [21:32:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:34:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.09, 17.27, 19.71 [21:35:43] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:40:33] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:41:58] [02mw-config] 07Universal-Omega created branch 03revert-5598-T12214 - 13https://github.com/miraheze/mw-config [21:41:59] [02mw-config] 07Universal-Omega pushed 031 commit to 03revert-5598-T12214 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d6487cebb4e6 [21:42:01] [02mw-config] 07Universal-Omega 03d6487ce - Revert "T12214: Avoid redirects if action=raw is set (#5598)" [21:42:20] [02mw-config] 07Universal-Omega opened pull request 03#5648: Revert "T12214: Avoid redirects if action=raw is set" - 13https://github.com/miraheze/mw-config/pull/5648 [21:42:29] [02mw-config] 07Universal-Omega closed pull request 03#5648: Revert "T12214: Avoid redirects if action=raw is set" - 13https://github.com/miraheze/mw-config/pull/5648 [21:42:30] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/601834a3d0ed...166fcef1bf5a [21:42:33] [02mw-config] 07Universal-Omega 03166fcef - Revert "T12214: Avoid redirects if action=raw is set" (#5648) [21:42:34] [02mw-config] 07Universal-Omega deleted branch 03revert-5598-T12214 - 13https://github.com/miraheze/mw-config [21:42:35] [02mw-config] 07Universal-Omega deleted branch 03revert-5598-T12214 [21:43:25] miraheze/mw-config - Universal-Omega the build passed. [21:43:30] miraheze/mw-config - Universal-Omega the build passed. [21:46:07] Account-level-firewall is an addon we don't have [21:48:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 21.79, 23.72 [21:49:54] the ip/asn/range access rules can be applied to all account domains [21:50:01] (have to be recreated) [21:50:17] the addon would be for the other waf rules [21:52:24] Just the access rules [21:52:27] Not whole WAF [21:52:57] when you go to create an access rule, you can set it to all websites in account [21:53:20] I started recreating them as account-wide [21:53:29] Nice thanks [21:53:37] Ye last time I tried it gave me an access denied [21:53:42] So I did them all as single zone [21:53:50] I'm out for the night now [21:54:05] account access shows us both with the same access which is account admin [21:54:17] Might have changed since [21:54:34] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.96, 22.73, 23.21 [21:58:03] @cosmicalpha after this is done, you or void will need to handle moving over the rest of our domains as the ones with access to update the NS for them [21:59:12] You accidentally had domain admin rather than admin at first but recently fixed. [22:00:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.94, 19.96, 17.60 [22:02:19] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 21.09, 18.28 [22:08:36] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.39, 20.52, 16.78 [22:08:40] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.38, 19.84, 15.75 [22:10:40] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 21.50, 16.82 [22:12:40] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.69, 20.36, 16.96 [22:14:35] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.54, 18.85, 17.50 [22:16:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.03, 22.71, 22.37 [22:26:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.13, 18.17, 20.20 [22:33:43] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [22:36:06] all rules are now account-wide [22:38:38] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:58:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 14.39, 20.60, 23.02 [23:12:34] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.94, 18.17, 19.99 [23:52:14] oh yeah, how's the cloudflare migration? our wiki still doesn't use it yet it seems [23:56:03] I’ll copy my announcement here:) [23:56:23] > All custom domains using CNAME are now on CloudFlare. If you're using our dns and/or have a subdomain of your domain(wiki.domain.tld), your probably behind the cloud. If you’re not sure, check the response headers or ask one of us. There is no current ETA for the remaining custom domains to be moved, we will be sure to let y’all know as soon as they are. :miraheze: [23:56:45] oh okay [23:58:03] Complications have presented themselves unfortunately [23:58:34] I believe we’re going to have to change our DNS system since it doesn’t support what we need [23:59:17] *.wikitide.org hasn't migrated yet? [23:59:22] smt smt BIND smt smt this is what you get for copying WMF [23:59:27] uh no [23:59:32] oh okay, just making sure [23:59:45] https://issue-tracker.miraheze.org/T12057