[00:01:15] <MacFan4000>	 of the wikitide owned domains only miraheze.org has moved at this time
[00:02:25] <MirahezeRelay>	 <pixldev, replying to MacFan4000> Custom domains partly done waiting on an infrastructure change I believe yes?
[00:02:31] <MacFan4000>	 yup
[00:02:32] <BlankEclair>	 wait, "If you're using our dns"? how do custom domains not point to ns{1,2}.wikitide.net?
[00:02:40] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 30.79, 23.21, 18.68
[00:02:47] <MirahezeRelay>	 <pixldev> CNAME
[00:03:01] <MirahezeRelay>	 <pixldev> Well CNAME is on CF
[00:03:14] <MirahezeRelay>	 <pixldev> used for subdomains
[00:03:26] <MacFan4000>	 gdnsd doesnt support CNAME flatning, meaning that only subdomains, and root domains using cname flattening (not under our DNS) use cloudflare
[00:03:35] <BlankEclair>	 oh okay
[00:04:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.13, 17.28, 13.04
[00:04:59] <MirahezeRelay>	 <pixldev> wdym not under our dns
[00:05:08] <BlankEclair>	 using ANAMEs
[00:05:30] <BlankEclair>	 (i.e. an outsider authoritative nameserver actually fetches miraheze's ips, and returns those in the form of A records)
[00:05:57] <MacFan4000>	 many other dns providers support CNAME flattning or ANAME/ALIAS which is a valid alternative to CNAME
[00:06:36] <BlankEclair>	 i assume CNAME flattening and ANAME/ALIAS are the same thing right?
[00:07:52] <MirahezeRelay>	 <pixldev> one day I’ll understand this stuff
[00:08:15] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.59, 20.78, 14.44
[00:08:22] <MirahezeRelay>	 <pixldev> its not possible to just have our nameservwrs give CF IPs? dumb question
[00:09:10] <BlankEclair>	 i love nesting references
[00:09:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.42, 17.60, 12.37
[00:09:23] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 19.64, 13.87
[00:09:25] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw171
[00:10:15] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.12, 21.45, 15.50
[00:10:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.26, 22.69, 17.01
[00:10:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.14, 23.44, 21.02
[00:11:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 19.71, 13.74
[00:11:22] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.50, 20.80, 15.00
[00:11:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.35, 23.04, 16.35
[00:12:09] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 21.77, 15.87
[00:12:14] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 26.48, 23.03, 16.79
[00:12:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 23.94, 18.20
[00:12:50] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181
[00:12:56] <MirahezeRelay>	 <pixldev> you’re evil
[00:13:21] <icinga-miraheze>	 PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181
[00:13:23] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.17, 22.39, 16.28
[00:13:58] <BlankEclair>	 the extends syntax of <ref> is not fully finished yet :(
[00:14:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.78, 23.31, 18.63
[00:14:30] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 23.07, 17.01
[00:15:16] <icinga-miraheze>	 RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy
[00:15:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.68, 20.98, 15.67
[00:15:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.73, 23.11, 17.90
[00:16:04] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.10, 22.33, 17.51
[00:16:14] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 19.09, 22.46, 18.01
[00:16:28] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.98, 23.44, 17.89
[00:17:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 22.94, 17.05
[00:17:23] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.02, 22.84, 17.87
[00:17:29] <BlankEclair>	 > Cite error: <ref> tag with name "miraheze-tech-ops_2024-08-31" defined in <references> is not used in prior text.
[00:17:30] <BlankEclair>	 > Cite error: <ref> tag with name "miraheze-tech-ops_2024-08-31" defined in <references> group "" has no content. 
[00:17:33] <BlankEclair>	 do i... want to know why?
[00:18:13] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 27.06, 24.25, 19.17
[00:18:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 23.93, 22.10
[00:19:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.77, 24.44, 19.54
[00:19:59] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 24.13, 19.36
[00:20:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.05, 23.45, 22.19
[00:20:39] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[00:21:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 13.28, 21.08, 17.91
[00:21:22] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.23, 23.49, 19.18
[00:21:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 16.05, 21.38, 19.01
[00:21:39] <BlankEclair>	 at least i'm not importing w:Template:Refn onto metawiki, that's for sure
[00:22:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 23.22, 20.45
[00:22:25] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.98, 23.90, 19.81
[00:24:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.34, 24.30, 22.78
[00:25:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.53, 23.16, 19.46
[00:25:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 23.56, 20.39
[00:25:25] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[00:26:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.80, 23.53, 22.70
[00:27:23] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.46, 23.11, 20.49
[00:27:23] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.51, 22.76, 20.54
[00:27:50] <icinga-miraheze>	 PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.28, 22.53, 20.64
[00:28:11] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.27, 23.61, 21.25
[00:28:22] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.27, 23.54, 21.15
[00:29:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:30:12] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/puppet/compare/4bb36d25f4b7...7c2224925c76
[00:30:14] <botifico>	 [02puppet] 07Universal-Omega 037c22249 - mediawiki: reduce PHP-FPM children to 20
[00:31:19] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.32, 22.01, 20.55
[00:31:22] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.77, 19.99, 19.91
[00:31:23] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.92, 19.98, 20.08
[00:31:45] <icinga-miraheze>	 RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.48, 19.57, 19.92
[00:32:10] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.71, 19.72, 20.23
[00:32:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.87, 22.74, 21.86
[00:32:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.57, 24.50, 23.23
[00:33:19] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.08, 19.74, 19.88
[00:34:19] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 22.49, 21.37
[00:34:19] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.51, 24.88, 22.74
[00:34:34] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.74, 21.04, 22.09
[00:35:11] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.84, 23.99, 22.56
[00:35:22] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.56, 21.03, 20.35
[00:36:17] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 14.03, 19.48, 20.42
[00:37:22] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 13.64, 18.47, 19.51
[00:37:36] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.61, 17.46, 19.64
[00:40:30] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.13, 17.94, 20.33
[00:41:06] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.08, 18.16, 20.39
[00:44:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[00:52:24] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3904: nginx: add default Content-Security-Policy header - 13https://github.com/miraheze/puppet/pull/3904
[00:52:26] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03nginx-csp
[00:52:28] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03nginx-csp - 13https://github.com/miraheze/puppet
[03:00:07] <CosmicAlpha>	 I guess that PHP change really did fix flapping mediawiki load warnings
[03:01:19] <CosmicAlpha>	 I don't remember the last time my email was this quiet as I fixed flapping MW today and Prometheus yesterday lol now I have some peace rather than like 500 emails per day
[03:22:22] <botifico>	 [02dns] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1cbb3812a583...480d1c67753a
[03:22:25] <botifico>	 [02dns] 07The-Voidwalker 03480d1c6 - new wiki.surf _acme-challenge
[04:19:03] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/23a47dba1ebe...5bae3016ea61
[04:19:04] <botifico>	 [02ssl] 07WikiTideSSLBot 035bae301 - Bot: Update SSL cert for wiki.surf
[04:57:40] <icinga-miraheze>	 RECOVERY - wiki.surf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.surf' will expire on Fri 29 Nov 2024 03:20:27 AM GMT +0000.
[05:01:34] <PixDeVl>	 YAAAAAAAAAAY wiki.surf my beloved
[05:37:30] <CosmicAlpha>	 I can't believe we went from multiple load alerts per like 2 minutes to none in hours after my last change lol. Nice!
[05:37:38] <BlankEclair>	 ikr? what the hell
[05:37:54] <PixDeVl>	 you are a DARK wizard of unimaginable power
[05:48:37] <icinga-miraheze>	 PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 6.40, 3.61, 1.59
[05:49:11] <icinga-miraheze>	 PROBLEM - mem161 APT on mem161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[05:49:38] <CosmicAlpha>	 Time to go after memcached load?
[05:51:11] <icinga-miraheze>	 RECOVERY - mem161 APT on mem161 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[05:52:09] <PixDeVl>	 CosmicAlpha hm?
[05:52:30] <CosmicAlpha>	 PixDeVl: "1:48 PM <icinga-miraheze> PROBLEM - mem161 Current Load on mem161 is CRITICAL: LOAD CRITICAL - total load average: 6.40, 3.61, 1.59"
[05:52:36] <icinga-miraheze>	 PROBLEM - mem161 Current Load on mem161 is WARNING: LOAD WARNING - total load average: 1.50, 3.61, 2.15
[05:53:08] <CosmicAlpha>	 Now I know my next target lol — though memcached is probably fine tbh
[05:53:27] <PixDeVl>	 black spelll
[05:54:16] <icinga-miraheze>	 PROBLEM - mem161 memcached on mem161 is CRITICAL: connect to address 10.0.16.131 and port 11211: Connection refused
[05:54:36] <icinga-miraheze>	 RECOVERY - mem161 Current Load on mem161 is OK: LOAD OK - total load average: 0.37, 2.46, 1.90
[05:56:30] <MirahezeRelay>	 <elliethearchiver> [1/2] mem161 randomly shot up in load lol
[05:56:31] <MirahezeRelay>	 <elliethearchiver> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1279318883165147136/eq2vKCU.png?ex=66d4020e&is=66d2b08e&hm=971cddab219cdc76f03c7b01c1ccdaf506ca538a6782c8de2c497f5e52769806&
[05:56:45] <PixDeVl>	 yup
[05:56:49] <CosmicAlpha>	 Hmm that may pose a problem if it happens again
[05:57:04] <PixDeVl>	 it's been happening for the past
[05:57:05] <PixDeVl>	 well
[05:57:06] <PixDeVl>	 ever
[05:57:12] <CosmicAlpha>	 If memcached goes down so will MediaWiki. If it has high load, then MediaWiki will be slow also
[05:58:16] <icinga-miraheze>	 RECOVERY - mem161 memcached on mem161 is OK: TCP OK - 0.000 second response time on 10.0.16.131 port 11211
[05:59:54] <PixDeVl>	 CosmicAlpha any idea what causes?
[06:00:31] <CosmicAlpha>	 Nope. I will check on it later today or tomorrow.
[06:00:40] <PixDeVl>	 bet
[06:00:48] <PixDeVl>	 whenever you're free :>
[06:01:15] <CosmicAlpha>	 Its not as urgent as MediaWiki was (or even prometheus lol)
[06:01:42] <PixDeVl>	 fair
[06:01:45] <CosmicAlpha>	 Just one doing it causes no issues, if they both did at once it would cause a lot of issues.
[06:25:42] <icinga-miraheze>	 PROBLEM - mem151 Current Load on mem151 is CRITICAL: LOAD CRITICAL - total load average: 7.21, 4.48, 1.99
[06:26:49] <icinga-miraheze>	 PROBLEM - mem151 Puppet on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:28:32] <icinga-miraheze>	 PROBLEM - mem151 APT on mem151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[06:32:18] <icinga-miraheze>	 RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 29 minutes ago with 0 failures
[06:33:29] <icinga-miraheze>	 RECOVERY - mem151 APT on mem151 is OK: APT OK: 46 packages available for upgrade (0 critical updates).
[06:35:42] <icinga-miraheze>	 PROBLEM - mem151 Current Load on mem151 is WARNING: LOAD WARNING - total load average: 0.71, 3.63, 3.25
[06:37:42] <icinga-miraheze>	 RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 0.48, 2.54, 2.89
[07:43:09] <MirahezeRelay>	 <cosmicalpha> !log updated nameservers for mirabeta.org to Cloudflare
[07:43:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:43:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[08:48:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[11:17:31] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#575: Update corru.wiki - 13https://github.com/miraheze/dns/pull/575
[11:18:22] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#576: Update mwcosmos.com - 13https://github.com/miraheze/dns/pull/576
[11:19:22] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#577: Update marinebiodiversitymatrix.org - 13https://github.com/miraheze/dns/pull/577
[11:20:32] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#578: Update christipedia.nl - 13https://github.com/miraheze/dns/pull/578
[11:21:27] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#579: Update satepedia.com - 13https://github.com/miraheze/dns/pull/579
[11:22:30] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#580: Update maxcapacity.wiki - 13https://github.com/miraheze/dns/pull/580
[11:23:44] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#581: Update permanentfuturelab.wiki - 13https://github.com/miraheze/dns/pull/581
[11:24:44] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#582: Update rosettacode.org - 13https://github.com/miraheze/dns/pull/582
[11:25:28] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#583: Update grayravens.com - 13https://github.com/miraheze/dns/pull/583
[11:26:23] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#584: Update trollpasta.com - 13https://github.com/miraheze/dns/pull/584
[11:28:16] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#585: Update opendatascot.org - 13https://github.com/miraheze/dns/pull/585
[11:29:10] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#586: Update pomologia.org - 13https://github.com/miraheze/dns/pull/586
[11:29:59] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#587: Update dcmultiversewiki.com - 13https://github.com/miraheze/dns/pull/587
[11:30:46] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#588: Update agesofconflict.wiki - 13https://github.com/miraheze/dns/pull/588
[11:31:34] <botifico>	 [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/5bae3016ea61...83ac21747cfe
[11:31:35] <botifico>	 [02ssl] 07redbluegreenhat 0383ac217 - Add Mirabeta Cloudflare Origin Certificate
[11:31:40] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#589: Update durawiki.com - 13https://github.com/miraheze/dns/pull/589
[11:32:38] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#590: Update cheeseepedia.org - 13https://github.com/miraheze/dns/pull/590
[11:33:24] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#591: Update dccomicswiki.com - 13https://github.com/miraheze/dns/pull/591
[11:34:13] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#592: Update portalsofphereon.com - 13https://github.com/miraheze/dns/pull/592
[11:35:18] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#593: Update journeytheword.wiki - 13https://github.com/miraheze/dns/pull/593
[11:36:07] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#594: Update aryavratpedia.co - 13https://github.com/miraheze/dns/pull/594
[11:36:47] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#595: Update kalons-reverie.com - 13https://github.com/miraheze/dns/pull/595
[11:38:42] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#596: Delete zones/pornwiki.org - 13https://github.com/miraheze/dns/pull/596
[11:38:57] <botifico>	 [02puppet] 07redbluegreenhat created branch 03wildcard-mirabeta - 13https://github.com/miraheze/puppet
[11:38:58] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03wildcard-mirabeta [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a028ed4808ee
[11:38:59] <botifico>	 [02puppet] 07redbluegreenhat 03a028ed4 - SSL: Add Mirabeta CF origin cert to wildcard
[11:39:07] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3910: SSL: Add Mirabeta CF origin cert to wildcard - 13https://github.com/miraheze/puppet/pull/3910
[11:39:13] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3910: SSL: Add Mirabeta CF origin cert to wildcard - 13https://github.com/miraheze/puppet/pull/3910#issuecomment-2322870260
[11:39:55] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3910: SSL: Add Mirabeta CF origin cert to wildcard - 13https://github.com/miraheze/puppet/pull/3910
[11:39:57] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c2224925c76...42dfe7a2533b
[11:39:58] <botifico>	 [02puppet] 07redbluegreenhat 0342dfe7a - SSL: Add Mirabeta CF origin cert to wildcard (#3910)
[11:40:00] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03wildcard-mirabeta - 13https://github.com/miraheze/puppet
[11:40:03] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03wildcard-mirabeta
[11:44:01] <botifico>	 [02ssl] 07redbluegreenhat pushed 031 commit to 03mirabeta-cf-origin [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/00511c5429d9
[11:44:02] <botifico>	 [02ssl] 07redbluegreenhat 0300511c5 - Switch Mirabeta to the Cloudflare origin certificate
[11:44:05] <botifico>	 [02ssl] 07redbluegreenhat created branch 03mirabeta-cf-origin - 13https://github.com/miraheze/ssl
[11:44:08] <botifico>	 [02ssl] 07redbluegreenhat opened pull request 03#793: Switch Mirabeta to the Cloudflare origin certificate - 13https://github.com/miraheze/ssl/pull/793
[11:44:15] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#793: Switch Mirabeta to the Cloudflare origin certificate - 13https://github.com/miraheze/ssl/pull/793#issuecomment-2322871447
[11:46:33] <RhinosF1>	 !log test ddos alert [DO NOT ACT]
[11:46:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[11:47:02] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#793: Switch Mirabeta to the Cloudflare origin certificate - 13https://github.com/miraheze/ssl/pull/793#issuecomment-2322871447
[11:47:04] <BlankEclair>	 huh... interesting
[11:49:35] <RhinosF1>	 BlankEclair: which
[11:49:44] <BlankEclair>	 > [31/08/2024 21:46] <RhinosF1> !log test ddos alert [DO NOT ACT]
[11:50:00] <RhinosF1>	 BlankEclair: i triggered a test alert to sre@ so was logging it was me
[11:50:06] <BlankEclair>	 oh okay
[14:04:42] <icinga-miraheze>	 PROBLEM - wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 15 day(s) (Mon 16 Sep 2024 01:51:37 PM GMT +0000).
[14:30:26] <botifico>	 [02puppet] 07redbluegreenhat created branch 03mediawiki-miraheze-cf-cert - 13https://github.com/miraheze/puppet
[14:30:27] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03mediawiki-miraheze-cf-cert [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/05391f180e87
[14:30:29] <botifico>	 [02puppet] 07redbluegreenhat 0305391f1 - MediaWiki: Use the Cloudflare Origin Certificate on all servers on miraheze.org
[14:31:04] <botifico>	 [02puppet] 07redbluegreenhat opened pull request 03#3911: MediaWiki: Use the Cloudflare Origin Certificate on all servers on miraheze.org - 13https://github.com/miraheze/puppet/pull/3911
[14:31:08] <botifico>	 [02puppet] 07coderabbitai[bot] commented on pull request 03#3911: MediaWiki: Use the Cloudflare Origin Certificate on all servers on miraheze.org - 13https://github.com/miraheze/puppet/pull/3911#issuecomment-2322915655
[14:32:36] <botifico>	 [02puppet] 07redbluegreenhat closed pull request 03#3911: MediaWiki: Use the Cloudflare Origin Certificate on all servers on miraheze.org - 13https://github.com/miraheze/puppet/pull/3911
[14:32:39] <botifico>	 [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/42dfe7a2533b...b52068133153
[14:32:41] <botifico>	 [02puppet] 07redbluegreenhat 03b520681 - MediaWiki: Use the Cloudflare Origin Certificate on all servers on miraheze.org (#3911)
[14:32:44] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03mediawiki-miraheze-cf-cert - 13https://github.com/miraheze/puppet
[14:32:47] <botifico>	 [02puppet] 07redbluegreenhat deleted branch 03mediawiki-miraheze-cf-cert
[15:21:24] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#597: Revert "depool tools1.mh.wiki from bots website DNS (#171)" - 13https://github.com/miraheze/dns/pull/597
[15:21:30] <botifico>	 [02dns] 07RhinosF1 closed pull request 03#597: Revert "depool tools1.mh.wiki from bots website DNS (#171)" - 13https://github.com/miraheze/dns/pull/597
[15:22:05] <RhinosF1>	 ^ ignore
[15:32:34] <botifico>	 [02mw-config] 07songnguxyz commented on pull request 03#5643: Update UploadWizard settings for kagagawiki - 13https://github.com/miraheze/mw-config/pull/5643#issuecomment-2322934699
[15:45:37] <RhinosF1>	 !log copy some WAF rules to the beta zone
[15:45:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:51:48] <icinga-miraheze>	 PROBLEM - poupedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'poupedia.com' expires in 15 day(s) (Mon 16 Sep 2024 03:24:05 PM GMT +0000).
[15:52:00] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/83ac21747cfe...b4364a482b0a
[15:52:03] <botifico>	 [02ssl] 07WikiTideSSLBot 03b4364a4 - Bot: Update SSL cert for poupedia.com
[15:55:26] <icinga-miraheze>	 PROBLEM - bobobay.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bobobay.wiki' expires in 15 day(s) (Mon 16 Sep 2024 03:23:44 PM GMT +0000).
[15:55:36] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b4364a482b0a...0968af23d780
[15:55:37] <botifico>	 [02ssl] 07WikiTideSSLBot 030968af2 - Bot: Update SSL cert for bobobay.wiki
[16:20:35] <icinga-miraheze>	 RECOVERY - poupedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'poupedia.com' will expire on Fri 29 Nov 2024 02:53:23 PM GMT +0000.
[16:24:52] <icinga-miraheze>	 RECOVERY - bobobay.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'bobobay.wiki' will expire on Fri 29 Nov 2024 02:57:01 PM GMT +0000.
[16:47:29] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[16:48:26] <RhinosF1>	 !log testing cloudflare lb alerts
[16:48:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:56:50] <icinga-miraheze>	 PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw171 mw181 mw182
[16:57:00] <RhinosF1>	 !log disabled cloudflare lb alerts, too sensitive 
[16:58:47] <icinga-miraheze>	 RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy
[17:16:29] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:17:00] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:21:51] <icinga-miraheze>	 PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.805B19C47D140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:22:00] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:23:26] <icinga-miraheze>	 PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807BD7547F140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:27:00] <icinga-miraheze>	 [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:27:53] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#596: Delete zones/pornwiki.org - 13https://github.com/miraheze/dns/pull/596
[17:27:55] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/480d1c67753a...5cb12dbe891d
[17:27:58] <botifico>	 [02dns] 07RhinosF1 035cb12db - Delete zones/pornwiki.org (#596)
[17:28:10] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#595: Update kalons-reverie.com - 13https://github.com/miraheze/dns/pull/595
[17:28:11] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5cb12dbe891d...6b5279f62cd2
[17:28:13] <botifico>	 [02dns] 07RhinosF1 036b5279f - Update kalons-reverie.com (#595)
[17:28:20] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#594: Update aryavratpedia.co - 13https://github.com/miraheze/dns/pull/594
[17:28:23] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/6b5279f62cd2...72d1aca25a0f
[17:28:26] <botifico>	 [02dns] 07RhinosF1 0372d1aca - Update aryavratpedia.co (#594)
[17:28:30] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#593: Update journeytheword.wiki - 13https://github.com/miraheze/dns/pull/593
[17:28:31] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/72d1aca25a0f...57dd7a3ce648
[17:28:34] <botifico>	 [02dns] 07RhinosF1 0357dd7a3 - Update journeytheword.wiki (#593)
[17:28:45] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#592: Update portalsofphereon.com - 13https://github.com/miraheze/dns/pull/592
[17:28:46] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/57dd7a3ce648...110f00811aa5
[17:28:47] <botifico>	 [02dns] 07RhinosF1 03110f008 - Update portalsofphereon.com (#592)
[17:28:56] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#591: Update dccomicswiki.com - 13https://github.com/miraheze/dns/pull/591
[17:28:58] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/110f00811aa5...00c12d1f9029
[17:29:01] <botifico>	 [02dns] 07RhinosF1 0300c12d1 - Update dccomicswiki.com (#591)
[17:29:05] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#590: Update cheeseepedia.org - 13https://github.com/miraheze/dns/pull/590
[17:29:08] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/00c12d1f9029...3239a384b8c6
[17:29:09] <botifico>	 [02dns] 07RhinosF1 033239a38 - Update cheeseepedia.org (#590)
[17:29:15] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#589: Update durawiki.com - 13https://github.com/miraheze/dns/pull/589
[17:29:16] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/3239a384b8c6...536300090ff1
[17:29:19] <botifico>	 [02dns] 07RhinosF1 035363000 - Update durawiki.com (#589)
[17:29:25] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#588: Update agesofconflict.wiki - 13https://github.com/miraheze/dns/pull/588
[17:29:28] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/536300090ff1...b1f9d838f377
[17:29:30] <botifico>	 [02dns] 07RhinosF1 03b1f9d83 - Update agesofconflict.wiki (#588)
[17:29:35] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#587: Update dcmultiversewiki.com - 13https://github.com/miraheze/dns/pull/587
[17:29:38] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b1f9d838f377...4f77c87f5aae
[17:29:41] <botifico>	 [02dns] 07RhinosF1 034f77c87 - Update dcmultiversewiki.com (#587)
[17:29:50] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#586: Update pomologia.org - 13https://github.com/miraheze/dns/pull/586
[17:29:53] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4f77c87f5aae...1c1f31d4af59
[17:29:55] <botifico>	 [02dns] 07RhinosF1 031c1f31d - Update pomologia.org (#586)
[17:30:00] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#585: Update opendatascot.org - 13https://github.com/miraheze/dns/pull/585
[17:30:01] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1c1f31d4af59...38ada8c9bf24
[17:30:02] <botifico>	 [02dns] 07RhinosF1 0338ada8c - Update opendatascot.org (#585)
[17:30:10] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#584: Update trollpasta.com - 13https://github.com/miraheze/dns/pull/584
[17:30:11] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/38ada8c9bf24...cbb89df6b6d6
[17:30:14] <botifico>	 [02dns] 07RhinosF1 03cbb89df - Update trollpasta.com (#584)
[17:30:21] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#583: Update grayravens.com - 13https://github.com/miraheze/dns/pull/583
[17:30:24] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cbb89df6b6d6...7f92392a28e9
[17:30:26] <botifico>	 [02dns] 07RhinosF1 037f92392 - Update grayravens.com (#583)
[17:30:36] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7f92392a28e9...86ad83a506d1
[17:30:37] <botifico>	 [02dns] 07RhinosF1 0386ad83a - Update rosettacode.org (#582)
[17:30:38] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#582: Update rosettacode.org - 13https://github.com/miraheze/dns/pull/582
[17:30:46] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#581: Update permanentfuturelab.wiki - 13https://github.com/miraheze/dns/pull/581
[17:30:49] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/86ad83a506d1...4bc7309b7986
[17:30:50] <botifico>	 [02dns] 07RhinosF1 034bc7309 - Update permanentfuturelab.wiki (#581)
[17:30:57] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#580: Update maxcapacity.wiki - 13https://github.com/miraheze/dns/pull/580
[17:30:59] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4bc7309b7986...b64346692da4
[17:31:00] <botifico>	 [02dns] 07RhinosF1 03b643466 - Update maxcapacity.wiki (#580)
[17:31:08] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b64346692da4...8d5f08cdcd02
[17:31:09] <botifico>	 [02dns] 07RhinosF1 038d5f08c - Update satepedia.com (#579)
[17:31:12] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#579: Update satepedia.com - 13https://github.com/miraheze/dns/pull/579
[17:31:18] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#578: Update christipedia.nl - 13https://github.com/miraheze/dns/pull/578
[17:31:21] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8d5f08cdcd02...b685e1320633
[17:31:22] <botifico>	 [02dns] 07RhinosF1 03b685e13 - Update christipedia.nl (#578)
[17:31:28] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#577: Update marinebiodiversitymatrix.org - 13https://github.com/miraheze/dns/pull/577
[17:31:30] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b685e1320633...182854354f5e
[17:31:32] <botifico>	 [02dns] 07RhinosF1 031828543 - Update marinebiodiversitymatrix.org (#577)
[17:31:40] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#576: Update mwcosmos.com - 13https://github.com/miraheze/dns/pull/576
[17:31:41] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/182854354f5e...5862c8b86aaa
[17:31:42] <botifico>	 [02dns] 07RhinosF1 035862c8b - Update mwcosmos.com (#576)
[17:31:51] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#575: Update corru.wiki - 13https://github.com/miraheze/dns/pull/575
[17:31:54] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5862c8b86aaa...6af2b2aa95de
[17:31:57] <botifico>	 [02dns] 07RhinosF1 036af2b2a - Update corru.wiki (#575)
[17:32:00] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:36:22] <MirahezeRelay>	 <cosmicalpha> !log updated nameservers for miraheze.com, miraheze.wiki, orain.org, wikitide.com, and nexttide.org to Cloudflare
[17:36:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:38:51] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> nexttide??
[17:40:07] <MirahezeRelay>	 <songngu.xyz, replying to pixldev> something somethine beta stuff
[17:40:21] <MirahezeRelay>	 <pixldev> What
[17:40:51] <MirahezeRelay>	 <cosmicalpha, replying to pixldev> Yep, a beta alt domain we've had for a while from the legacy wikitide we haven't used but can use as an alt like miraheze.org and wikitide.org is available in prod, nexttide.org can be an alternative in beta.
[17:41:38] <MirahezeRelay>	 <rhinosf1, replying to pixldev> Yup
[17:41:55] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> Task updated
[17:42:54] <MirahezeRelay>	 <pixldev, replying to cosmicalpha> I think that with so many domains we could probably use on of them if we eventually decide to make our own tool forge type thing
[17:43:27] <MirahezeRelay>	 <rhinosf1, replying to pixldev> Tools would probably live on wikitide.net tbh
[17:43:41] <MacFan4000>	 that leaves wikitide.net/.org and wiki.surf
[17:44:04] <MirahezeRelay>	 <rhinosf1> And wikitide.org
[17:44:12] <MacFan4000>	 i said .net/.org
[17:44:21] <MirahezeRelay>	 <rhinosf1> Ye you did
[17:44:25] <MirahezeRelay>	 <rhinosf1> Oops
[17:44:32] <MirahezeRelay>	 <rhinosf1> Blockers for wikitide are on the task
[17:44:42] <MirahezeRelay>	 <rhinosf1> Wiki.surf is on a different registrar
[17:45:17] <CosmicAlpha>	 Yeah I can't do wiki.surf yet but can hopefully soon.
[17:46:19] <MirahezeRelay>	 <pixldev, replying to rhinosf1> Why
[17:46:34] <MirahezeRelay>	 <rhinosf1, replying to pixldev> Because it is
[17:46:41] <MirahezeRelay>	 <pixldev, replying to rhinosf1> Eh that should be reserved for internal services I think
[17:46:53] <RhinosF1>	 CosmicAlpha: has dns deployed the PRs yet
[17:48:07] <CosmicAlpha>	 Deployed manually now
[17:50:42] <icinga-miraheze>	 RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'poserdazfreebies.orain.org' will expire on Fri 29 Nov 2024 04:24:24 PM GMT +0000.
[17:52:19] <icinga-miraheze>	 PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804B45E5CA140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:52:22] <icinga-miraheze>	 RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'allthetropes.orain.org' will expire on Fri 29 Nov 2024 04:30:28 PM GMT +0000.
[17:52:40] <icinga-miraheze>	 PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.804BCCCD5E140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:52:42] <RhinosF1>	 CosmicAlpha: ty
[17:53:13] <CosmicAlpha>	 No problem
[17:53:24] <RhinosF1>	 CosmicAlpha: nexttide.org is configured. I just need to copy the WAF custom rules.
[17:53:26] <icinga-miraheze>	 PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.80DBD63E50150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:53:50] <RhinosF1>	 them errors are fine
[17:53:57] <RhinosF1>	 cloudflare will deploy the certs soon
[17:54:13] <icinga-miraheze>	 PROBLEM - www.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[17:55:05] <icinga-miraheze>	 PROBLEM - files.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[17:55:59] <icinga-miraheze>	 PROBLEM - en.pornwiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[17:56:54] <icinga-miraheze>	 PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.800B62EAA4140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:57:06] <RhinosF1>	 !log deploy WAF for nexttide.org
[17:57:08] <RhinosF1>	 CosmicAlpha: done
[17:57:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:57:54] <RhinosF1>	 MacFan4000: nexttide.org is actually now done btw
[17:58:14] <icinga-miraheze>	 PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.807B1E225A140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:58:37] <MirahezeRelay>	 <pixldev> Is anything current on that domain
[17:58:57] <botifico>	 [02ssl] 07RhinosF1 opened pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[17:59:25] <icinga-miraheze>	 PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806BEF34C3140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[17:59:32] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[17:59:54] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[18:00:03] <icinga-miraheze>	 PROBLEM - www.pomologia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.806B1AB4A7140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[18:00:12] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[18:00:25] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[18:00:37] <RhinosF1>	 pixldev: no
[18:00:48] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794#issuecomment-2322995479
[18:00:50] <RhinosF1>	 MacFan4000, CosmicAlpha: ^ can be merge
[18:01:12] <icinga-miraheze>	 PROBLEM - www.trollpasta.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.802B3E0A36150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1590:SSL alert number 40
[18:01:14] <botifico>	 [02ssl] 07Universal-Omega closed pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794
[18:01:16] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-3/±2] 13https://github.com/miraheze/ssl/compare/0968af23d780...8387d3cfc567
[18:01:17] <botifico>	 [02ssl] 07RhinosF1 038387d3c - Delete Pornwiki (#794)
[18:01:28] <RhinosF1>	 ty CosmicAlpha 
[18:01:38] <CosmicAlpha>	 No problem
[18:02:02] <RhinosF1>	 CosmicAlpha: icinga will eventually resolve btw
[18:03:07] <CosmicAlpha>	 Nice thanks
[18:03:20] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#794: Delete Pornwiki - 13https://github.com/miraheze/ssl/pull/794#issuecomment-2322995479
[18:03:43] <icinga-miraheze>	 PROBLEM - pornwiki.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[18:04:03] <RhinosF1>	 CosmicAlpha: it seems cloudflare is being slow to notice most of them
[18:04:38] <botifico>	 [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/8387d3cfc567...b68236c102b4
[18:04:39] <botifico>	 [02ssl] 07redbluegreenhat 03b68236c - Add nexttide.org Cloudflare Origin certificate
[18:06:47] * RhinosF1 has kicked it
[18:07:56] <MirahezeRelay>	 <pixldev> Is that cloudflare or the bucket
[18:08:40] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#598: Delete zones/eerstelijnszones.be - 13https://github.com/miraheze/dns/pull/598
[18:09:24] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#598: Delete zones/eerstelijnszones.be - 13https://github.com/miraheze/dns/pull/598
[18:09:27] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/6af2b2aa95de...bb3cd6ecb196
[18:09:29] <botifico>	 [02dns] 07RhinosF1 03bb3cd6e - Delete zones/eerstelijnszones.be (#598)
[18:10:57] <RhinosF1>	 CosmicAlpha: can you see what's wrong with www.thegreatwar.uk
[18:11:04] <RhinosF1>	 doesn't seem to have updated for www
[18:12:38] <RhinosF1>	 all the icinga alerts apart from www.greatwar.uk should self resolve
[18:14:24] <RhinosF1>	 oh it's not in dns
[18:14:42] <RhinosF1>	 ye it is i'm fine
[18:14:46] <RhinosF1>	 it's thegreatwar
[18:15:22] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#599: Update thegreatwar.uk - 13https://github.com/miraheze/dns/pull/599
[18:15:34] <RhinosF1>	 CosmicAlpha: i forgot the pr
[18:18:48] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#599: Update thegreatwar.uk - 13https://github.com/miraheze/dns/pull/599
[18:18:49] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bb3cd6ecb196...7de94e6fb096
[18:18:52] <botifico>	 [02dns] 07RhinosF1 037de94e6 - Update thegreatwar.uk (#599)
[18:19:29] <RhinosF1>	 CosmicAlpha: can you deploy dns and I'll refresh it
[18:21:37] <icinga-miraheze>	 RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dcmultiversewiki.com' will expire on Fri 29 Nov 2024 05:08:41 PM GMT +0000.
[18:22:13] <CosmicAlpha>	 RhinosF1: Done
[18:22:20] <icinga-miraheze>	 RECOVERY - www.journeytheword.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.journeytheword.wiki' will expire on Fri 29 Nov 2024 05:07:52 PM GMT +0000.
[18:22:22] <icinga-miraheze>	 RECOVERY - www.mwcosmos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mwcosmos.com' will expire on Fri 29 Nov 2024 05:06:15 PM GMT +0000.
[18:26:19] <icinga-miraheze>	 RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Fri 29 Nov 2024 05:09:33 PM GMT +0000.
[18:28:13] <icinga-miraheze>	 RECOVERY - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.permanentfuturelab.wiki' will expire on Fri 29 Nov 2024 05:09:06 PM GMT +0000.
[18:28:19] <icinga-miraheze>	 RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Fri 29 Nov 2024 05:09:19 PM GMT +0000.
[18:29:35] <icinga-miraheze>	 RECOVERY - www.pomologia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.pomologia.org' will expire on Fri 29 Nov 2024 05:09:02 PM GMT +0000.
[18:30:23] <icinga-miraheze>	 RECOVERY - www.trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.trollpasta.com' will expire on Fri 29 Nov 2024 05:06:20 PM GMT +0000.
[18:36:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:36:28] <RhinosF1>	 CosmicAlpha: all domains moved today are Green in cf now
[18:36:36] <RhinosF1>	 so icinga-miraheze is just slow
[18:36:40] <MirahezeRelay>	 <cosmicalpha> Nice
[18:36:59] <RhinosF1>	 CosmicAlpha: do we have any ssl errors in icinga
[18:37:13] <RhinosF1>	 if we wait for all these www ones to be green
[18:38:13] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:40:33] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[18:42:18] <MacFan4000>	 ssl checks only execute every half hour
[18:42:33] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002610385418 secs
[18:42:38] <CosmicAlpha>	 RhinosF1: yeah quite a bit of ssl errors since yesterday even if I manually recheck tbey still error
[18:44:06] <CosmicAlpha>	 74 alerts like that
[18:48:45] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[18:56:39] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[18:59:08] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[19:01:08] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0002152919769 secs
[19:04:03] <icinga-miraheze>	 PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw151
[19:04:03] <RhinosF1>	 CosmicAlpha: that's bad. Can you send me a list?
[19:06:00] <icinga-miraheze>	 RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy
[19:06:04] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw151 mw181 mw182
[19:07:16] <MirahezeRelay>	 <cosmicalpha> DMed
[19:07:20] <MirahezeRelay>	 <cosmicalpha> on Discord
[19:08:04] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[19:18:17] <botifico>	 [02mw-config] 07songnguxyz synchronize pull request 03#5645: Chore: Setting changes for certain projects - 13https://github.com/miraheze/mw-config/pull/5645
[19:19:19] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[19:36:17] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#600: Update istpcomputing.com - 13https://github.com/miraheze/dns/pull/600
[19:36:49] <botifico>	 [02dns] 07RhinosF1 opened pull request 03#601: Update farthestfrontier.wiki - 13https://github.com/miraheze/dns/pull/601
[19:36:54] <RhinosF1>	 CosmicAlpha: ^
[19:39:03] <botifico>	 [02mw-config] 07songnguxyz synchronize pull request 03#5645: Chore: Setting changes for certain projects - 13https://github.com/miraheze/mw-config/pull/5645
[19:40:00] <miraheze-github>	 miraheze/mw-config - songnguxyz the build passed.
[19:41:25] <botifico>	 [02ssl] 07RhinosF1 opened pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795
[19:41:42] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795
[19:41:54] <botifico>	 [02ssl] 07RhinosF1 synchronize pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795
[19:42:15] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795#issuecomment-2323026327
[19:43:15] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#601: Update farthestfrontier.wiki - 13https://github.com/miraheze/dns/pull/601
[19:43:16] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7de94e6fb096...061c538d3efb
[19:43:18] <botifico>	 [02dns] 07RhinosF1 03061c538 - Update farthestfrontier.wiki (#601)
[19:43:20] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795#issuecomment-2323026327
[19:43:24] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795#issuecomment-2323026327
[19:43:25] <botifico>	 [02dns] 07Universal-Omega closed pull request 03#600: Update istpcomputing.com - 13https://github.com/miraheze/dns/pull/600
[19:43:27] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/061c538d3efb...4319c9069ee3
[19:43:28] <botifico>	 [02dns] 07RhinosF1 034319c90 - Update istpcomputing.com (#600)
[19:44:22] <botifico>	 [02ssl] 07Universal-Omega closed pull request 03#795: Update certs.yaml - 13https://github.com/miraheze/ssl/pull/795
[19:44:25] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-2/±1] 13https://github.com/miraheze/ssl/compare/b68236c102b4...f80346d343a1
[19:44:28] <botifico>	 [02ssl] 07RhinosF1 03f80346d - Update certs.yaml (#795)
[19:45:52] <CosmicAlpha>	 RhinosF1: deployed
[19:48:45] <RhinosF1>	 CosmicAlpha: also can you delete petrawiki.org, I'm pretty sure it's not pointing at us I thought I'd already done it but your icinga list still has it
[19:56:21] <RhinosF1>	 CosmicAlpha: your icinga list is importing
[19:59:07] <icinga-miraheze>	 RECOVERY - wiki.aoemods.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Fri 29 Nov 2024 07:57:25 PM GMT +0000.
[20:01:01] <icinga-miraheze>	 RECOVERY - wiki.mc.marochuru.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mc.marochuru.net' will expire on Fri 29 Nov 2024 07:58:02 PM GMT +0000.
[20:01:23] <icinga-miraheze>	 RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Fri 29 Nov 2024 07:58:06 PM GMT +0000.
[20:01:24] <icinga-miraheze>	 RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Fri 29 Nov 2024 07:58:05 PM GMT +0000.
[20:01:29] <icinga-miraheze>	 RECOVERY - wiki.orvyn.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.orvyn.ca' will expire on Fri 29 Nov 2024 07:58:09 PM GMT +0000.
[20:01:51] <icinga-miraheze>	 RECOVERY - grk.repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'grk.repository.archiopedia.org' will expire on Fri 29 Nov 2024 07:55:09 PM GMT +0000.
[20:03:07] <icinga-miraheze>	 RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Fri 29 Nov 2024 07:57:49 PM GMT +0000.
[20:04:57] <RhinosF1>	 CosmicAlpha, MacFan4000: all the domains on https://issue-tracker.miraheze.org/P510 are broken and need removing from ssl/dns where they exist. Can one of you do it?
[20:05:12] <icinga-miraheze>	 RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Fri 29 Nov 2024 07:59:33 PM GMT +0000.
[20:06:22] <icinga-miraheze>	 RECOVERY - resources.africanvision.org - LetsEncrypt on sslhost is OK: OK - Certificate 'resources.africanvision.org' will expire on Fri 29 Nov 2024 07:56:19 PM GMT +0000.
[20:07:05] <RhinosF1>	 I did https://issue-tracker.miraheze.org/T12528
[20:07:16] <icinga-miraheze>	 RECOVERY - wiki.aridia.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Fri 29 Nov 2024 07:57:26 PM GMT +0000.
[20:08:10] <icinga-miraheze>	 RECOVERY - wiki.sukotuze.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sukotuze.com' will expire on Fri 29 Nov 2024 07:58:49 PM GMT +0000.
[20:08:17] <icinga-miraheze>	 RECOVERY - www.publictestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.publictestwiki.com' will expire on Fri 29 Nov 2024 07:59:52 PM GMT +0000.
[20:08:21] <icinga-miraheze>	 RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Fri 29 Nov 2024 07:59:06 PM GMT +0000.
[20:09:19] <icinga-miraheze>	 RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Fri 29 Nov 2024 07:55:21 PM GMT +0000.
[20:09:34] <icinga-miraheze>	 RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Fri 29 Nov 2024 07:42:23 PM GMT +0000.
[20:09:37] <icinga-miraheze>	 RECOVERY - wiki.zahira.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zahira.me' will expire on Fri 29 Nov 2024 07:59:09 PM GMT +0000.
[20:09:42] <icinga-miraheze>	 RECOVERY - wiki.bluevertigo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bluevertigo.org' will expire on Fri 29 Nov 2024 07:57:29 PM GMT +0000.
[20:10:02] <icinga-miraheze>	 RECOVERY - iewiki.auroville.org - LetsEncrypt on sslhost is OK: OK - Certificate 'iewiki.auroville.org' will expire on Fri 29 Nov 2024 07:55:15 PM GMT +0000.
[20:10:08] <icinga-miraheze>	 RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Fri 29 Nov 2024 07:58:47 PM GMT +0000.
[20:10:20] <icinga-miraheze>	 RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Fri 29 Nov 2024 07:56:12 PM GMT +0000.
[20:10:47] <icinga-miraheze>	 RECOVERY - wiki.controlistes.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.controlistes.fr' will expire on Fri 29 Nov 2024 06:59:16 PM GMT +0000.
[20:11:30] <icinga-miraheze>	 RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Fri 29 Nov 2024 07:57:55 PM GMT +0000.
[20:13:46] <icinga-miraheze>	 RECOVERY - wiki.gestaltonline.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gestaltonline.ca' will expire on Fri 29 Nov 2024 07:57:51 PM GMT +0000.
[20:14:45] <icinga-miraheze>	 RECOVERY - www.glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.glitchcity.wiki' will expire on Fri 29 Nov 2024 07:59:35 PM GMT +0000.
[20:16:00] <icinga-miraheze>	 RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Fri 29 Nov 2024 07:57:33 PM GMT +0000.
[20:16:35] <icinga-miraheze>	 PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns]
[20:17:34] <icinga-miraheze>	 RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Fri 29 Nov 2024 07:57:33 PM GMT +0000.
[20:19:10] <icinga-miraheze>	 RECOVERY - repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'repository.archiopedia.org' will expire on Fri 29 Nov 2024 07:56:16 PM GMT +0000.
[20:19:27] <icinga-miraheze>	 RECOVERY - wiki.jeremyboyd.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jeremyboyd.org' will expire on Fri 29 Nov 2024 07:57:52 PM GMT +0000.
[20:19:49] <RhinosF1>	 MacFan4000, CosmicAlpha: also https://issue-tracker.miraheze.org/P511
[20:20:21] <icinga-miraheze>	 RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Fri 29 Nov 2024 07:57:56 PM GMT +0000.
[20:20:39] <icinga-miraheze>	 RECOVERY - wiki.closai.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.closai.com' will expire on Fri 29 Nov 2024 07:57:29 PM GMT +0000.
[20:20:57] <icinga-miraheze>	 RECOVERY - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Fri 29 Nov 2024 07:57:50 PM GMT +0000.
[20:21:29] <icinga-miraheze>	 RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Fri 29 Nov 2024 07:00:17 PM GMT +0000.
[20:21:56] <icinga-miraheze>	 RECOVERY - wiki.meeusen.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meeusen.net' will expire on Fri 29 Nov 2024 07:58:03 PM GMT +0000.
[20:22:05] <icinga-miraheze>	 RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Fri 29 Nov 2024 07:59:53 PM GMT +0000.
[20:22:55] <icinga-miraheze>	 RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Fri 29 Nov 2024 07:58:02 PM GMT +0000.
[20:23:36] <icinga-miraheze>	 RECOVERY - lcn.zfc.id.lv - LetsEncrypt on sslhost is OK: OK - Certificate 'lcn.zfc.id.lv' will expire on Fri 29 Nov 2024 07:55:16 PM GMT +0000.
[20:25:53] <icinga-miraheze>	 RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Fri 29 Nov 2024 07:57:35 PM GMT +0000.
[20:25:59] <icinga-miraheze>	 RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Fri 29 Nov 2024 07:57:57 PM GMT +0000.
[20:28:12] <icinga-miraheze>	 RECOVERY - www.lostidols.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lostidols.wiki' will expire on Fri 29 Nov 2024 07:05:01 PM GMT +0000.
[20:34:06] <icinga-miraheze>	 RECOVERY - gufengcheng.top - LetsEncrypt on sslhost is OK: OK - Certificate 'gufengcheng.top' will expire on Fri 29 Nov 2024 08:05:47 PM GMT +0000.
[20:45:34] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000.
[20:52:19] <MacFan4000>	 Will do
[20:52:54] <MacFan4000>	 Having those list makes it simple as I have scripts that read it and do all the work
[20:53:43] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/4319c9069ee3...8cbaecb4c108
[20:53:46] <botifico>	 [02dns] 07Universal-Omega 038cbaecb - Delete zones/psychevos.org
[20:54:15] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/8cbaecb4c108...fcb146b5a17f
[20:54:16] <botifico>	 [02dns] 07Universal-Omega 03fcb146b - Delete zones/shiroko.us
[20:54:36] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/fcb146b5a17f...478b9063986f
[20:54:39] <botifico>	 [02dns] 07Universal-Omega 03478b906 - Delete zones/smafallfest.org
[20:55:08] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/478b9063986f...bf53831e6f51
[20:55:09] <botifico>	 [02dns] 07Universal-Omega 03bf53831 - Delete zones/touhouplace.wiki
[20:56:26] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/bf53831e6f51...78bcc016f5d0
[20:56:28] <botifico>	 [02dns] 07Universal-Omega 0378bcc01 - Delete zones/encyclopedia.eu.org
[20:56:37] <MirahezeRelay>	 <rhinosf1> Cool
[20:58:00] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/78bcc016f5d0...324db740c501
[20:58:02] <botifico>	 [02dns] 07Universal-Omega 03324db74 - Delete zones/wkmr.liao.media
[21:02:36] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-24/±2] 13https://github.com/miraheze/ssl/compare/f80346d343a1...9139f9bf8d02
[21:02:37] <botifico>	 [02ssl] 07MacFan4000 039139f9b - remove non-pointed domains
[21:04:44] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9139f9bf8d02...b5b3cfd1e372
[21:04:47] <botifico>	 [02ssl] 07MacFan4000 03b5b3cfd - Update redirects.yaml
[21:05:10] <icinga-miraheze>	 RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[21:10:14] <botifico>	 [02ssl] 07Universal-Omega created branch 03remove-petrawiki - 13https://github.com/miraheze/ssl
[21:10:15] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/0d9f2bd407d2
[21:10:16] <botifico>	 [02ssl] 07Universal-Omega 030d9f2bd - Remove petrawiki.org
[21:10:19] <botifico>	 [02ssl] 07Universal-Omega opened pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:10:25] <botifico>	 [02ssl] 07coderabbitai[bot] commented on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:10:52] <RhinosF1>	 MacFan4000: what wiki has airlineinsider.org
[21:11:03] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0d9f2bd407d2...0e346523a64c
[21:11:06] <botifico>	 [02ssl] 07Universal-Omega 030e34652 - Update redirects.yaml
[21:11:09] <botifico>	 [02ssl] 07Universal-Omega synchronize pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:11:24] <RhinosF1>	 MacFan4000: and who requested it
[21:11:29] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/0e346523a64c...697d2ae08c93
[21:11:29] <botifico>	 [02ssl] 07Universal-Omega 03697d2ae - Delete certificates/en.petrawiki.org.crt
[21:11:32] <botifico>	 [02ssl] 07Universal-Omega synchronize pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:11:45] <MacFan4000>	 airlineinsiderwiki which appears to have been deleted
[21:11:47] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/697d2ae08c93...aafb9b027719
[21:11:50] <botifico>	 [02ssl] 07Universal-Omega 03aafb9b0 - Delete certificates/files.petrawiki.org.crt
[21:11:51] <botifico>	 [02ssl] 07Universal-Omega synchronize pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:11:53] <MacFan4000>	 https://airlineinsider.miraheze.org
[21:12:02] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:12:05] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:12:07] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/aafb9b027719...544b1fe09766
[21:12:10] <botifico>	 [02ssl] 07Universal-Omega 03544b1fe - Delete certificates/hr.petrawiki.org.crt
[21:12:11] <botifico>	 [02ssl] 07Universal-Omega synchronize pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:12:24] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03remove-petrawiki [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/544b1fe09766...452917028b43
[21:12:25] <botifico>	 [02ssl] 07Universal-Omega 034529170 - Delete certificates/translate.petrawiki.org.crt
[21:12:28] <botifico>	 [02ssl] 07Universal-Omega synchronize pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:12:38] <RhinosF1>	 MacFan4000: I can delete it rather than fix it then
[21:12:45] <RhinosF1>	 Can you remove that custom domain then
[21:12:47] <MacFan4000>	 and requested by https://issue-tracker.miraheze.org/p/Qlyoung/
[21:12:57] <MirahezeRelay>	 <cosmicalpha, replying to RhinosF1> https://github.com/miraheze/ssl/pull/796
[21:13:18] <botifico>	 [02ssl] 07Universal-Omega closed pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796
[21:13:19] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-4/±2] 13https://github.com/miraheze/ssl/compare/b5b3cfd1e372...86f553ee8366
[21:13:20] <botifico>	 [02ssl] 07Universal-Omega 0386f553e - Remove petrawiki.org (#796)
[21:13:21] <botifico>	 [02ssl] 07Universal-Omega deleted branch 03remove-petrawiki
[21:13:22] <botifico>	 [02ssl] 07Universal-Omega deleted branch 03remove-petrawiki - 13https://github.com/miraheze/ssl
[21:13:36] <RhinosF1>	 CosmicAlpha: hopefully icinga is blank now of errors
[21:14:08] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/86f553ee8366...7f6dd7c3a28d
[21:14:11] <botifico>	 [02ssl] 07MacFan4000 037f6dd7c - Delete certificates/airlineinsider.org.crt
[21:14:22] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:14:31] <botifico>	 [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7f6dd7c3a28d...9f400535437f
[21:14:34] <botifico>	 [02ssl] 07MacFan4000 039f40053 - Update certs.yaml
[21:14:40] <MirahezeRelay>	 <cosmicalpha> RhinosF1: privacywiki.xyz also
[21:14:42] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:14:44] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:14:47] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:14:54] <MirahezeRelay>	 <cosmicalpha> and airlineinsider
[21:15:15] <botifico>	 [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#796: Remove petrawiki.org - 13https://github.com/miraheze/ssl/pull/796#issuecomment-2323043945
[21:15:17] <RhinosF1>	 CosmicAlpha: privacy wiki is broken
[21:15:22] <RhinosF1>	 Airline insider we just killed
[21:15:27] <RhinosF1>	 Privacywiki can be dropped
[21:16:02] <MirahezeRelay>	 <pixldev> Are all the broken wikis deleted
[21:16:12] <MacFan4000>	 @pixldev yes
[21:16:21] <MirahezeRelay>	 <rhinosf1, replying to pixldev> some long don't point at us too
[21:17:01] <MirahezeRelay>	 <rhinosf1> that makes 416 active custom hostnames!
[21:17:40] <MirahezeRelay>	 <rhinosf1> isocasg.org is now the only broken wiki
[21:17:47] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/324db740c501...4cf607fecb00
[21:17:50] <botifico>	 [02dns] 07Universal-Omega 034cf607f - Delete zones/privacywiki.xyz
[21:17:52] <MirahezeRelay>	 <pixldev> A records?
[21:18:10] <MacFan4000>	 07I have previously removed big batches of them now and then
[21:18:11] <MirahezeRelay>	 <rhinosf1> with wiki.surf, wikitide.net and phorge-static all pending cause we haven't moved them yet
[21:18:15] <MirahezeRelay>	 <rhinosf1, replying to pixldev> yup
[21:18:26] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/9f400535437f...4d15c8ccd843
[21:18:28] <MirahezeRelay>	 <rhinosf1> they are pointing directly at cp* as far as I can tell
[21:18:29] <botifico>	 [02ssl] 07Universal-Omega 034d15c8c - Delete certificates/privacywiki.xyz.crt
[21:18:32] <MirahezeRelay>	 <rhinosf1> I left a message for them
[21:18:46] <MirahezeRelay>	 <rhinosf1> you can still access the wiki through cp*
[21:18:58] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4d15c8ccd843...3bf4b3ef0049
[21:19:01] <botifico>	 [02ssl] 07Universal-Omega 033bf4b3e - Update certs.yaml
[21:19:02] <MirahezeRelay>	 <pixldev> cp* or a specific server
[21:19:13] <MirahezeRelay>	 <rhinosf1> didn't check which they'd added
[21:19:47] <MirahezeRelay>	 <rhinosf1> cp36/7
[21:20:26] <MirahezeRelay>	 <pixldev> Is the wiki actice?
[21:20:52] <MirahezeRelay>	 <rhinosf1> yup
[21:21:16] <MirahezeRelay>	 <pixldev> crates active?
[21:21:22] <MirahezeRelay>	 <rhinosf1> yep
[21:21:39] <MirahezeRelay>	 <pixldev> Already contacted?
[21:21:41] <MirahezeRelay>	 <rhinosf1> they have an undetermined amount of time to fix it
[21:21:44] <MirahezeRelay>	 <rhinosf1, replying to pixldev> yup
[21:22:54] <MirahezeRelay>	 <pixldev> cool
[21:22:59] <MirahezeRelay>	 <pixldev> if they don’t respond?
[21:24:05] <MirahezeRelay>	 <rhinosf1> the custom domain will be removed when we kill cp*
[21:24:33] <botifico>	 [02ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3bf4b3ef0049...fc8d5ac56b23
[21:24:34] <botifico>	 [02ssl] 07Universal-Omega 03fc8d5ac - Use additional_domain for orain.org
[21:24:39] <MirahezeRelay>	 <rhinosf1> godaddy have the same problem as gdnsd
[21:24:55] <MirahezeRelay>	 <rhinosf1> they will need to move to a less shitty registrar
[21:25:13] <MirahezeRelay>	 <rhinosf1> or at least move dns off goddady
[21:25:22] <MirahezeRelay>	 <pixldev> Is this the only one on GD?
[21:25:33] <MirahezeRelay>	 <rhinosf1> seems to be the only one broken
[21:25:39] <MirahezeRelay>	 <pixldev, replying to rhinosf1> which registars are not shitty
[21:26:07] <MirahezeRelay>	 <rhinosf1, replying to pixldev> pretty much any other one that isn't a free one
[21:26:17] <MirahezeRelay>	 <rhinosf1> a lot of the broken domains were from free registrars
[21:26:21] <MirahezeRelay>	 <rhinosf1> domains aren't free
[21:26:28] <MirahezeRelay>	 <rhinosf1> it's too good to be true, seriously
[21:26:38] <MirahezeRelay>	 <rhinosf1> also i'm off to get a shower
[21:26:42] <MirahezeRelay>	 <rhinosf1> it's 22:26 now
[21:27:04] <MirahezeRelay>	 <pixldev> What are you’re thoughts on free subdomain services
[21:27:15] <MirahezeRelay>	 <rhinosf1, replying to pixldev> not free
[21:27:33] <MirahezeRelay>	 <rhinosf1> you're giving away something
[21:27:36] <MirahezeRelay>	 <rhinosf1> just not money
[21:27:55] <MirahezeRelay>	 <pixldev> Fair fair
[21:28:27] <MirahezeRelay>	 <pixldev> although not to say all non profits offering subdomains are evil
[21:29:00] <MirahezeRelay>	 <rhinosf1> If you're with us it's your time and sanity you give away
[21:30:19] <MirahezeRelay>	 <pixldev> And to some small extend control
[21:30:52] <MirahezeRelay>	 <pixldev> although miraheze in exchange for following our polices and things we don’t allow you to do is a pretty fucking good tradeoff
[21:35:11] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b52068133153...0ebe408cc929
[21:35:13] <botifico>	 [02puppet] 07Universal-Omega 030ebe408 - Add support for monitoring_url
[21:37:49] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ebe408cc929...c85667e3d97d
[21:37:50] <botifico>	 [02puppet] 07Universal-Omega 03c85667e - Actually just use sslname if present
[21:42:32] <icinga-miraheze>	 PROBLEM - luxuryelevator.miraheze.org - Sectigo on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[21:42:48] <RhinosF1>	 CosmicAlpha: is ^ you?
[21:43:11] <CosmicAlpha>	 Yeah fixing
[21:44:07] <icinga-miraheze>	 PROBLEM - allthetropes.miraheze.org - Sectigo on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[21:44:10] <icinga-miraheze>	 PROBLEM - thelangyalist.miraheze.org - Sectigo on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[21:45:06] <icinga-miraheze>	 PROBLEM - strinova.miraheze.org - Sectigo on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[21:45:23] <icinga-miraheze>	 PROBLEM - fanpedia.miraheze.org - Sectigo on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[21:45:55] <icinga-miraheze>	 RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[21:46:00] <icinga-miraheze>	 RECOVERY - *.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Wed 30 Oct 2024 11:03:09 PM GMT +0000.
[21:46:05] <icinga-miraheze>	 RECOVERY - fanpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[21:46:09] <icinga-miraheze>	 RECOVERY - luxuryelevator.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[21:46:13] <icinga-miraheze>	 RECOVERY - strinova.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[21:46:17] <icinga-miraheze>	 RECOVERY - thelangyalist.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000.
[21:46:41] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c85667e3d97d...dd2de5e5b694
[21:46:45] <botifico>	 [02puppet] 07Universal-Omega 03dd2de5e - Only use sslname for wildcards
[21:47:02] <CosmicAlpha>	 RhinosF1: https://github.com/miraheze/puppet/commit/dd2de5e5b694fb0350210bd4dfbf88ba7733e4cc
[21:47:12] <CosmicAlpha>	 Was doing something to fix wikitide.org monitoring
[21:48:53] <CosmicAlpha>	 We now have 0!!! critical alerts in icinga (first time I can ever remember)
[21:49:17] <MirahezeRelay>	 <pixldev> That’s.. is that awesome sad or both
[21:50:34] <RhinosF1>	 Both
[21:50:42] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fc8d5ac56b23...326b8cdc58a0
[21:50:45] <botifico>	 [02ssl] 07WikiTideSSLBot 03326b8cd - Bot: Update SSL cert for rodzinka.wiki
[21:51:04] <RhinosF1>	 CosmicAlpha: we can remove most ssl alerts to be honest
[21:51:20] <RhinosF1>	 If it's using the origin cert, we only need to monitor the origin cert once
[21:52:26] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/326b8cdc58a0...6a543336038c
[21:52:29] <botifico>	 [02ssl] 07WikiTideSSLBot 036a54333 - Bot: Update SSL cert for exordium.wiki
[21:53:04] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6a543336038c...7b44c9f7603d
[21:53:06] <botifico>	 [02ssl] 07WikiTideSSLBot 037b44c9f - Bot: Update SSL cert for farthestfrontier.wiki
[21:53:29] <CosmicAlpha>	 ^ those were me btw to fix the warning level alerts for SSL expire
[21:54:15] <RhinosF1>	 CosmicAlpha: if any that are on cloudflare get close to expiry before we switch everything over, they should be just moved to the origin cert
[21:55:08] <RhinosF1>	 Any of the 416 domains in cloudflare can be moved to using that cert instead
[21:58:18] <icinga-miraheze>	 RECOVERY - rodzinka.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rodzinka.wiki' will expire on Fri 29 Nov 2024 08:52:05 PM GMT +0000.
[21:58:24] <icinga-miraheze>	 RECOVERY - farthestfrontier.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'farthestfrontier.wiki' will expire on Fri 29 Nov 2024 08:54:27 PM GMT +0000.
[21:58:28] <icinga-miraheze>	 RECOVERY - exordium.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'exordium.wiki' will expire on Fri 29 Nov 2024 08:53:51 PM GMT +0000.
[22:04:28] <icinga-miraheze>	 RECOVERY - graphite151 Disk Space on graphite151 is OK: DISK OK - free space: / 22328MiB (80% inode=94%);
[22:09:20] <botifico>	 [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4cf607fecb00...b154a40c4c72
[22:09:23] <botifico>	 [02dns] 07Universal-Omega 03b154a40 - -
[22:14:13] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b44c9f7603d...a2379e30f553
[22:14:16] <botifico>	 [02ssl] 07WikiTideSSLBot 03a2379e3 - Bot: Update SSL cert for wikitide.net
[22:21:08] <MirahezeRelay>	 <bluemoon0332> wikitide.net got renewed?
[22:21:59] <MirahezeRelay>	 <bluemoon0332> slapd should be restarted on ldap171, last time I renewed that cert slapd did not restart automatically upon seeing the new cert
[22:22:11] <icinga-miraheze>	 RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Fri 29 Nov 2024 09:15:37 PM GMT +0000.
[22:22:49] <CosmicAlpha>	 bluemoon0332: yeah, and will do
[22:26:23] <icinga-miraheze>	 RECOVERY - cloud15 IPMI Sensors on cloud15 is OK: IPMI Status: OK
[22:26:49] <CosmicAlpha>	 I am just going through fixing every alert in icinga lol
[22:26:54] <BlankEclair>	 <3
[22:27:05] <CosmicAlpha>	 3 left
[22:28:55] <icinga-miraheze>	 RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 3 seconds old and 146264 bytes
[22:29:19] <icinga-miraheze>	 RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 4 seconds old and 146034 bytes
[22:29:42] <icinga-miraheze>	 RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 4 seconds old and 146409 bytes
[22:30:11] <CosmicAlpha>	 All done!
[22:30:32] <CosmicAlpha>	 Never since I have been here in 2020 have we had icinga 100% cleared
[22:31:13] <MirahezeRelay>	 <pixldev> hell yeah
[22:31:26] <MirahezeRelay>	 <pixldev> two days of records after another
[22:31:32] <BlankEclair>	 all like 75% (random number) took was one (1) config change
[22:33:41] <CosmicAlpha>	 It definitely won't last long but at least the consistent ones for months are resolved.
[22:35:43] <MirahezeRelay>	 <pixldev> What would be a proper fix
[22:36:48] <MirahezeRelay>	 <cosmicalpha> Well it was a proper fix but something is bound to flap at some point
[22:37:19] <BlankEclair>	 this reminds me of the time where someone saved a company half a million dollars
[22:37:57] <BlankEclair>	 https://ludic.mataroa.blog/blog/i-accidentally-saved-half-a-million-dollars/
[22:38:11] <BlankEclair>	 (except that miraheze doesn't have the corporate bs this person has)
[22:38:22] <BlankEclair>	 s/this person/the company/