[00:00:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:36] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:21] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:29] !log [@mwtask181] starting deploy of {'config': True} to all [00:02:36] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:38] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [00:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:10] miraheze/MirahezeMagic - amire80 the build has errored. [00:03:25] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 8693MiB (9% inode=98%); [00:05:25] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21586MiB (24% inode=98%); [00:06:04] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 23438MiB (25% inode=98%); [00:06:52] !log [@mwtask171] starting deploy of {'config': True} to all [00:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:00] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [00:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:33] [02MirahezeMagic] 07Universal-Omega closed pull request 03#484: Improve the text of Commons options - 13https://github.com/miraheze/MirahezeMagic/pull/484 [00:25:34] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/0372a365454a...1b40e38d13d0 [00:25:35] [02miraheze/MirahezeMagic] 07amire80 031b40e38 - Improve the text of Commons options (#484) [00:29:45] miraheze/MirahezeMagic - Universal-Omega the build has errored. [00:32:49] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [00:32:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:59] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [00:33:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:44] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:53:46] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [00:53:47] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:53:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:55] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [00:53:56] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:54:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:04] !log [oa@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta --wikidb=testglobal (END - exit=2) [00:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:28] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:17:26] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=321nailswiki [06:17:31] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=321nailswiki (START) [06:17:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:17:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:18:28] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=321nailswiki (END - exit=65280) [06:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:33] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=321nailswiki (START) [06:29:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=321nailswiki (END - exit=0) [06:29:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:29:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:35] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/021154844699...0cf6b2956ea9 [06:31:36] [02miraheze/dns] 07Reception123 030cf6b29 - add spiceandwolf.wiki zone [06:38:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=bugfableswiki --search-recursively /home/reception/bugfables --summary=Imported from https://bugfables.fandom.com (START) [06:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:25] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=bugfableswiki --search-recursively /home/reception/bugfables --summary=Imported from https://bugfables.fandom.com (END - exit=2) [06:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:52] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=bugfableswiki --search-recursively /home/reception/bugfables --summary=Imported from https://bugfables.fandom.com (START) [06:49:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c88726324058...62f0e70a91b0 [06:51:22] [02miraheze/mw-config] 07Universal-Omega 0362f0e70 - T11970: disable drafts [06:52:17] miraheze/mw-config - Universal-Omega the build passed. [06:53:30] !log [@test151] starting deploy of {'config': True} to test151 [06:53:31] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:53:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:01:51] !log [@mwtask181] starting deploy of {'config': True} to all [07:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:02:01] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [07:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:06:59] !log [@mwtask171] starting deploy of {'config': True} to all [07:07:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:07] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [07:07:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:26:36] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=themanaworldwiki /home/reception/themanaworld --search-recursively --summary=Imported from https://wiki.themanaworld.org/index.php/ [08:26:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:32:19] !log [salt-user@mwtask181] Renamed impostorswiki to malorittozwiki using renamewiki.py [08:32:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:46] !log sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=gengbaikewiki [08:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:57] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=gengbaikewiki (START) [08:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:37:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=gengbaikewiki (END - exit=0) [08:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:38:06] !log [salt-user@mwtask181] Renamed orthworldwiki to laculturewiki using renamewiki.py [08:38:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:13:26] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=bugfableswiki --search-recursively /home/reception/bugfables --summary=Imported from https://bugfables.fandom.com (END - exit=0) [09:13:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:48:17] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:49:57] PROBLEM - db181 SSH on db181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:20] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.220 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [09:51:17] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 40.96, 33.27, 14.51 [09:51:52] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [09:56:59] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.40, 10.77, 10.10 [09:58:53] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.18, 7.37, 8.94 [11:35:51] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=lemmingmediawikiwiki (END - exit=0) [11:35:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:42] !log [alex@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [11:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:36:52] !log [alex@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [11:36:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:46:20] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'm.miraheze.org' expires in 15 day(s) (Mon 08 Apr 2024 11:43:44 AM GMT +0000). [12:05:10] PROBLEM - mirabeta.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mirabeta.org' expires in 15 day(s) (Mon 08 Apr 2024 11:37:28 AM GMT +0000). [13:36:19] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:05:43] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:56:36] PROBLEM - www.pomologia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.pomologia.org' expires in 15 day(s) (Mon 08 Apr 2024 02:38:30 PM GMT +0000). [16:03:17] [1/2] 50$ for steam [steamcommunity.com/gift/758474483](https://u.to/gDeDIA ) [16:03:18] [2/2] @everyone [16:04:57] @brandon.wm here too [16:05:04] Donezo [16:28:45] PROBLEM - en.clockup.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.clockup.wiki' expires in 15 day(s) (Mon 08 Apr 2024 03:58:44 PM GMT +0000). [16:33:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03uploadstash-hololivewiki [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4d37155d98e2 [16:33:47] [02miraheze/mw-config] 07Universal-Omega 034d37155 - Set $wgUploadStashMaxAge to 7 days for hololivewiki [16:33:50] [02mw-config] 07Universal-Omega created branch 03uploadstash-hololivewiki - 13https://github.com/miraheze/mw-config [16:33:56] [02mw-config] 07Universal-Omega opened pull request 03#5510: T11973: set $wgUploadStashMaxAge to 7 days for hololivewiki - 13https://github.com/miraheze/mw-config/pull/5510 [16:34:14] [02mw-config] 07Universal-Omega synchronize pull request 03#5510: T11973: set $wgUploadStashMaxAge to 7 days for hololivewiki - 13https://github.com/miraheze/mw-config/pull/5510 [16:34:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03uploadstash-hololivewiki [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4d37155d98e2...cfd3a8d8f72b [16:34:17] [02miraheze/mw-config] 07Universal-Omega 03cfd3a8d - - [16:34:58] miraheze/mw-config - Universal-Omega the build passed. [16:35:06] miraheze/mw-config - Universal-Omega the build passed. [16:36:21] [02mw-config] 07Universal-Omega closed pull request 03#5510: T11973: set $wgUploadStashMaxAge to 7 days for hololivewiki - 13https://github.com/miraheze/mw-config/pull/5510 [16:36:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/62f0e70a91b0...7284e434a179 [16:36:24] [02miraheze/mw-config] 07Universal-Omega 037284e43 - T11973: set $wgUploadStashMaxAge to 7 days for hololivewiki (#5510) [16:36:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03uploadstash-hololivewiki [16:36:26] [02mw-config] 07Universal-Omega deleted branch 03uploadstash-hololivewiki - 13https://github.com/miraheze/mw-config [16:37:17] miraheze/mw-config - Universal-Omega the build passed. [16:37:46] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:55] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [16:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7284e434a179...562f5505dc2e [16:46:16] [02miraheze/mw-config] 07Universal-Omega 03562f550 - T11956: add wgRevokePermissions for cookbookwiki [16:47:10] miraheze/mw-config - Universal-Omega the build passed. [16:53:19] !log [@test151] starting deploy of {'config': True} to test151 [16:53:20] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [16:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:53:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:55:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1520d437799...6f380f02c423 [16:55:12] [02miraheze/puppet] 07Universal-Omega 036f380f0 - Disable irc::pywikibot for now [16:57:08] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:02:07] !log [@mwtask181] starting deploy of {'config': True} to all [17:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:16] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [17:02:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:10] !log [@mwtask171] starting deploy of {'config': True} to all [17:07:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:18] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [17:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:08] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta --wikidb=testglobal (END - exit=2) [17:25:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:03] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/eval.php --wiki=metawikibeta (END - exit=65280) [17:26:04] CosmicAlpha or Agent: any idea why `fortressblast.miraheze.org` redirects to a non-Miraheze wiki and why it's redirecting to the custom domain of Miraheze community banned user's wiki? I tried looking in the `ssl` repo to see if I could find the redirect, but no luck :( [17:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:45] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/eval.php --wiki=metawikibeta (END - exit=2) [17:27:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/562f5505dc2e...1fc6a634d64e [17:29:20] [02miraheze/mw-config] 07Universal-Omega 031fc6a63 - Use $wgDBname for siteGlobalID [17:30:12] miraheze/mw-config - Universal-Omega the build passed. [17:32:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1fc6a634d64e...8ccf73fbc76d [17:32:03] [02miraheze/mw-config] 07Universal-Omega 038ccf73f - Set $wgWBRepoSettings['siteGlobalID'] [17:32:25] !log [@mwtask181] starting deploy of {'config': True} to all [17:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:34] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [17:32:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:54] miraheze/mw-config - Universal-Omega the build passed. [17:36:56] !log [@mwtask171] starting deploy of {'config': True} to all [17:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:04] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [17:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:42:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8ccf73fbc76d...367046c02401 [17:42:51] [02miraheze/mw-config] 07Universal-Omega 03367046c - T11747: add $wgNamespaceLogos to ManageWikiNamespaces [17:43:37] Agent: did you see my question a little further up? [17:43:47] miraheze/mw-config - Universal-Omega the build passed. [17:44:05] dmehus: I did and I've deleted the wiki [17:44:25] Nale set up a redirect before he was locked [17:45:40] How do redirects work anyways [17:45:46] Agent: ah, but still no reason to keep it? The wiki is 'exempt from inactivity', so that may be preventing its deletion when the deletion script was last run [17:46:04] or the deletion script still hasn't been run [17:46:06] about that, we aren't running the deletion script [17:46:15] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [17:46:19] there's a bug with it currently [17:46:24] How’d ya find that wiki? [17:46:32] Orange_Star: ah, do you know how to fix it? [17:46:43] I thought the issue was with the mark as deleted not the permanent deletion script [17:46:54] pixldev: I knew about that wiki, but believe it showed up when I checked Agent's CA [17:47:08] yeah, the problem is with marking as deleted [17:47:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/367046c02401...1313e602da89 [17:47:13] [02miraheze/mw-config] 07Universal-Omega 031313e60 - T11762: enable wgRSSAllowImageTag for jwmeetingwiki [17:47:29] Orange_Star, ah [17:47:33] however a wiki marked as deleted will be permanently deleted when the script runs [17:47:37] dmehus: it shouldn't prevent it from being deleted [17:48:00] The problem was that closed wikis were getting marked as deleted prematurely [17:48:02] miraheze/mw-config - Universal-Omega the build passed. [17:48:16] not sure if they would've been deleted on the next run but a good number closed wikis were affected [17:48:17] Agent: ah, okay, that's good. But still, we could delete the redirect. Better to have it point to the default Miraheze/WikiTide page for deleted wikis [17:48:33] There is no server-side redirect [17:48:40] That redirect was achieved through JavaScript [17:48:54] Agent: I'm not sure that they were, the way John explained it to me? [17:49:57] Agent: yeah, but DNS supercedes JavaScript. How is he able to control a subdomain redirect if the subdomain does not exist in the miraheze.org zone file? [17:50:38] I'm not sure you understand how he achieved the redirect [17:50:49] He inserted code into MediaWiki:Common.js that would redirect you to his new wiki [17:50:59] I know because if you enabled safe mode, it wouldn't redirect you [17:51:10] There is no DNS or webserver redirect for it [17:51:15] Oh [17:52:16] Agent: yeah, that makes sense. I get that, but if I manually typed in `fortressblast.miraheze.org` into my web browser address bar, how is that being resolved in the first place is my question [17:52:30] wildcards [17:52:45] every subdomain in miraheze.org directs you to the cache proxies [17:52:51] Yeah, but the wildcard points to a Miraheze server [17:53:18] Our servers are finally fast enough that the custom JavaScript loaded almost immediately after you loaded the page [17:53:27] so it looked like it redirected you instantly to his new wiki [17:53:28] Something is therefore pointing to his server [17:53:37] and plus with cache, it probably was almost instantly [17:53:51] !log [@test151] starting deploy of {'config': True} to test151 [17:53:52] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:53:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:59] Agent, yes I have noticed how fast the cache proxies are. Would love to discuss that [17:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:35] that something pointing to his server is the redirect he was doing via JavaScript [17:54:55] but how do the cache proxies know to point to his server? It's got to still exist, in order for his wiki JavaScript to work [17:55:13] New servers go vroom vroom [17:55:21] the cache proxies aren't pointing to his server [17:55:27] they're pointing to the miraheze wiki [17:55:58] however you are redirected when you ran his JS code, which is on the Miraheze wiki [17:56:47] *run [17:56:57] Orange_Star, yes, thank you, that last part was what I was not understanding. So the wiki is still active in a sense to allow the JS to load? [17:57:35] Can we not perform a manual deletion of the wiki, then, if the wiki deletion script needs to be fixed, performing the manual steps the wiki deletion script would normally perform? [17:58:12] No, the wiki is no longer active, it's been deleted [17:58:21] You should see the "Wiki deleted" screen soon [17:58:25] Agent, so JS still loads on deleted wikis? [17:58:26] You probably need to clear your cache still [17:58:28] no [17:58:43] dmehus: clear browser cache [17:58:51] I haven't visited it in months and months, though, I'm surprised my browser cache is still active? [17:59:22] https://usercontent.irccloud-cdn.com/file/bfnFTGzF/1000005206.png [17:59:26] this is what I see [17:59:36] Agent: try banning varnish cache just in case [17:59:51] It shouldn't be doing anything but see the bug I told orange about yesterday [18:01:21] ah, fixed [18:01:24] local cache issue [18:01:28] my apologies :) [18:02:16] !log [@mwtask181] starting deploy of {'config': True} to all [18:02:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:25] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [18:02:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:03:11] when in doubt, blame cache [18:03:41] I haven't been on IRC in a while [18:04:07] I'm sort of waiting for a load alert to come up [18:05:12] Eh there’s other things to blame as well [18:05:16] depends on the area [18:05:46] Like what? [18:06:10] Maybe dns [18:06:20] But DNS cache makes stuff weird [18:06:58] !log [@mwtask171] starting deploy of {'config': True} to all [18:07:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:06] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [18:07:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:10] [1/2] 50$ for steam [steamcommunity.com/gift/758474483](https://u.to/gDeDIA ) [18:10:10] [2/2] @everyone [18:16:40] Agent: yeah, sorry, my bad. Should've checked that first :P [18:22:35] oh hi dmehus [18:26:46] hey CosmicAlpha :) [20:08:15] [02MirahezeMagic] 07dmehus opened pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485 [20:10:55] [02MirahezeMagic] 07dmehus synchronize pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485 [20:11:24] [02MirahezeMagic] 07dmehus synchronize pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485 [20:12:01] miraheze/MirahezeMagic - dmehus the build has errored. [20:14:37] miraheze/MirahezeMagic - dmehus the build has errored. [20:15:39] miraheze/MirahezeMagic - dmehus the build has errored. [20:20:16] [02MirahezeMagic] 07dmehus commented on pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485#issuecomment-2016593322 [20:20:32] [02MirahezeMagic] 07dmehus deleted a comment on pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485#issuecomment-2016593322 [21:27:59] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/eval.php --wiki=metawikibeta (END - exit=2) [21:28:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:28:26] ^ was testing the Title object for development of https://github.com/miraheze/CreateWiki/pull/487, no writes made [21:32:21] [02MirahezeMagic] 07dmehus synchronize pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485 [21:34:39] [02MirahezeMagic] 07dmehus synchronize pull request 03#485: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/485 [21:35:57] miraheze/MirahezeMagic - dmehus the build has errored. [21:36:36] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#487: T10683: Expose the canned responses to JavaScript - 13https://github.com/miraheze/CreateWiki/pull/487 [21:36:37] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [21:37:10] [02CreateWiki] 07redbluegreenhat edited pull request 03#487: T10683: Expose the canned responses to JavaScript - 13https://github.com/miraheze/CreateWiki/pull/487 [21:38:09] miraheze/MirahezeMagic - dmehus the build has errored. [21:38:54] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#487: T10683: Expose the canned responses to JavaScript - 13https://github.com/miraheze/CreateWiki/pull/487 [21:39:41] miraheze/CreateWiki - redbluegreenhat the build has errored. [21:42:10] miraheze/CreateWiki - redbluegreenhat the build has errored. [21:44:15] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:54:20] [02miraheze/puppet] 07Universal-Omega created branch 03changeprop 13https://github.com/miraheze/puppet/commit/6f380f02c423412a6bd4e6186c59c23d67f25ec7 [21:54:22] [02puppet] 07Universal-Omega created branch 03changeprop - 13https://github.com/miraheze/puppet [21:54:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop [+8/-0/±10] 13https://github.com/miraheze/puppet/compare/6f380f02c423...c5f7cb7863a9 [21:54:47] [02miraheze/puppet] 07Universal-Omega 03c5f7cb7 - Add changeprop [21:55:01] [02puppet] 07Universal-Omega opened pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [21:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:55:40] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03changeprop [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/c5f7cb7863a9...b6e80aa9443e [21:55:42] [02miraheze/puppet] 07github-actions 03b6e80aa - CI: lint puppet code to standards [21:55:45] [02puppet] 07github-actions[bot] synchronize pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [21:59:00] [02puppet] 07Universal-Omega synchronize pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [21:59:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b6e80aa9443e...db338efb2f89 [21:59:04] [02miraheze/puppet] 07Universal-Omega 03db338ef - Fix [22:01:40] [02puppet] 07Universal-Omega synchronize pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [22:01:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/db338efb2f89...c47a36cdd1a9 [22:01:44] [02miraheze/puppet] 07Universal-Omega 03c47a36c - - [22:04:44] [02miraheze/changeprop-deploy] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/changeprop-deploy/commit/5118ea0e993c [22:04:46] [02miraheze/changeprop-deploy] 07Universal-Omega 035118ea0 - Add initial README [22:04:47] [02changeprop-deploy] 07Universal-Omega created branch 03master - 13https://github.com/miraheze/changeprop-deploy [22:05:59] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [22:30:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:32:27] [02puppet] 07Universal-Omega synchronize pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [22:32:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c47a36cdd1a9...db089f4d2ecf [22:32:30] [02miraheze/puppet] 07Universal-Omega 03db089f4 - Fix [22:34:17] [02puppet] 07Universal-Omega synchronize pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [22:34:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/db089f4d2ecf...14c0be7322f9 [22:34:20] [02miraheze/puppet] 07Universal-Omega 0314c0be7 - Fix [22:35:25] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:49:17] [02dns] 07MacFan4000 opened pull request 03#501: Add volunteerforukraine.com zone - 13https://github.com/miraheze/dns/pull/501 [22:56:13] [02miraheze/changeprop-deploy] 07Universal-Omega pushed 032 commits to 03master [+3000/-1/±0] 13https://github.com/miraheze/changeprop-deploy/compare/5118ea0e993c...4ef5728a69e8 [22:56:15] [02miraheze/changeprop-deploy] 07Universal-Omega 03878b94e - Initial changeprop import [22:56:16] [02miraheze/changeprop-deploy] 07Universal-Omega 034ef5728 - Fix push [23:03:15] !log created changeprop151 VM [23:03:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:58] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0cf6b2956ea9...58d812a6d3c5 [23:05:59] [02miraheze/dns] 07Universal-Omega 0358d812a - Add changeprop151 to DNS [23:12:43] changeprop? [23:13:51] It's the last service (I hope) needed to migrate the job queue to Kafka [23:17:25] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9699MiB (10% inode=98%); [23:24:43] ooooooh so that’s what kafka is [23:25:01] Moving from redis? [23:26:01] Well Kafka is very complicated to explain but it is a lot more then just that. It is more reliable and ensures things get run more, but even the Kafka Job Queue uses redis somewhat (only for removing duplicates though,l not fully redis like now. [23:27:50] lol when puppetizing a server I have to install puppet agent package twice as the first run removes puppet itself lol [23:28:12] why… [23:28:22] my guess is due to versioning [23:29:18] is this related to those cases where thousands of jobs pile up and sucker punch the servers [23:29:45] this is aimed at helping that yeah [23:34:03] Kafka has been something we have wanted for years, but until the new infra we did not have even a fraction of the resources needed for it [23:34:17] grrrrrrrrr you wiki tide folk are taking away everything that makes miraheze great1!1!!11! Like 504 and 502 gateways errors1!1!1!1!1!111 and DDoSs and crashes and no images and pileing job queues!! And uh absurd server costs i think. [23:35:44] [02puppet] 07Universal-Omega closed pull request 03#3788: Add changeprop - 13https://github.com/miraheze/puppet/pull/3788 [23:35:45] [02miraheze/puppet] 07Universal-Omega deleted branch 03changeprop [23:35:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+8/-0/±10] 13https://github.com/miraheze/puppet/compare/6f380f02c423...a2c99a68467d [23:35:48] [02miraheze/puppet] 07Universal-Omega 03a2c99a6 - Add changeprop (#3788) [23:35:50] [02puppet] 07Universal-Omega deleted branch 03changeprop - 13https://github.com/miraheze/puppet [23:36:59] (Do i need to say im joking? Lmao but seriously thank yall :EpicFaceMH: ) [23:37:42] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a2c99a68467d...01f6311a6f5e [23:37:43] [02miraheze/puppet] 07Universal-Omega 0301f6311 - Fix [23:38:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/01f6311a6f5e...d61ce4e98bd2 [23:38:59] [02miraheze/puppet] 07Universal-Omega 03d61ce4e - Fix [23:40:32] PROBLEM - jobchron171 Puppet on jobchron171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:45:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d61ce4e98bd2...012b10f917cd [23:45:05] [02miraheze/puppet] 07Universal-Omega 03012b10f - Fix: remove else [23:45:11] I still prefer redis 🤷🏼‍♂️ [23:45:37] or even rabbit mq [23:47:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/012b10f917cd...2ed58e03dc64 [23:47:35] [02miraheze/puppet] 07Universal-Omega 032ed58e0 - Remove realm [23:48:24] PROBLEM - changeprop151 Puppet on changeprop151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:48:25] Redis sucks as it can't handle high volume as much and also things are not guaranteed in redis like they are in Kafka [23:48:41] boooo [23:48:43] redis 4 life [23:48:45]