[00:02:13] You may need to insert extension content models like wikibase ones or template styles one. I can look in a bit. [00:02:14] RhinosF1, ack, yikes, I have no idea what the issue is :( [00:02:19] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown) [00:02:57] Universal_Omega, ack, thanks :) [00:03:37] Ty Universal_Omega [00:03:49] dmehus: technology hasn't liked me today [00:03:58] No problem. [00:04:15] RhinosF1, oh heh [00:04:26] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8036 MB (20% inode=96%); [00:05:02] Universal_Omega: we need to be cautious of the fact appservers keep flapping [00:05:22] I think I closed my lid so my jobs script might have been killed although it looked calm [00:05:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.66, 24.35, 20.04 [00:05:43] dmehus: every fire alarm decided to run out of battery at once [00:06:18] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:40] PROBLEM - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.marinebiodiversitymatrix.org could not be found [00:06:41] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [00:06:44] PROBLEM - lakehub.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lakehub.ga could not be found [00:06:47] PROBLEM - sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sdiy.info could not be found [00:06:48] PROBLEM - tools1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tools1.miraheze.wiki could not be found [00:06:48] PROBLEM - www.recyclerie-uzege.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.recyclerie-uzege.org could not be found [00:06:49] PROBLEM - wiki.homiedb.party - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.homiedb.party could not be found [00:06:51] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found [00:06:51] PROBLEM - sopel.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for sopel.bots.miraheze.wiki could not be found [00:06:52] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [00:06:53] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [00:06:54] PROBLEM - warmage.necrocity.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for warmage.necrocity.co.uk could not be found [00:06:55] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [00:06:56] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tep.wiki could not be found [00:06:56] PROBLEM - wiki.xysspon.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.xysspon.com could not be found [00:06:58] PROBLEM - wiki.mnzp.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mnzp.xyz could not be found [00:07:00] PROBLEM - history.sdtef.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for history.sdtef.org could not be found [00:07:01] Okay fun we're alerting again [00:07:02] PROBLEM - www.baharna.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.baharna.org could not be found [00:07:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::1c30/cpweb [00:07:24] Not sure what to do [00:07:29] It's not php-fpm [00:07:30] are those rDNS alerts related to/a knock-on effect from the high load issues? [00:07:36] dmehus: yep [00:07:39] ah [00:07:48] have we updated to Debian Bullseye yet? [00:07:48] There's not much we can do [00:07:55] No it's out in 2 weeks [00:07:58] PROBLEM - nonciclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for nonciclopedia.org could not be found [00:07:58] ah [00:08:05] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for you.r-fit.cc could not be found [00:08:12] I've asked them to just do the new servers on Buster [00:08:14] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 19986 bytes in 0.392 second response time [00:08:20] But we're pretty much at capacity [00:08:29] ah [00:08:32] So doing stuff is just destroying us [00:09:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:10:52] I'll try and look once I've slept a bit if it doesn't calm down [00:11:04] okay, sounds good [00:11:21] R4356th might have to use one of his bot/alt accounts for the time being when he wakes up [00:11:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.75, 23.20, 21.28 [00:11:57] If we really have to I can probably trick it into skipping and repair it a later date [00:12:12] I'm more worried about the fact we're borderline going down if load doesn't calm [00:12:17] Please ping if we do [00:12:24] Phone isn't on do not disturb [00:13:31] RECOVERY - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: SSL OK - www.marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:31] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitaldesignhq.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:33] RECOVERY - lakehub.ga - reverse DNS on sslhost is OK: SSL OK - lakehub.ga reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:37] RECOVERY - warmage.necrocity.co.uk - reverse DNS on sslhost is OK: SSL OK - warmage.necrocity.co.uk reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:38] ok, sounds good [00:13:39] and yeah [00:13:40] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:41] RECOVERY - tools1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - tools1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:43] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:46] RECOVERY - wiki.xysspon.com - reverse DNS on sslhost is OK: SSL OK - wiki.xysspon.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:47] RECOVERY - www.recyclerie-uzege.org - reverse DNS on sslhost is OK: SSL OK - www.recyclerie-uzege.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:47] RECOVERY - petrawiki.org - reverse DNS on sslhost is OK: SSL OK - petrawiki.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:48] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:49] RECOVERY - wiki.homiedb.party - reverse DNS on sslhost is OK: SSL OK - wiki.homiedb.party reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:50] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:50] RECOVERY - sopel.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - sopel.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:13:51] RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: SSL OK - wiki.fraterniterebellion.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:52] RECOVERY - wiki.mnzp.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.mnzp.xyz reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:52] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: SSL OK - history.sdtef.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:13:56] RECOVERY - www.baharna.org - reverse DNS on sslhost is OK: SSL OK - www.baharna.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:14:46] RECOVERY - nonciclopedia.org - reverse DNS on sslhost is OK: SSL OK - nonciclopedia.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:15:05] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:19:12] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.15, 3.54, 3.00 [00:19:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.20, 22.83, 21.47 [00:19:32] !log sudo -u www-data php /srv/mediawiki/w/maintenance/populateContentTables.php --wiki=afterlifewiki [00:19:40] Universal_Omega, fixed afterlifewiki [00:19:53] * dmehus crosses his fingers Voidwalker's command fixed it [00:19:58] unless you did something at the same time :) [00:20:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:11] Voidwalker, heh [00:20:14] Nice! :) [00:20:23] RhinosF1, ^ I think that calls for a ping [00:20:33] Well, at the very least you're back, for now (I think?) [00:21:11] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.93, 3.63, 3.09 [00:21:29] I never left? I'm just less active than you've become accustomed to. I'm still trying to be quite active on weekends and, when I can, on weeknights :) [00:21:53] I'm guessing this has to do with a new job or something? [00:22:08] !log void@mw11:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=afterlifewiki --logwiki=metawiki R4356th Redmin [00:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:22:16] PROBLEM - www.opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.opendatascot.org could not be found [00:22:17] PROBLEM - wiki.digitaldesignhq.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.digitaldesignhq.com could not be found [00:22:18] PROBLEM - speleo.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for speleo.wiki could not be found [00:22:20] PROBLEM - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wildcard.bots.miraheze.wiki could not be found [00:22:22] PROBLEM - lakehub.ga - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lakehub.ga could not be found [00:22:23] PROBLEM - bots1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for bots1.miraheze.wiki could not be found [00:22:28] thanks, Voidwalker :) [00:22:29] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tep.wiki could not be found [00:22:30] PROBLEM - trollpasta.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for trollpasta.com could not be found [00:22:34] did the script run that time? [00:22:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stardewmodding.wiki could not be found [00:22:35] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [00:22:37] PROBLEM - wiki.villagecollaborative.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.villagecollaborative.net could not be found [00:22:38] PROBLEM - tools1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tools1.miraheze.wiki could not be found [00:22:39] PROBLEM - spiral.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spiral.wiki could not be found [00:22:40] PROBLEM - wikiru.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiru.wildterra2.com could not be found [00:22:41] darkmatterman450, exactly that :) [00:22:52] I wouldn't be logging if it didn't :) [00:23:10] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.14, 3.31, 3.03 [00:23:13] oh, heh [00:23:20] I thought you had to log even if it errored [00:23:54] @dmehus Well, figures you'd confirm that. I've seen the update on your FaceBook page that confirmed it. [00:27:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.80, 23.76, 22.91 [00:29:03] RECOVERY - www.opendatascot.org - reverse DNS on sslhost is OK: SSL OK - www.opendatascot.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:05] RECOVERY - wiki.digitaldesignhq.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitaldesignhq.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:29:07] RECOVERY - speleo.wiki - reverse DNS on sslhost is OK: SSL OK - speleo.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:07] RECOVERY - bots1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - bots1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:11] RECOVERY - lakehub.ga - reverse DNS on sslhost is OK: SSL OK - lakehub.ga reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:12] RECOVERY - wildcard.bots.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - wildcard.bots.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:17] RECOVERY - stardewmodding.wiki - reverse DNS on sslhost is OK: SSL OK - stardewmodding.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:18] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:29:19] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: SSL OK - madgenderscience.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:20] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:29:23] RECOVERY - wikiru.wildterra2.com - reverse DNS on sslhost is OK: SSL OK - wikiru.wildterra2.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:29:28] RECOVERY - spiral.wiki - reverse DNS on sslhost is OK: SSL OK - spiral.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:30] RECOVERY - trollpasta.com - reverse DNS on sslhost is OK: SSL OK - trollpasta.com reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:29:31] RECOVERY - tools1.miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - tools1.miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [00:32:15] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:36:04] found problem, cp13 is randomly blocking ns1 in firewall [00:39:22] Voidwalker, oh wow [00:39:35] that makes sense in as much as that's the likely cause [00:39:44] why cp13 was doing that, I'm not sure heh [00:45:10] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.65, 17.75, 20.01 [00:55:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 21.16, 20.80 [00:57:51] !log [rhinos@jobrunner3] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [00:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:06:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.19, 17.87, 19.67 [01:10:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.44, 6.35, 7.85 [01:12:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.78, 5.92, 7.69 [01:20:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.95, 5.39, 6.66 [01:26:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.15, 5.82, 6.67 [01:34:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.31, 7.97, 7.20 [01:35:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.40, 8.62, 7.28 [01:37:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 7.58, 7.07 [01:38:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.08, 7.70, 7.30 [01:43:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.30, 6.69 [01:46:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 6.12, 6.73 [01:51:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.84, 7.20, 6.79 [01:53:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.17, 7.54, 6.96 [01:54:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.57, 3.20 [01:54:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 6.92, 6.89 [01:56:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.28, 3.15 [01:56:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.58, 6.48, 6.74 [01:56:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.52, 7.49, 7.08 [02:00:44] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.90, 6.30, 6.70 [02:06:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.86, 7.03, 6.92 [02:08:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.03, 6.65, 6.79 [02:22:23] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.37, 1.72, 1.27 [02:24:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.76, 1.44, 1.23 [02:34:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 6.93, 6.18 [02:35:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.58, 3.12 [02:35:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 6.88, 6.57 [02:36:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.83, 6.86, 6.25 [02:37:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.47, 3.10, 2.99 [02:37:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.72, 6.48, 6.46 [02:38:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.49, 6.79, 6.30 [02:42:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.86, 6.97, 6.45 [02:44:44] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.01, 6.68, 6.42 [02:57:45] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNqa [02:57:46] [02miraheze/RemovePII] 07Universal-Omega 03695ad36 - Update RemovePIIJob.php [02:57:48] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [02:58:43] miraheze/RemovePII - Universal-Omega the build passed. [03:00:16] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNm0 [03:00:18] [02miraheze/RemovePII] 07Universal-Omega 0320eb1b2 - Update RemovePIIJob.php [03:00:19] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:01:12] miraheze/RemovePII - Universal-Omega the build passed. [03:01:56] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNmA [03:01:58] [02miraheze/RemovePII] 07Universal-Omega 03962a8cc - Update RemovePIIJob.php [03:01:59] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:03:00] miraheze/RemovePII - Universal-Omega the build passed. [03:06:15] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNOE [03:06:16] [02miraheze/RemovePII] 07Universal-Omega 0349dac67 - Update RemovePIIJob.php [03:06:17] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:06:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.22, 6.59 [03:06:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.52, 6.81, 6.46 [03:07:21] miraheze/RemovePII - Universal-Omega the build passed. [03:08:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.13, 6.55, 6.43 [03:08:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.33, 6.33 [03:12:14] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNsi [03:12:15] [02miraheze/RemovePII] 07Universal-Omega 03f48e7a3 - Update RemovePIIJob.php [03:12:17] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:13:09] miraheze/RemovePII - Universal-Omega the build passed. [03:20:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.26, 6.69, 6.39 [03:23:00] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNc9 [03:23:02] [02miraheze/RemovePII] 07Universal-Omega 03a2eac2f - Update RemovePIIJob.php [03:23:03] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:24:00] miraheze/RemovePII - Universal-Omega the build passed. [03:26:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.64, 6.33, 6.44 [03:28:19] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBNWa [03:28:20] [02miraheze/RemovePII] 07Universal-Omega 037133c1d - Update RemovePIIJob.php [03:28:22] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:29:17] miraheze/RemovePII - Universal-Omega the build passed. [03:32:44] [02RemovePII] 07Universal-Omega commented on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JBNly [03:35:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.65, 7.00, 6.66 [03:35:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.64, 7.05, 6.59 [03:37:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.81, 6.48, 6.51 [03:37:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.42, 6.02, 6.27 [03:39:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.26, 21.64, 18.12 [03:41:00] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.25, 19.26, 17.68 [03:41:36] [02RemovePII] 07Universal-Omega closed pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [03:41:37] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+3/-0/±6] 13https://git.io/JBNBw [03:41:39] [02miraheze/RemovePII] 07Universal-Omega 037ac312d - Create SpecialRemovePII (T7286: phase two and phase three) (#3) [03:41:40] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:41:42] [02RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/Jsfpj [03:42:31] miraheze/RemovePII - Universal-Omega the build passed. [03:46:51] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:47:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JBN0e [03:47:01] [02miraheze/mediawiki] 07Universal-Omega 0388c80b6 - Update RemovePII [03:49:53] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki && sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki on mw* / jbr* / test3 [03:50:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.14, 19.61, 17.68 [03:52:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.63, 6.76, 6.35 [03:54:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.17, 6.48, 6.29 [03:56:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.49, 22.51, 19.27 [03:58:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.43, 20.65, 18.98 [04:00:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.48, 17.17, 17.91 [04:11:33] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:00] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:10] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:23] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:34:09] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [04:35:31] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:01] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:30] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:35:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JBNbu [05:35:47] [02miraheze/mw-config] 07Universal-Omega 032fd06bf - T7732: Set wgMinervaTalkAtTop for isvwiki [05:35:48] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [05:35:51] [02mw-config] 07Universal-Omega opened pull request 03#4034: T7732: Set wgMinervaTalkAtTop for isvwiki - 13https://git.io/JBNba [05:36:47] miraheze/mw-config - Universal-Omega the build passed. [05:37:08] [02mw-config] 07Reception123 closed pull request 03#4034: T7732: Set wgMinervaTalkAtTop for isvwiki - 13https://git.io/JBNba [05:37:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBNbx [05:37:11] [02miraheze/mw-config] 07Universal-Omega 03c732a80 - T7732: Set wgMinervaTalkAtTop for isvwiki (#4034) [05:37:13] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [05:37:15] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 [05:37:44] [02mw-config] 07Reception123 commented on pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBNNU [05:37:45] [02mw-config] 07Reception123 closed pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [05:37:57] miraheze/mw-config - Reception123 the build passed. [05:40:47] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JBNNy [05:40:49] [02miraheze/dns] 07Reception123 03bd9a21a - add wiki-asterix.cf zone [05:57:57] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [06:01:15] Reception123: ^ [06:02:41] that's weird [06:02:46] !log [@test3] starting deploy of {'config': True} to skip [06:02:47] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [06:02:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:02:57] uh did you do that Universal_Omega ^ [06:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:05] (the test3 thing I mean) [06:03:21] also not sure what ns1 was all about because puppet ran fine for me [06:03:25] Reception123: Nope. [06:03:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:04:42] Reception123: It didn't say a username in log, just @test3 so I'd assume puppet did that with RhinosF1s MediaWiki Deploy logging. [06:05:11] It was logged at the exact time puppet ran. [06:05:37] Also there is composer cache file access denied stuff on test3 again I noticed that eariler today. [06:05:43] ah, well I hope it doesn't keep logging that every 30 mins [06:05:58] where? is it for an extension? [06:06:07] OAUTH. [06:06:20] ok, will fix [06:06:32] that's from the time I had to update composer and it only let me do it with sudo [06:06:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.31, 6.32, 5.20 [06:06:50] And yeah I don't think it'll keep logging. Also Reception123, it's been fixed twice before but it keeps reappearing. [06:08:09] hmm that's weird [06:08:21] Reception123: Void completely deleted the directory and RhinosF1 recloned it today and it worked for about 10 minutes, and now it happened again. [06:08:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.04, 6.07, 5.23 [06:08:56] hmm [06:09:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.15, 7.47, 5.88 [06:12:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.12, 7.75, 6.16 [06:12:56] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JBAvb [06:12:57] [02miraheze/ssl] 07Reception123 03337d529 - add wiki-asterix.cf cert [06:14:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.42, 7.19, 6.15 [06:15:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 7.51, 6.44 [06:16:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 6.56, 6.05 [06:19:07] [02MirahezeMagic] 07Reception123 closed pull request 03#266: Update ManageWiki database type when renaming or deleting wikis - 13https://git.io/JnKfd [06:19:08] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBAJy [06:19:10] [02miraheze/MirahezeMagic] 07Universal-Omega 0367c687f - Update ManageWiki database type when renaming or deleting wikis (#266) [06:19:25] [02MirahezeMagic] 07Reception123 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [06:19:27] [02miraheze/MirahezeMagic] 07Reception123 deleted branch 03Universal-Omega-patch-1 [06:19:58] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JBAJA [06:20:00] [02miraheze/mediawiki] 07Reception123 03d18a83f - Update MirahezeMagic [06:20:03] miraheze/MirahezeMagic - Reception123 the build passed. [06:20:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.28, 7.42, 6.52 [06:23:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.86, 6.93 [06:25:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.54, 6.98, 6.72 [06:26:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.33, 6.41, 6.40 [06:27:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.94, 6.67, 6.63 [06:30:08] [02CreateWiki] 07Universal-Omega edited pull request 03#233: Add bundling support to Echo notifications - 13https://git.io/JBSa3 [06:31:11] [02CreateWiki] 07Universal-Omega commented on pull request 03#233: Add bundling support to Echo notifications - 13https://git.io/JBAIo [06:34:53] SRE, ```Error 503 Backend fetch failed, forwarded for REDACTED, 127.0.0.1 [06:34:53] (Varnish XID 457951657) via cp12.miraheze.org at Tue, 03 Aug 2021 06:34:16 GMT.``` [06:35:30] Where dmehus? [06:35:49] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.20, 21.96, 18.47 [06:36:28] Meta loads fine for me so not sure where at. [06:37:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.25, 7.02, 6.69 [06:39:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.25, 6.83 [06:40:31] was metawiki [06:40:38] but seems to have been transitory [06:40:40] temporary [06:40:44] so should be okay [06:40:59] unsure if it's worth looking in the logs though [06:41:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.68, 22.82, 20.32 [06:42:13] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:42:52] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:43:38] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:43:42] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.89, 18.87, 19.20 [06:45:10] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.72, 6.56, 6.68 [06:45:15] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.33, 6.76, 6.76 [06:51:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.01, 7.21, 6.85 [06:53:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.78, 6.70, 6.72 [07:00:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBAnz [07:00:16] [02miraheze/services] 07MirahezeSSLBot 035d25ec0 - BOT: Updating services config for wikis [07:05:00] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:05:23] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:05:44] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:26:32] !log renamed testrenamewiki to testrename1wiki [07:26:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:57:15] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBAor [07:57:17] [02miraheze/MirahezeMagic] 07Universal-Omega 030181d91 - Actually commit changes to the database [07:57:18] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [07:57:20] [02MirahezeMagic] 07Universal-Omega opened pull request 03#276: Actually commit changes to the database - 13https://git.io/JBAoo [07:57:43] Reception123: will hopefully fix it. [07:58:11] miraheze/MirahezeMagic - Universal-Omega the build passed. [08:04:35] PROBLEM - patternarchive.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patternarchive.online could not be found [08:04:36] PROBLEM - wiki.nowchess.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.nowchess.org could not be found [08:04:36] PROBLEM - savagepedia.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for savagepedia.wiki could not be found [08:04:39] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [08:04:39] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [08:04:42] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.wiki could not be found [08:04:43] PROBLEM - animatedfeet.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for animatedfeet.miraheze.org could not be found [08:04:57] PROBLEM - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.marinebiodiversitymatrix.org could not be found [08:05:05] Oh go away [08:11:19] RECOVERY - patternarchive.online - reverse DNS on sslhost is OK: SSL OK - patternarchive.online reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [08:11:19] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: SSL OK - wiki.nowchess.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:11:27] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: SSL OK - www.mcpk.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:11:30] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - miraheze.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [08:11:32] RECOVERY - savagepedia.wiki - reverse DNS on sslhost is OK: SSL OK - savagepedia.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [08:11:36] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: SSL OK - vitriol.spoon.army reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:11:36] RECOVERY - animatedfeet.miraheze.org - reverse DNS on sslhost is OK: SSL OK - animatedfeet.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [08:11:38] RECOVERY - www.marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: SSL OK - www.marinebiodiversitymatrix.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [08:22:16] !log reset w on test3 [08:22:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:53] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.009 second response time [08:26:19] me [08:33:38] !log [rhinos@test3] starting deploy of {'world': True} to skip [08:33:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:57] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS [08:34:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:34:15] !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True} to skip [08:34:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:54] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20023 bytes in 0.839 second response time [08:37:32] [02MirahezeMagic] 07Reception123 closed pull request 03#276: Actually commit changes to the database - 13https://git.io/JBAoo [08:37:33] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBAHH [08:37:35] [02miraheze/MirahezeMagic] 07Universal-Omega 0355ba389 - Actually commit changes to the database (#276) [08:38:16] !log [rhinos@test3] finished deploy of {'config': True, 'world': True, 'l10n': True} to skip - FAIL: [0, 256, 0, 0] [08:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:38:45] miraheze/MirahezeMagic - Reception123 the build passed. [08:38:58] !log [rhinos@test3] starting deploy of {'world': True} to skip [08:39:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:39:14] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS [08:39:16] !log [rhinos@test3] starting deploy of {'world': True} to skip [08:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:39:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:39:28] !log [rhinos@test3] finished deploy of {'world': True} to skip - FAIL: [2] [08:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:40:22] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JBAQy [08:40:23] [02miraheze/mediawiki] 07Reception123 0356657ec - Update MirahezeMagic [08:42:33] RhinosF1: let me know when you're done with test3 (no rush) [08:42:47] Reception123: [08:42:52] magic is pulled [08:42:57] i'll sync in a sec [08:43:07] ok, thanks. test3 is HTTP 500 [08:43:14] though I'm in no rush so take your time if you still have things to test [08:43:17] yeah that's OAuth [08:43:27] trying to get my head round what it's doing [08:43:50] something is messing with the perms and setting cache files as read only [08:43:53] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.041 second response time [08:44:25] RhinosF1: I should probably mention that when I did composer update it only worked when I used sudo (without www-data) [08:44:39] I tried not to but it didn't want to work until I did [08:44:40] Reception123: never do that [08:44:48] you need to file a task and panic [08:44:55] that only makes things worse [08:45:08] (also please do it in staging not mediawiki) [08:45:12] I thought it could easily be changed with chown, but I did realise it wasn't the best idea [08:46:09] !log [rhinos@test3] starting deploy of {'world': True} to skip [08:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:46:25]