[00:02:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.54, 3.07 [00:03:00] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:31] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.74, 9.47, 6.51 [00:04:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 2.93, 2.90 [00:05:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.20, 8.18, 6.03 [00:06:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 15.08, 11.36, 7.61 [00:07:19] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.26, 21.98, 17.88 [00:08:22] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.98, 3.51, 2.51 [00:09:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 21.45, 18.16 [00:10:22] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.69, 3.55, 2.65 [00:10:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.58, 3.18 [00:11:21] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.31, 24.39, 19.58 [00:12:22] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.04, 3.31, 2.66 [00:13:11] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.70, 7.89, 6.89 [00:14:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.79, 7.34 [00:14:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.61, 3.28 [00:16:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.38, 7.52, 7.37 [00:18:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.18, 3.24, 3.19 [00:19:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.16, 7.90, 7.14 [00:20:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.98, 8.84, 7.77 [00:20:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 11.01, 8.59, 7.76 [00:22:32] PROBLEM - db11 Puppet on db11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [00:22:51] PROBLEM - db12 Puppet on db12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [00:24:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.74, 7.25, 7.41 [00:25:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 7.74, 7.43 [00:25:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.08, 23.46, 23.08 [00:26:16] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.18, 7.10, 7.47 [00:30:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.36, 6.06, 6.76 [00:31:07] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:31:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.58, 5.84, 6.67 [00:31:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.53, 3.35 [00:32:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.86, 5.49, 6.67 [00:35:45] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.24, 7.55, 7.16 [00:36:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.61, 6.90, 6.90 [00:36:33] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.14, 8.27, 7.60 [00:37:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 4.06, 3.64 [00:37:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.23, 7.10 [00:38:07] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.18, 6.48, 6.76 [00:38:31] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.38, 7.23, 7.30 [00:38:49] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.75, 2.11, 1.52 [00:39:43] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.32, 5.81, 6.59 [00:40:51] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.14, 1.74, 1.45 [00:41:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.25, 3.71, 3.59 [00:41:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.88, 18.17, 20.28 [00:42:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.75, 1.46, 1.39 [00:44:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.64, 6.29, 6.78 [00:45:14] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.88, 3.65 [00:47:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.56, 3.56 [00:50:25] RECOVERY - db11 Puppet on db11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:50:43] RECOVERY - db12 Puppet on db12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:58:46] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.04, 3.36 [01:05:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 6.69, 5.87 [01:07:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.73, 5.92, 5.69 [01:10:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 2.94, 3.07 [01:12:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.20, 2.87, 3.02 [01:21:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.47, 3.28 [01:27:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 3.17, 3.25 [01:31:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.12, 4.23, 3.66 [01:41:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.88, 3.85 [01:48:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.96, 3.39 [01:52:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.47, 3.54 [01:56:44] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.09, 3.37 [02:20:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.56, 3.39 [02:24:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.39, 3.39 [02:32:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.72, 3.51 [02:34:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.90, 3.59 [02:36:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.65, 3.54 [02:38:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.99, 4.11, 3.72 [02:40:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.97, 3.72 [02:50:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.60, 3.59 [02:52:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.52, 3.43, 3.55 [02:54:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.65, 3.60 [02:56:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.41, 3.52 [02:58:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.89, 2.84, 3.29 [04:48:07] PROBLEM - gp.ct777.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gp.ct777.cf' expires in 15 day(s) (Sat 11 Sep 2021 04:39:11 GMT +0000). [04:48:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEBG4 [04:48:32] [02miraheze/ssl] 07MirahezeSSLBot 0309bd495 - Bot: Update SSL cert for gp.ct777.cf [04:54:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.33, 2.95 [04:56:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.50, 2.96, 2.86 [05:08:43] RECOVERY - gp.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'gp.ct777.cf' will expire on Wed 24 Nov 2021 03:48:25 GMT +0000. [05:48:21] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Package[python-pip],File[/etc/init/nutcracker.override] [07:09:23] PROBLEM - zh.gyaanipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zh.gyaanipedia.com' expires in 15 day(s) (Sat 11 Sep 2021 07:06:34 GMT +0000). [07:09:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEB1V [07:09:47] [02miraheze/ssl] 07MirahezeSSLBot 0339f0d75 - Bot: Update SSL cert for zh.gyaanipedia.com [07:18:12] PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fbpml.org' expires in 15 day(s) (Sat 11 Sep 2021 07:10:15 GMT +0000). [07:24:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEByD [07:24:12] [02miraheze/ssl] 07MirahezeSSLBot 03bcefb49 - Bot: Update SSL cert for wiki.fbpml.org [07:38:28] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Wed 24 Nov 2021 06:24:05 GMT +0000. [07:42:52] RECOVERY - zh.gyaanipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zh.gyaanipedia.com' will expire on Wed 24 Nov 2021 06:09:39 GMT +0000. [07:59:09] Reception123: morning [08:07:27] RhinosF1: morning. Was waiting for you as there's a problem: seems like the python-pip package doesn't want to work anymore [08:07:30] it only wants python3-pip [08:11:21] Reception123: yes that will be true [08:11:27] What requires it [08:14:01] It's 3 years old [08:14:48] We should probably just remove & hope [08:15:43] (whats the worst that could happen) [08:16:38] Tn: something that's been marked as please update me for a long time dies [08:17:08] At this point python2 is a sec risk [08:17:10] RhinosF1: I mean couldn't we just use python3-pip instead of python-pip on all mw* servers (even the ones that don't have bullseye yet) [08:17:16] ehhh, at least it'll get updated pretty darn quick afterwards [08:17:23] Reception123: python means python2 [08:17:45] So I would just remove from puppet and leave mw* as is until we work out exactly what needs it [08:17:57] I imagined that because python1 would be super old [08:18:13] but yeah, I guess it's best to comment out python-pip until we figure it out [08:18:15] Yeah 2 & 3 are not drop in replacements [08:18:33] Commenting it out only affects new installs [08:18:36] (2 -> 3 is *normally* fairly trivial if y'all did need to run around fixing things... famous last words I know..) [08:18:37] So I don't see issue [08:19:35] tn: https://github.com/miraheze/puppet/search?q=%23%21%2Fusr%2Fbin%2Fpython&type= says nothing [08:19:36] [url] Search · #!/usr/bin/python · GitHub | github.com [08:19:40] yeah, that's why commenting out is easiest as we'll just have it fix test3 and then when John or Paladox are around I'll ask them if they have any idea why we need it [08:20:34] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JEBx1 [08:20:35] [02miraheze/puppet] 07Reception123 03e846410 - temporarily remove python-pip [08:20:56] I imagine a leftover from https://phabricator.miraheze.org/T3647 [08:20:57] [url] ⚓ T3647 Convert all our python2 scripts to python3 | phabricator.miraheze.org [08:21:06] perhaps [08:29:29] Reception123: is that running better [08:30:49] RhinosF1: unfortunately not, next error: Could not set 'present' on ensure: No such file or directory - A directory component in /etc/init/nutcracker.override2021[id].lock does not exist or is a dangling symbolic link [08:31:15] Reception123: does /etc/init exist? [08:31:22] /etc/puppetlabs/puppet/environments/production/modules/nutcracker/manifests/init.pp [08:31:35] nope just init.d [08:32:06] Reception123: just mkdir /etc/init then as sudo [08:32:16] just did that, let's see if it works [08:32:46] ergh seriously, new error [08:32:53] 🔥 [08:32:57] mergeMessageFilesList.php returned 1 instead of [0] [08:33:31] Reception123: that's expected [08:33:43] where does that come from [08:33:50] RhinosF1: because w is empty [08:33:52] just has LS.php [08:34:14] Reception123: yes again expected [08:34:22] so I guess I have to clone manually then? [08:34:30] though in mediawiki-staging? [08:34:32] Reception123: no it's staging, it needs a sync [08:34:38] ah [08:34:40] using the deploy tool [08:34:49] But what is triggering that script run [08:35:03] It's just in the puppet run [08:35:21] !log [@test3] starting deploy of {'config': True} to skip [08:35:22] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:54] !log [@test3] starting deploy of {'folders': 'w/extensions/Echo,w/skins/Vector'} to skip [08:35:55] !log [@test3] finished deploy of {'folders': 'w/extensions/Echo,w/skins/Vector'} to skip - FAIL: [256, 256] in 0s [08:35:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:36:08] !log [@test3] starting deploy of {'folders': 'w'} to skip [08:36:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:37:06] !log [@test3] finished deploy of {'folders': 'w'} to skip - SUCCESS in 57s [08:37:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:37:40] Reception123: it's an exec call [08:38:17] Reception123: run --world --config --l10n --gitinfo [08:38:26] After running composer install in staging [08:38:33] ok [08:38:36] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 761 bytes in 0.624 second response time [08:40:39] https://phabricator.miraheze.org/T7712#158562 re the exec [08:40:39] [url] ⚓ T7712 New MediaWiki deployment system | phabricator.miraheze.org [08:40:52] ack [08:41:32] !log [@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip [08:41:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:42:49] Reception123: did you composer install first [08:42:59] yeah [08:43:12] !log composer install done first for record [08:43:13] though after I still have some issues to resolve with puppet [08:43:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:43:26] Reception123: we will fix them [08:43:39] In about 5 minutes when sync has run [08:44:37] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20036 bytes in 0.684 second response time [08:45:07] \o/ [08:45:20] No it's still a fatal [08:45:51] !log [@test3] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip - FAIL: [0, 256, 0, 0, 0] in 259s [08:45:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:46:23] Oh great a fail [08:46:47] That's world [08:47:01] Reception123: try delete the /srv/mediawiki/w folder [08:47:18] Unless you know what the error is [08:47:22] Actually no [08:47:28] Run --world by itself [08:47:58] didn't see any error [08:48:03] but ok will do deploy-mediawiki --world then [08:48:06] Yes [08:48:11] !log [@test3] starting deploy of {'world': True} to skip [08:48:14] It's definately deployed [08:48:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:48:31] test3 is still a fatal [08:48:46] well puppet still hasn't finished doing it's thing either [08:48:48] !log [@test3] finished deploy of {'world': True} to skip - FAIL: [2] in 36s [08:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:48:55] Reception123: what's it say? [08:49:02] ergh there's some stupid composer permission error [08:49:10] inside OAuth/.cache [08:49:17] Reception123: puppet has a built in hack to unmess that [08:49:23] A simple puppet run should fix it [08:49:34] We should check why it's down though [08:49:37] ok, then I need to fix the user ID mess in puppet and get it to run successfully [08:49:52] [13acd40244634434a9083eff] 2021-08-26 08:49:38: Fatal exception of type "Error" [08:49:54] Yeah [08:50:46] yay, applied catalog! [08:50:57] now let's run puppet without --server and if that works that's not bad [08:51:12] let me actually see if I can log in normally with my account first [08:51:16] puppet working is fun! [08:51:37] yup, everything works fine now I can leave the Promox console [08:52:19] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:53:04] huh, running composer now gives some not fun errors [08:53:04] Warning from https://repo.packagist.org: Support for Composer 1 is deprecated an d some packages will not be available. You should upgrade to Composer 2. See htt ps://blog.packagist.com/deprecating-composer-1-support/ [08:53:05] [url] Packagist | repo.packagist.org [08:53:10] https://www.irccloud.com/pastebin/JSDKAzyw/ [08:53:25] JohnLewis: puppet (nutcracker) needs to not trust /etc/init exists [08:53:28] but the exception on test3 is completely unrelated [08:53:29] Class 'Wikibase\DataModel\Services\Lookup\DisabledEntityTypesEntityLookup' not found [08:54:27] RhinosF1: hm? [08:54:57] JohnLewis: the nutcracker module assumes /etc/init exists wrongly [08:55:09] Reception123: ye if no composer no wiki [08:55:29] well I'm not sure what to do with that runtime exception [08:56:54] Reception123: nothing yet [08:57:22] hm? [08:58:49] We shouldn’t be using nutcracker anymore [08:59:14] JohnLewis: test3 installs it [08:59:50] Also has any work been done to ensure puppet repo is Debian 11 compliant? [08:59:59] Reception123: do 'sudo pecl install wddx' [09:00:06] This sounds like a Deb 10-11 difference [09:00:16] JohnLewis: php7.4 issue [09:00:18] No releases available for package "pecl.php.net/wddx" [09:00:27] This extension has been moved to the » PECL repository and is no longer bundled with PHP as of PHP 7.4.0 [09:01:20] Reception123: makes zero sense [09:01:53] Reception123: ok scrap that [09:02:19] We need a flag I think to just ignore it [09:03:30] JohnLewis: do we have a way to test for OS [09:05:02] It comes from https://github.com/miraheze/puppet/blob/master/modules/php/manifests/php_fpm.pp#L170 but is dead [09:05:03] [url] puppet/php_fpm.pp at master · miraheze/puppet · GitHub | github.com [09:05:56] $facts[‘os’] it’s all in that I believe [09:06:49] JohnLewis: so we can use that to tell its bullseye and then ignore wddx [09:07:08] Should be able to [09:07:16] Though what uses wddx? [09:07:28] JohnLewis: what I just linked [09:07:36] It's dead though [09:08:45] And pecl refuses to install it back [09:08:51] No, I mean what uses wddx [09:09:10] Something currently broken [09:10:01] Composer was refusing to start because of it [09:10:37] it's been in puppet like Python-pip since it was tidied up 3 years ago at least [09:11:07] Composer was failing because it can’t find the so file that PHP is told exists [09:11:32] Last mention of wddx I can find is https://github.com/wikimedia/mediawiki/commit/240f4c15d5a62fe3868d55390f43a026723a0daa [09:11:33] [url] Merge "API: Remove WDDX and dump formats" · wikimedia/mediawiki@240f4c1 · GitHub | github.com [09:13:35] So it was rightly killed and never removed from puppet [09:14:41] Which should have been picked up when MediaWiki was upgraded [09:15:30] We know how badly 1.36 went [09:16:51] [02puppet] 07RhinosF1 opened pull request 03#1890: Fpm: kill wddx, forgot with 1.36 - 13https://git.io/JERqD [09:17:19] ^ can be merged but we still need to clean up after it [09:19:45] what kind of clean up? [09:20:23] Removing everything php::extension installs for wddx [09:22:49] ah [09:23:07] It's not an auto cleanup thing [09:50:50] hmm so where do we start then [10:02:42] Find the appropiate ini files and delete them id say [10:02:49] The module should give some hint [10:02:56] I doubt it actually installed [10:04:44] let me look [10:09:06] [02puppet] 07Reception123 closed pull request 03#1890: Fpm: kill wddx, forgot with 1.36 - 13https://git.io/JERqD [10:09:08] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JER4k [10:09:09] [02miraheze/puppet] 07RhinosF1 03f2d04e6 - Fpm: kill wddx, forgot with 1.36 (#1890) [10:13:08] interesting, we've already got a new error (didn't figure out the cleanup just yet) [10:13:08] Could not delete /srv/mediawiki-staging/w/vendor/justinrainbow/json-schema/phpunit.xml.dist: [10:16:19] yeah don't run composer until you've cleaned up php [10:16:30] but can you delete it [10:17:59] Look for something in /etch/php7.4/mods-available [10:18:33] And at https://github.com/miraheze/puppet/blob/master/modules/php/manifests/extension.pp#L42 [10:18:33] [url] puppet/extension.pp at master · miraheze/puppet · GitHub | github.com [10:18:39] ./etc/ even [10:20:35] RhinosF1: delete the phpunit file you mean? [10:20:52] Reception123: anything for wddx [10:21:03] and there's no php7.4 dir just php [10:21:14] and then 7.4 inside that (just FYI) [10:21:28] and yes found the wddx.ini [10:22:14] Oh ok but if it talks about wddx, delete it [10:22:39] There should be some folders with a conf.d inside [10:22:42] Check there too [10:23:06] let me see [10:23:41] JohnLewis: VE is sometimes stupid and needs a cookie reset or something [10:24:34] That should be reported upstream then if the extension just sometimes decides not to work [10:24:56] JohnLewis: that's a very good point [10:25:10] Also is there anywhere ive missed for blowing up references to wddx [10:26:03] Could not delete /srv/mediawiki-staging/w/vendor/justinrainbow/json-schema/README.md: - hmm, don't get what composer's problem with a README.md is [10:27:30] Check file perms [10:27:45] A reclone wouldn't be a stupid idea [10:29:56] RhinosF1: yeah, I think I'll try that [10:30:53] !log rm -rf /w and sudo -u www-data git clone https://github.com/miraheze/mediawiki --recursive on test3 [10:30:54] [url] GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. | github.com [10:30:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:49] Reception123: let puppet do it [10:33:14] well that's what I did the first time and somehow it got us some strange permissions [10:33:38] Reception123: because the whole setup was in a stupid state [10:33:46] It should be less stupid now [10:33:59] Hopefully [10:35:09] JohnLewis: is flaggedrevs_stats or _stats2 existing anywhere [10:35:55] RhinosF1: unsure, dealing with that private task that's been unactioned for 24hrs. Why? [10:36:22] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki core] [10:36:56] JohnLewis: because it's potential junk half deleted [10:37:46] it's definately not updated [10:39:19] RhinosF1: doesn't exist on any c4 wikis [10:49:15] * Reception123 will be back later but after reclone same sort of issues, can't seem to remove anything [11:08:19] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [11:13:09] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JERoB [11:13:11] [02miraheze/landing] 07translatewiki 031c5896a - Localisation updates from https://translatewiki.net. [11:13:12] [url] Main page - translatewiki.net | translatewiki.net [11:13:12] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JERoR [11:13:14] [02miraheze/CreateWiki] 07translatewiki 03123ed63 - Localisation updates from https://translatewiki.net. [11:13:14] [url] Main page - translatewiki.net | translatewiki.net [11:13:15] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JERo0 [11:13:17] [02miraheze/ManageWiki] 07translatewiki 034ed0092 - Localisation updates from https://translatewiki.net. [11:13:17] [url] Main page - translatewiki.net | translatewiki.net [11:14:08] miraheze/ManageWiki - translatewiki the build passed. [11:14:12] miraheze/landing - translatewiki the build passed. [11:14:19] miraheze/CreateWiki - translatewiki the build passed. [12:33:37] PROBLEM - linkwiki.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'linkwiki.cf' expires in 15 day(s) (Sat 11 Sep 2021 12:25:45 GMT +0000). [12:38:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JERFm [12:38:31] [02miraheze/ssl] 07MirahezeSSLBot 032f17ae4 - Bot: Update SSL cert for linkwiki.cf [12:41:47] Reception123: need what? [12:45:23] paladox: python2 [12:45:33] We think pip was left installed [12:45:41] But for no reason [12:45:44] In puppet [12:45:56] PROBLEM - wiki.slimelab.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.slimelab.net' expires in 15 day(s) (Sat 11 Sep 2021 12:42:01 GMT +0000). [12:48:25] Oh [12:49:32] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JERN4 [12:49:33] [02miraheze/ssl] 07MirahezeSSLBot 03261bc38 - Bot: Update SSL cert for wiki.slimelab.net [12:49:35] https://phabricator.miraheze.org/T7145#158571 [12:49:36] [url] ⚓ T7145 Upgrade MediaWiki cluster to Debian Bullseye | phabricator.miraheze.org [12:49:43] RhinosF1: what do you mean by this? [12:52:00] paladox: Reception123 started updating to bullseye, ran puppet and packages.pp for mediawiki wanted python-pip [12:52:05] We assumed it was not needed [12:52:29] Oh [12:53:01] test3 php is still spectacularly broke though [12:53:09] Composer hates Reception123 [12:53:23] I got a driving lesson though in a minute [12:53:27] RhinosF1: I'm back, removing w again and trying to reclone with puppet [12:53:49] what does composer install --no-dev say? [12:53:58] in /srv/mediawiki/w [12:54:07] paladox: not mediawiki [12:54:12] Please do stuff in staging now [12:54:28] mediawiki-staging then use the deploy tool [12:54:44] what deploy tool? [12:54:52] paladox: deploy-mediawiki [12:54:58] oh [12:55:01] The new deployment system from right before you joined [12:55:14] That just needs me to fix the last 2 PRs before it's prod ready [12:55:24] i'm at a hotel, what does composer say when you try and install [13:04:27] ohh we can install https://packages.debian.org/bullseye/ploticus rather then doing it manually. [13:04:28] [url] Debian -- Details of package ploticus in bullseye | packages.debian.org [13:05:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JERps [13:05:57] [02miraheze/puppet] 07paladox 030b62a2f - mediawiki: Install ploticus from debian rather than manually [13:05:58] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [13:06:00] [02puppet] 07paladox opened pull request 03#1891: mediawiki: Install ploticus from debian rather than manually - 13https://git.io/JERpZ [13:07:45] RECOVERY - linkwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'linkwiki.cf' will expire on Wed 24 Nov 2021 11:38:24 GMT +0000. [13:13:52] RECOVERY - wiki.slimelab.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.slimelab.net' will expire on Wed 24 Nov 2021 11:49:25 GMT +0000. [13:15:02] Hmm, i guess we need to add 'main' e.g. deb http://ftp.de.debian.org/debian stretch main [13:15:03] [url] Index of /debian | ftp.de.debian.org [13:18:17] [02MirahezeMagic] 07amire80 opened pull request 03#284: Remove double spaces from message files - 13https://git.io/JERjM [13:29:05] https://www.irccloud.com/pastebin/myv0taZj/ [13:29:09] ^ paladox this is what it says [13:29:18] and if I manually delete that file it just goes to another and so on [13:37:21] rm -rf /srv/mediawiki/w/vendor && sudo-u www-data php composer.phar install —no-dev [13:37:26] Reception123: ^ [13:37:41] paladox: ok, let me try [13:40:34] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1287 bytes in 0.007 second response time [13:40:41] https://www.irccloud.com/pastebin/ybXEOEZn/ [13:40:43] ^ paladox [13:41:04] Huh [13:41:13] I’ve never had that issue [13:41:20] yeah, it's very strange [13:41:27] as if it doesn't have permissions to do anything [13:44:03] Reception123: composer require wikimedia/composer-merge-plugin [13:44:13] i guess we'll need to add that package [13:44:23] to composer.json but ^ should do for now [13:44:40] I'd probably guess if we added that it would just go to another file but I can try [13:44:51] https://github.com/wikimedia/mediawiki/blob/master/composer.json#L53 [13:44:52] oh [13:44:52] [url] mediawiki/composer.json at master · wikimedia/mediawiki · GitHub | github.com [13:45:01] hmm yeah [13:50:43] i guess just reset /w (git reset --hard origin/REL1_36 && php composer.phar install --no-dev && chown -R www-data:www-data vendor) [14:12:54] ok, I'll try that thanks! [14:14:44] https://www.irccloud.com/pastebin/Y7XozMcu/ [14:14:53] ^ paladox I'm quite sure if I ran it as root it would work but you're not supposed to that [14:20:46] Reception123: it sounds to me like /srv/mediawiki-staging/w/vendor isn't writable by www-data currently, paladox command includes a bit to fix that but after composer has run, I would try running the permissions change first and then run composer [14:21:44] I tried chmod so far but let me reverse the change and see if that helps [14:22:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.64, 3.18 [14:23:53] https://www.irccloud.com/pastebin/zbjujCJU/ [14:23:57] it appears to work but test3 doesn't [14:24:05] * Reception123 will brb a bit later [14:24:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.36, 3.13 [14:27:09] https://phabricator.miraheze.org/T7867 [14:27:10] [url] ⚓ T7867 Consider updating composer to v2 | phabricator.miraheze.org [15:05:41] Reception123: if you've been doing it in staging then deploy-mediawiki --world [15:05:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.22, 3.02 [15:06:51] MacFan4000: https://phabricator.miraheze.org/T7867#158606 [15:06:51] [url] ⚓ T7867 Consider updating composer to v2 | phabricator.miraheze.org [15:09:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.13, 3.03 [15:09:50] RhinosF1: https://packages.debian.org/bullseye/composer [15:09:50] [url] Debian -- Details of package composer in bullseye | packages.debian.org [15:10:03] looks like there is a native v2 package in bullseye [15:10:25] MacFan4000: we don't use native composer for a forgotten reason [15:10:37]