[00:17:52] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Krinkle) [00:18:03] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Krinkle) >>! In T285232#7355412, @Joe wrote: > As it stands, my new configuration would mean that we... [00:25:24] 10Release-Engineering-Team (Radar), 10Analytics-Radar, 10WikimediaDebug, 10observability, and 4 others: Create a separate 'mwdebug' cluster - https://phabricator.wikimedia.org/T262202 (10Krinkle) This is still unresolved and I believe would similarly be beneficial in mw-on-k8s as well. If anything, when mw... [01:14:20] 10Release-Engineering-Team (Yak Shaving 🐃🪒): pipeline-promote: Add patch author as reviewer to promote patch - https://phabricator.wikimedia.org/T281392 (10Legoktm) I noticed this as part of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/720980 - while well intentioned, I'm not sure it makes sen... [01:34:39] 10Release-Engineering-Team (Yak Shaving 🐃🪒): pipeline-promote: Add patch author as reviewer to promote patch - https://phabricator.wikimedia.org/T281392 (10jeena) Thanks for your feedback @Legoktm. We can definitely re-think how this should work. [02:20:05] Project beta-update-databases-eqiad build #53463: 04FAILURE in 4.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53463/ [03:20:05] Project beta-update-databases-eqiad build #53464: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53464/ [04:20:09] Project beta-update-databases-eqiad build #53465: 04STILL FAILING in 7.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53465/ [05:20:05] Project beta-update-databases-eqiad build #53466: 04STILL FAILING in 4.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53466/ [06:16:57] 10Phabricator: give visibility for "in progress" tasks on a work board - https://phabricator.wikimedia.org/T291593 (10mmodell) p:05Triage→03Medium [06:18:13] 10Phabricator: give visibility for "in progress" tasks on a work board - https://phabricator.wikimedia.org/T291593 (10mmodell) {F34653325} [06:20:05] Project beta-update-databases-eqiad build #53467: 04STILL FAILING in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53467/ [06:23:23] Error 1091: Can't DROP INDEX `afl_filter_timestamp`; [06:23:25] oh joyce [06:24:10] 10Phabricator: Searching for "gerrit" in global Phabricator search leads to upstream request timeout (too many results?) - https://phabricator.wikimedia.org/T258803 (10mmodell) p:05Medium→03Low [06:25:02] 10Phabricator: Searching for "gerrit" in global Phabricator search leads to upstream request timeout (too many results?) - https://phabricator.wikimedia.org/T258803 (10mmodell) a:05mmodell→03None [06:27:23] hashar still? It should have been fixed at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/723662 [06:28:48] DannyS712: that patch does not touch the related index? [06:30:05] 10Beta-Cluster-Infrastructure, 10AbuseFilter: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10hashar) [06:30:55] oh, so the issue was that that sql file was broken, so I assume the first part (dropping index afl_filter_timestamp) ran successfully but the failure in the second part (afl_filter_id) means that its still being applied? [06:31:17] DannyS712: yes https://phabricator.wikimedia.org/T291725 ;) [06:31:50] maybe it needs something like DROP IF EXISTS [06:32:01] I have no clue really, I haven't written sql patch in ages [06:32:22] anyway I just filed the task, i am having breakfast with kids [06:32:23] or split in two patches, the first dropping the index and the second the fields [06:32:58] uhh [06:33:36] the table on aawiki.beta has an index named filter_timestamp (note missing afl_ prefix) [06:36:32] majavah does it have wiki_timestamp or afl_wiki_timestamp index? The patch to rename that should also cover filter_timestamp [06:38:45] I am escaping to complete the coffee & croissant morning dance. [06:39:07] DannyS712: https://phabricator.wikimedia.org/P17327 [06:42:08] it seems like https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/593906/20/db_patches/mysql/patch-rename-indexes.sql isn't being completely applied... [06:49:09] the first change had a syntax issue [06:49:17] so my guess is the DROP index worked [06:49:35] then the ALTER TABLE failed. It got fixed by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AbuseFilter/+/723662/1/db_patches/mysql/patch-remove-afl_filter.sql [06:50:03] and on the second run of update.php the updater triggers the patch run since the afl_filter still exists [06:50:06] it applies the patch [06:50:17] but fails cause the INDEX already had been dropped previously [06:50:39] so tentatively, the index should be recreated on the databases, then update.php run again [06:50:59] or adjust the mediawiki logic to split the patch, but that might be overkill [06:51:11] * hashar I am off [06:58:05] Yeah, thats what I was thinking, but does it make sense to split when one depends on the other? Maybe just live-hack comment out that one line of the update patch? [07:20:04] Project beta-update-databases-eqiad build #53468: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53468/ [08:20:05] Project beta-update-databases-eqiad build #53469: 04STILL FAILING in 3.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53469/ [08:48:34] https://phabricator.wikimedia.org/T291731 [08:49:00] "Core tests failing - "PHP Deprecated" output instead of PHPUnit deprecation" [08:49:00] All patches seem to be failing [09:17:29] DannyS712: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/723643 was first to fail [09:17:41] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/723636 passed [09:18:12] Something between 07:09 & 09:01 BST [09:18:22] RhinosF1 I know, thats my patch (both of them) - I sent an empty patch to demonstrate that it wasn't my code [09:18:47] DannyS712: just trying to work out when it broke [09:20:06] Project beta-update-databases-eqiad build #53470: 04STILL FAILING in 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53470/ [09:26:12] Project beta-update-databases-eqiad build #53471: 04STILL FAILING in 4.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53471/ [09:28:59] 10Beta-Cluster-Infrastructure, 10AbuseFilter: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10hashar) Looks like we can get paste the issue by recreating the index which let the fixed sql patch to be applied. Example fo... [09:36:46] Yippee, build fixed! [09:36:47] Project beta-update-databases-eqiad build #53472: 09FIXED in 1 min 43 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53472/ [09:36:57] DannyS712: majavah: I have fixed the beta cluster update.php script that choked on an AbuseFilter index. I simply recreated the index on all database so that the .sql patch properly applies [09:37:54] 10Beta-Cluster-Infrastructure, 10AbuseFilter: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10hashar) 05Open→03Resolved a:03hashar I ran for all labs wiki with: ` for wiki in $(grep -v '#' dblists/all-labs.dblist)... [09:42:41] yippee :) thanks [09:43:11] any chance you can take a look at the core tests failing? I have to go but other people will probably start waking up and complaining at some point [09:47:12] to be clear, I didn't mean complaining at you, just that it seems there are fewer people online right now and so it hasn't been noticed as much I think [09:48:44] the build failure for core with deprecation [09:48:55] that is due to PHPUnit 8.5.21 [09:49:01] I am writing a reply on the task [09:58:15] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/723668 phpunit: explicitly fail on deprecation [NEW] [09:58:25] which I guess I will just self approve if it passes all fine on our CI [09:58:48] ...well, I sent a patch too :) edit conflict I guess [09:59:08] but shouldn't we also update tests/phpunit/suite.xml? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/723688/2/tests/phpunit/suite.xml [09:59:09] OH [09:59:17] DannyS712: yeah and you update the other phpunit config file [09:59:21] I will just abandon mine ;) [10:00:02] sorry, I should have realized that if you had tracked it down you were probably sending a patch for it too [10:00:22] DannyS712: no worries! [10:00:25] that is a common issue hehe [10:00:40] I like to refer to the upstream code https://github.com/sebastianbergmann/phpunit/commit/fac02620f6b38ae54d47fe840e0095e68226a56c [10:00:59] which would eventually ping them on github when our patch get merged [10:01:03] but that is not important [10:02:15] DannyS712: and might want to put convertDeprecationsToExceptions above the other convert* settings to keep them alphabetically sorted [10:02:32] oh, I had them ordered by severity [10:02:59] errors > notices > warning [10:03:26] it is alphabetically sorted :] I have fall in the same pit [10:04:01] anyway we will see when CI pass. I am going to finish lunch preparation, watch the kids. Will catch up when it is done and just +2 it [10:04:17] so in ~ 1H30 [10:04:53] okay, switching to alphabetical, updating commit message, and adding you as co-author in the commit message. Enjoy lunch [10:05:38] DannyS712: you are awesome. [10:06:16] thanks [10:06:50] DannyS712: I realized I can just +2 it without waiting for the `test` pipeline to report [10:07:30] also, lol another edit conflict - we both set it ready for review? [10:08:08] yeah but it is in the gate-and-submit [10:08:28] and we run 13 jobs for a mediawiki/core change doooh [10:09:01] happy week-end and thanks for the task filing/patch etc! [10:09:06] * hashar slices onions and potatoes [10:11:58] 10Beta-Cluster-Infrastructure, 10AbuseFilter: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10Reedy) Just to point out, it was not broken by my fix in "Make mysql/patch-remove-afl_filter.sql valid SQL" (made at nearly 4... [10:13:01] Reedy: 4am patches are risky :) that was an easy fix thanksfully! [10:13:13] The patch was fine [10:13:16] Beta cluster wasn't :P [10:13:29] :p [10:13:36] I had tested it locally (after running it originally an dit also breaking), and it had worked [10:14:59] yaeh no worries :) [10:15:26] * hashar slices more onions [10:19:29] DannyS712: looks like we will need the phpunit patch for 1.35, 1.36 and 1.37 [10:19:51] they all use `^8.5` [11:37:48] I merged the backports [11:38:12] * majavah complains about breaking changes in patch releases [11:39:53] Don't try and complain upstream [11:40:04] Literally no point [17:44:06] uh-oh, how was https://phabricator.wikimedia.org/p/Mp3cloud/ created? it's not linked to ldap or mw.o [18:34:16] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [18:36:22] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 3 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [19:19:38] majavah: thank you for the phpunit merges ;) [20:33:31] 10Project-Admins: Create #Croatian-Sites tag - https://phabricator.wikimedia.org/T291751 (10Ivi104) [23:10:57] majavah: perhaps created through mw.org and then "unlinked" in Phab settings [23:11:10] I'm not going to try it with my own account, but it seems at least that such unlink button exists [23:11:28] whether it is possible to use it for only secondary/additional ones or also for the last/only one, I don't know. [23:41:22] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [23:43:18] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 6 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator