[00:01:34] 10WikimediaDebug, 06MediaWiki-Platform-Team, 06Traffic: X-Wikimedia-Debug cookie not routed correctly in Kubernetes on POST requests - https://phabricator.wikimedia.org/T397439#11052608 (10Tgr) 05Open→03Resolved Thanks! [00:57:13] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11052679 (10AntiCompositeNumber) Yup, it's working now. Thanks! [01:44:49] 10Beta-Cluster-Infrastructure, 10Acme-chief, 07Beta-Cluster-reproducible, 13Patch-For-Review: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11052715 (10BCornwall) 05Open→03In progress p:... [06:00:16] 10Phabricator, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11052828 (10Joe) Please note, this solution is temporary: bots working from clouds will break repeatedly if they're not properly identified with... [07:31:41] 10GitLab (Infrastructure), 06collaboration-services: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971 (10Jelto) 03NEW [09:16:52] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Gitlab switchover (gitlab2002 → gitlab1004) - https://phabricator.wikimedia.org/T400252#11053168 (10Jelto) >>! In T400252#11052877, @gerritbot wrote: > Change #1174976 had a related patch set uploaded (by Jelto; author: Jelto): > %%%[op... [09:28:13] 10Phabricator: Herald Rule: Remove WikiLove project from Growth Team rule H26 - https://phabricator.wikimedia.org/T400982 (10Michael) 03NEW [10:14:07] hashar: do you need a build of the new python images today, or can it wait for the weekly rebuild on sunday? [10:14:29] (basically asking if I need to trigger a manual build, or if I can just merge and forget about it) [10:14:35] re: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1174506 [10:15:19] claime: I personally don't need it, I only commented cause we previously had the same issue with pip/break system packages :) [10:15:29] Right [10:15:35] looks like it is for an image intended for abstractwiki? [10:15:38] Well I guess I'll merge and wait for dancy to be around [10:15:43] No, that's an unrelated change [10:16:02] The abstractwiki is just an in-flight fix for docker-pkg to work properly [10:17:20] I guess he had the issue when reusing the image in gitlab or similar [10:17:41] anyway, it probably does not hurt to run docker-pkg now and let him know [10:18:07] ack, will do [10:27:28] {{done}} [10:58:57] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥): Folks no longer able to merge gerrit dashboard changes - https://phabricator.wikimedia.org/T400829#11053368 (10hashar) Indeed I have changed the project owner to tightened the rights a little bit. The {nav Label Code-Review} permissions is granted to {... [11:00:05] (03PS1) 10Hashar: Review access change [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 [11:04:34] (03PS2) 10Hashar: Allow wmf members to edit dashboards [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 (https://phabricator.wikimedia.org/T400829) [11:04:36] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Herald Rule: Remove WikiLove project from Growth Team rule H26 - https://phabricator.wikimedia.org/T400982#11053412 (10Aklapper) 05Open→03Resolved p:05Triage→03Low a:03Aklapper Done. I also removed the team project tag from [25 open Wikilove ta... [11:05:27] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Folks no longer able to merge gerrit dashboard changes - https://phabricator.wikimedia.org/T400829#11053429 (10hashar) a:03hashar [11:21:22] (03CR) 10Umherirrender: "ACL looks good (from what I understand this repo is used for), but needs a Project Owner to merge" [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 (https://phabricator.wikimedia.org/T400829) (owner: 10Hashar) [12:10:00] 10Phabricator, 10Release-Engineering-Team (Doing 😎): Herald Rule: Remove WikiLove project from Growth Team rule H26 - https://phabricator.wikimedia.org/T400982#11053590 (10Michael) >>! In T400982#11053414, @Aklapper wrote: > Done. I also removed the team project tag from [25 open Wikilove tasks](https://ph... [12:23:51] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#11053617 (10Aklapper) [12:40:50] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053657 (10... [12:48:33] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053686 (10... [13:04:57] 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11053743 (10Jelto) [13:05:43] 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11053744 (10Jelto) [13:15:51] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#11053762 (10Aklapper) [13:19:30] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053765 (10... [13:20:59] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#11053766 (10Aklapper) [13:21:23] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053767 (10Lucas_We... [13:22:32] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053780 (10Lucas_We... [13:27:07] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11053800 (10Umherirr... [13:48:14] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Rename of a repository is not replicated - https://phabricator.wikimedia.org/T398401#11053871 (10hashar) After having added the Gerrit daemons host key to the known hosts (https://gerrit.wikimedia.org/r/1175114), I went to try a rename again: ` $ ssh -p... [13:56:53] 10Beta-Cluster-Infrastructure, 06MW-Interfaces-Team, 10WMF-JobQueue, 10Event-Platform: eqiad.mediawiki.job.sendCampaignEmail (and possibly others) seemingly aren't enqueued in beta - https://phabricator.wikimedia.org/T401002 (10Daimona) 03NEW [14:04:26] hashar: I forgot to CC you earlier, but I’d love to have your input on T400918 at some point, there’s some very strange Git behavior going on in CI there :) [14:04:27] T400918: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918 [14:04:36] (doesn’t have to be today though, if you’re busy with that repository rename or other stuff ^^) [14:21:39] Lucas_WMDE: I ll check [14:22:27] ahghaze [14:22:28] hmm [14:22:36] I dont' know anything about the doctrine thing though [14:24:02] the doctrine thing doesn’t matter at this point, it’s a more general issue about how Zuul puts together the working tree that ends up being tested [14:24:20] CI was testing different code than what was actually merged [14:24:28] fun [14:25:25] at least you have managed to retrieve the tree from the Zuul merger [14:27:38] yeah I had a good time in there :D [14:27:54] then I don't understand what the problem was [14:28:32] the git log shows Zuul has merged refs/changes/08/1174508/5 which had two changes: [14:28:33] * bd29e25951 Temporary disable doctrine tests for doctrine update [14:28:33] * 145631afaf Reenable doctrine tests for doctrine update [14:28:42] which I guess meant the tests are updated [14:30:50] also `git show` is misleading when it comes to displaying merge commits ;] [14:32:13] but in the tree that Zuul tested, the PHP changes from “Reenable doctrine tests” were not included [14:32:17] so the tests were not actually reenabled [14:32:27] and thus we didn’t notice that the SQL files were actually wrong [14:32:30] * b67e7a07a9 Merge commit 'refs/changes/08/1174508/5' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/Wikibase into HEAD [14:32:30] |\ [14:32:30] | * 145631afaf Reenable doctrine tests for doctrine update [14:32:31] ? [14:32:32] we only noticed when every subsequent build started failing [14:32:33] it was? [14:33:07] though the branch already had "Temporary disable doctrine tests for doctrine update" [14:33:14] pff [14:33:15] yes, exactly, that’s what confused git [14:33:50] so Git did the merge in a way that did not take over the PHP changes from “Reenable doctrine tests” [14:34:45] what was the expected results? For the tests to be reenabled isn't it? [14:35:04] yes [14:35:46] they were reenabled when the change was actually merged on the master branch, but not during the CI run [14:36:31] because Gerrit rebased the change while Zuul tried to merge it, and the merge got confused by the previous rebase-on-merge of “Temporary disable doctrine tests…” [14:36:45] (the merge in Zuul got confused, that is) [14:36:59] yeah I am looking at the tree at https://phabricator.wikimedia.org/T400918#11053499 [14:37:55] so the change under test is refs/changes/08/1174508/5 which enables the doctrine tests [14:38:23] its parent is the change "Temporary disable doctrine tests" https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1174431/2 [14:38:49] and somehow that got merged before? [14:39:29] yes, that intentionally got merged some time earlier [14:40:01] so when Zuul went to assemble the tree for CI, there were two versions of “Temporary disable doctrine tests” – the rebase-merged one on the master branch, and the “original” one as the direct parent commit of the change under test [14:40:38] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 07Developer Productivity, 07Upstream: Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps - https://phabricator.wikimedia.org/T288117#11054045 (10Novem_Linguae) I've filed a... [14:40:44] hmm [14:41:01] which I believe led Git to conclude that in the merge commit, the PHP files should be in the “Temporary disable” state, rather than the “Reenable” state [14:41:08] and thus the tests stayed disabled [14:42:36] well [14:42:49] I don't know the detials about how git merge operates [14:43:49] also the patch to reenable the test was processed after the parent got merged [14:44:25] but Zuul is not smart enough to get rid of it [14:45:10] I think if we’re going to stick with the rebase mode, then Zuul would also need to assemble the trees to test using rebase rather than merge [14:45:26] then it should end up with the same trees again [14:45:33] (probably much easier said than done…) [14:47:49] 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11054064 (10Novem_Linguae) 05Open→03Resolved a:03Novem_Linguae Marking as resolved. Thanks! [15:02:58] Lucas_WMDE: rebase does the same afaik :) [15:03:04] but yeah who knows [15:03:32] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11054094 (10hashar)... [15:04:46] hashar: but rebase would’ve noticed that the “Temporary disable” commit was already present [15:04:57] and then would’ve applied “Reenable” on top of that, i.e. actually reenable [15:04:58] I think [15:06:41] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11054110 (10Lucas_We... [15:17:43] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11054132 (10matmarex) This still happens oc... [15:17:46] 10Phabricator, 06collaboration-services, 06SRE, 06Traffic: traffic from Discord and Slack unfurler service is blocked by phabricator.wikimedia.org - https://phabricator.wikimedia.org/T400540#11054134 (10Pppery) a:05Novem_Linguae→03None [15:30:50] Lucas_WMDE: well I don't enough about how git merge operates :( [15:38:23] Yippee, build fixed! [15:38:24] Project mediawiki-core-phpmetrics build #465: 09FIXED in 4 min 22 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/465/ [15:43:37] 10Phabricator: phabricator start page should show "my created tasks" - https://phabricator.wikimedia.org/T112363#11054201 (10Aklapper) "Authored Tasks" will become available on your user profile in the menu on the left once we have resolved T393840. [15:43:40] 10Phabricator (Upstream), 07Upstream: Get my created tasks, get my created tasks or commented by me, get the tasks of a project - https://phabricator.wikimedia.org/T150527#11054204 (10Aklapper) > Get my created tasks. "Authored Tasks" will become available on your user profile in the menu on the left once... [17:47:51] 06Project-Admins, 10Q1 FY2025/26 July 1st - September 30th: Consider archiving `Q1 FY2025/26 July 1st - September 30th` - https://phabricator.wikimedia.org/T401018 (10A_smart_kitten) 03NEW [18:37:26] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Offboard Noarave from WMF systems - https://phabricator.wikimedia.org/T399953#11054679 (10CDobbins) [19:46:42] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Offboard Noarave from WMF systems - https://phabricator.wikimedia.org/T399953#11054816 (10CDobbins) [19:46:55] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Offboard Noarave from WMF systems - https://phabricator.wikimedia.org/T399953#11054817 (10CDobbins) 05Open→03Resolved p:05Triage→03Medium a:05joanna_borun→03CDobbins [20:40:03] maintenance-disconnect-full-disks build 724402 integration-agent-docker-1049 (/: 26%, /srv: 96%, /var/lib/docker: 38%): OFFLINE due to disk space [20:45:03] maintenance-disconnect-full-disks build 724403 integration-agent-docker-1049 (/: 26%, /srv: 76%, /var/lib/docker: 37%): RECOVERY disk space OK [21:00:43] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for aaronshaw - https://phabricator.wikimedia.org/T400869#11054927 (10thcipriani) 05Open→03Resolved a:03thcipriani Looks like [[https://wikitech.wikimedia.org/wiki/Tool:Gitlab-account-approval|gitlab-account... [21:02:56] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SaintWolf - https://phabricator.wikimedia.org/T399836#11054936 (10thcipriani) hi @SaintWolf I can't find any history for accounts on gerrit or phab. What do you plan to work on on GitLab? [21:05:23] 10GitLab (Support), 10Release-Engineering-Team (Doing 😎), 07Essential-Work: GitLab Private Repository Request for: research/npov-workstream-research - https://phabricator.wikimedia.org/T399696#11054940 (10thcipriani) 05Open→03Resolved a:03thcipriani Thanks for confirming @Pablo, closing this out! [22:07:47] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11055027 (10hashar)... [22:12:39] (03CR) 10Hashar: "Indeed, I'll talk with @tcipriani@wikimedia.org about it." [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 (https://phabricator.wikimedia.org/T400829) (owner: 10Hashar) [22:12:51] (03PS3) 10Hashar: Allow wmf members to edit dashboards [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 (https://phabricator.wikimedia.org/T400829) [22:16:48] (03CR) 10Thcipriani: [V:03+2 C:03+2] "Looks like it'll let people edit dashboards again." [wikimedia] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1175067 (https://phabricator.wikimedia.org/T400829) (owner: 10Hashar) [23:10:15] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for aaronshaw - https://phabricator.wikimedia.org/T400869#11055065 (10thcipriani) [23:11:18] 10Phabricator (2025-07-29), 10Release-Engineering-Team (Doing 😎), 06collaboration-services, 07Essential-Work: Deploy Phabricator/Phorge 2025-07-29 - https://phabricator.wikimedia.org/T400718#11055066 (10thcipriani) [23:31:08] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails to @wikimedia.org addresses - https://phabricator.wikimedia.org/T212327#11055070 (10bd808)