[00:09:34] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10BDC-Implementation, 06collaboration-services: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#10794327 (10Dwisehaupt) @JMando Thanks, we'll need to add access to the iptab... [00:16:52] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10BDC-Implementation, 06collaboration-services, 10fundraising-tech-ops: All fundraising analytics host to access gitlab in addition to gerrit - https://phabricator.wikimedia.org/T393423 (10Dwisehaupt) 03NEW [00:17:13] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10BDC-Implementation, 06collaboration-services: Move wikimedia/fundraising/analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#10794357 (10Dwisehaupt) Created {T393423} to track the firewall changes. [02:47:56] (03CR) 10Jakob: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1141863 (owner: 10Jakob) [03:57:26] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T392171#10794564 (10egardner) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1139942 | Update... [06:53:29] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10BDC-Implementation, 06collaboration-services, 10fundraising-tech-ops: All fundraising analytics host to access gitlab in addition to gerrit - https://phabricator.wikimedia.org/T393423#10794650 (10Jelto) During the [datacente... [07:19:23] 10GitLab (Project Migration), 10BDC-Implementation, 06collaboration-services, 10fundraising-tech-ops: All fundraising analytics host to access gitlab in addition to gerrit - https://phabricator.wikimedia.org/T393423#10794684 (10Aklapper) [07:26:50] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Bmartinezcalvo - https://phabricator.wikimedia.org/T393385#10794689 (10Aklapper) @bmartinezcalvo: The header on gitlab.wikimedia.org says: > Only if you are WMF staff: Create your GitLab account, have "Wmf" group... [07:30:45] (03PS2) 10Jakob: Add missing OpenSearch plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142413 (https://phabricator.wikimedia.org/T386691) [07:42:54] 10Phabricator: Custom task form for #MW-Interfaces-Team - https://phabricator.wikimedia.org/T392598#10794726 (10Aklapper) We can create a custom form for your team if you share how to pre-fill/pre-set the current fields. :) See https://phabricator.wikimedia.org/maniphest/task/edit/form/17/?projects=Phabricator p... [08:25:39] 10Scap (SpiderPig πŸ•ΈοΈ): scap spiderpig-otp should not print a password that expires in 0 seconds - https://phabricator.wikimedia.org/T392815#10794806 (10Lucas_Werkmeister_WMDE) Amazing, thanks! [08:57:25] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10794870 (10Jdforrester-WMF) [09:04:43] (03approved) 10jforrester: Install php-gcmath [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/76 (owner: 10taavi) [09:04:52] (03update) 10jforrester: Install php-gcmath [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/76 (owner: 10taavi) [09:04:56] (03merge) 10jforrester: Install php-gcmath [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/76 (owner: 10taavi) [09:11:08] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10794896 (10A_smart_kitten) Should `REL1_43` in the task description be `REL1_44`? [09:15:26] 10MediaWiki-Releasing, 10MediaWiki-Docker: Remove docker-compose.yml from MediaWiki archive releases - https://phabricator.wikimedia.org/T393183#10794912 (10Douginamug) [09:17:22] 10MediaWiki-Releasing, 10MediaWiki-Docker: Remove docker-compose.yml from MediaWiki archive releases - https://phabricator.wikimedia.org/T393183#10794917 (10taavi) Files are excluded from the tarball via the `.gitattributes` file. [09:20:01] 10MediaWiki-Releasing, 10MediaWiki-Docker: docker-compose.yml in standard package refers to absent DEVELOPERS.md - https://phabricator.wikimedia.org/T393193#10794926 (10Douginamug) 05Openβ†’03Invalid Closing this. Task better understood and formulated at T393183 [09:28:42] hashar: bonjour! thanks for releasing the new quibble version and switching jobs to it! I unfortunately realized that I did a sloppy job checking that things are working as expected, and after an adventurous debugging session found out that those OpenSearch plugins dcausse tried to warn me about last week are indeed needed :/ [09:29:11] https://gerrit.wikimedia.org/r/c/integration/config/+/1142413 adds those plugins [09:29:27] Yeah I have been looking at that [09:29:32] but the good news is that now `--run api-testing` is passing locally [09:29:53] and I don't get why SRE went with thirdparty/opensearch1 AND component/opensearch13 [09:29:58] it is inconsistent [09:31:47] erm, there is always a chance that I'm just doing something silly, but I'm pretty sure that it couldn't find the plugins with just thirdparty/opensearch1 though [09:32:13] 10MediaWiki-Releasing, 10MediaWiki-Docker: Remove docker-compose.yml from MediaWiki archive releases - https://phabricator.wikimedia.org/T393183#10795003 (10Douginamug) a:03Douginamug Thanks @taavi I'll try and submit my first patch 🍿 [09:37:08] (03CR) 10Hashar: [C:03+2] Add missing OpenSearch plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142413 (https://phabricator.wikimedia.org/T386691) (owner: 10Jakob) [09:38:21] (03Merged) 10jenkins-bot: Add missing OpenSearch plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142413 (https://phabricator.wikimedia.org/T386691) (owner: 10Jakob) [09:40:55] (03PS1) 10Hashar: jjb: update Quibble opensearch job for plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142544 (https://phabricator.wikimedia.org/T386691) [09:41:37] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Bmartinezcalvo - https://phabricator.wikimedia.org/T393385#10795071 (10bmartinezcalvo) >>! El T393385#10794689, @Aklapper escribiΓ³: > @bmartinezcalvo: The header on gitlab.wikimedia.org says: >> Only if you are WM... [09:44:59] hashar: ah! there was one more odd thing that I've encountered while trying this out locally. I've found that somehow the `QUIBBLE_OPENSEARCH` env variable wasn't picked up by the two CirrusSearch maintenance scripts, which caused UpdateSearchIndexConfig to fail. does that ring a bell for you? [09:45:25] that would be a bug in Quibble itself [09:45:43] the quibble.mediawiki.maintenance functions would call subprocess without passing the whole environent [09:46:55] yes that is it [09:47:06] both commands set `env = {}` [09:47:11] add `MW_INSTALL_PATH` [09:47:22] then use rthat as the environment passed tu subprocess.Popen [09:47:47] ahh! [09:47:50] * jakob_WMDE tries [09:47:57] the `def update` has the "right" way to do it: [09:48:05] update_env = {} [09:48:05] update_env.update(os.environ) [09:48:05] if mwdir is not None: [09:48:05] update_env['MW_INSTALL_PATH'] = mwdir [09:48:16] def install does the same [09:48:27] I copy pasta'd the wrong function [09:48:53] (03CR) 10Hashar: [C:03+2] jjb: update Quibble opensearch job for plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142544 (https://phabricator.wikimedia.org/T386691) (owner: 10Hashar) [09:49:43] yeah that addSite() does not pass env [09:49:46] which might be a bug [09:50:19] so I guess we would later need to replace that `subprocess.Popen` to some intermediate function that uses the original env + MW_INSTALL_PATH and that is it [09:50:25] (03Merged) 10jenkins-bot: jjb: update Quibble opensearch job for plugins [integration/config] - 10https://gerrit.wikimedia.org/r/1142544 (https://phabricator.wikimedia.org/T386691) (owner: 10Hashar) [09:50:37] I also don't think we should keep using `communicate()`, that sounds old [09:50:58] at least I have switched the jobs to the image having plugins [09:55:39] jakob_WMDE: you'd also need opensearch and its plugins to be added to the Quibble bookworm images [09:55:43] I guess later on :) [09:56:04] I am off to prepare lunch [09:56:21] ok, enjoy! :) [09:57:51] hashar: so with the env fixed for the maintenance scripts in quibble and the jobs switched to the image that has the plugins, could we already set QUIBBLE_OPENSEARCH for the wikibase api-testing job? or is there something missing still? [10:16:42] 10MediaWiki-Releasing, 10MediaWiki-Docker: docker-compose.yml in standard package refers to absent DEVELOPERS.md - https://phabricator.wikimedia.org/T393193#10795134 (10Aklapper) β†’14Duplicate dup:03T393183 [10:16:43] 10MediaWiki-Releasing, 10MediaWiki-Docker: Remove docker-compose.yml from MediaWiki archive releases - https://phabricator.wikimedia.org/T393183#10795136 (10Aklapper) [10:23:39] (03merge) 10addshore: Refactor internal/mw and internal/mwdd [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/621 (owner: 10mstyles) [10:25:22] (03merge) 10addshore: txeh 1.5.5 doesn't exist, don't try to use (again) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/625 (owner: 10hoo) [10:27:17] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T393334#10795159 (10Aklapper) > from Developer Outreach I'm curious where to find out about that Developer Outreach team (but that's off-topic here). > What would you recommend as the best approach? Unfortunately I don't think I... [10:39:31] 10Phabricator, 03Wikimedia-Hackathon-2025: Phorge (Phabricator) Code Sprint 2025 - https://phabricator.wikimedia.org/T392411#10795232 (10Aklapper) a:03valerio.bozzolan [10:49:12] (03PS2) 10Jakob: Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 [11:02:06] Project ArticlePlaceholder-phpmetrics build #59: 04FAILURE in 6.1 sec: https://integration.wikimedia.org/ci/job/ArticlePlaceholder-phpmetrics/59/ [11:09:44] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T393334#10795291 (10Addshore) >>! In T393334#10795158, @Aklapper wrote: >> from Developer Outreach > I'm curious where to find out about that Developer Outreach team (but that's off-topic here). CC @Mohammed_Sadat_WMDE (So this... [11:26:50] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10795326 (10Jelto) I disabled object storage on `gitlab1003` again. The migration back included setting t... [11:28:20] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10795341 (10Jelto) [11:28:33] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 2 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10795345 (10Jelto) [11:30:23] Project mediawiki-core-doxygen build #10202: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/10202/ [11:40:52] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T393334#10795402 (10Aklapper) Ah, makes sense (and I guess I could have known myself, sorry!). [11:54:53] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10795510 (10Jdforrester-WMF) [12:15:41] jakob_WMDE: opensearch got added to bullseye/php74 image which is the one used by the job `integration-quibble-fullrun-opensearch` [12:16:14] bunch of php 7.4 jobs are still based on Buster [12:16:33] and the 8.1 ones are on Bullseye [12:18:44] the apitests job runs on php74 https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php74 [12:18:47] which uses the buster image [12:18:53] hashar: so if I want opensearch in mediawiki-quibble-apitests-vendor-php74 I'd have to add it to buster? [12:18:55] ya ok :) [12:19:09] which is obsolete / EOL :D [12:20:04] James as a series of patch to drop php7.4 ( https://gerrit.wikimedia.org/r/c/integration/config/+/1127083/ ) [12:21:45] which is T328921 [12:21:45] T328921: Drop PHP 7.4 and PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328921 [12:22:08] and I guess that is blocked on the remaining last bits of stuff running php7.4 in prod which is https://phabricator.wikimedia.org/T319432 [12:23:24] the apitests php7.4 job can surely be moved to a bullseye image though [12:24:42] I see. moving apitests to the bullseye image sounds lovely, but I'm happy with whatever you think makes most sense! [12:25:48] rust? [12:25:54] haha [12:31:26] Yippee, build fixed! [12:31:26] Project mediawiki-core-doxygen build #10203: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/10203/ [12:37:18] (03PS3) 10Hashar: Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:37:29] (03CR) 10Hashar: [C:03+2] Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:37:45] (03CR) 10Hashar: [C:03+2] "I have amended your change to add tests :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:38:56] (03CR) 10Jakob: "thank you!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:39:36] (03PS1) 10Hashar: release: Quibble 1.14.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142584 [12:41:44] (03PS2) 10Hashar: release: Quibble 1.14.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142584 [12:41:44] (03PS1) 10Hashar: release: Start 1.14.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142585 [12:42:03] jakob_WMDE: ^ :) [12:42:30] for Buster > Bullseye we would need James_F cause I have lost track of the OS/php upgrades :/ [12:42:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:42:42] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T393460 (10phaultfinder) 03NEW [12:42:47] hashar: yay, thanks! [12:42:58] (03CR) 10Hashar: [C:03+2] release: Quibble 1.14.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142584 (owner: 10Hashar) [12:43:09] (03CR) 10Hashar: [C:03+2] release: Start 1.14.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142585 (owner: 10Hashar) [12:44:09] hashar: ok, is the buster -> bullseye switch for the apitests what you'd suggest? I'm also fine with making a patch to add opensearch to the buster image [12:46:52] 10Phabricator: Need Access to Facts (Charts) - https://phabricator.wikimedia.org/T393297#10795754 (10DSantamaria) Hey, I should rephrase my question to: - Is it possible to create a new Chart / Facts? [12:47:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:49:35] 10Phabricator: Phabricator workboard import: Array for %Ls conversion is empty. Query: projectPHID IN (%Ls) - https://phabricator.wikimedia.org/T392168#10795772 (10Aklapper) > In the first step, entering the search term "chocolate" should match a sprint name which has "Chocolate cake", but no result is found I v... [12:51:01] jakob_WMDE: you would not have the opensearch plugins (they are only available for bullseye bookworm) and who knows what else might be broken [12:51:11] 10Phabricator: Need Access to Facts (Charts) - https://phabricator.wikimedia.org/T393297#10795787 (10Aklapper) Yes but only by writing/adding more PHP code. [12:51:13] ahh :/ [12:55:16] (03open) 10jnuche: patches: ensure a security task notified of a failure is open [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/790 (https://phabricator.wikimedia.org/T393361) [12:56:39] (03update) 10jnuche: patches: ensure a security task notified of a failure is open [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/790 (https://phabricator.wikimedia.org/T393361) [12:57:20] (03CR) 10CI reject: [V:04-1] Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:58:09] 00:02:39.028 cmdline: git fetch -v --tags --force -- origin [12:58:09] 00:02:39.028 stderr: 'error: RPC failed; curl 55 Failed sending data to the peer [12:58:10] fun [12:58:26] (03CR) 10Hashar: [C:03+2] Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [12:59:49] (03merge) 10jnuche: patches: ensure a security task notified of a failure is open [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/790 (https://phabricator.wikimedia.org/T393361) [12:59:49] (03Merged) 10jenkins-bot: Ensure env vars are present in maintenance scripts [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142556 (owner: 10Jakob) [13:00:04] ah that is the test pipeline which had failed tests [13:00:27] 10Phabricator: New Column Trigger: Assign unassigned task to... - https://phabricator.wikimedia.org/T393464 (10DSantamaria) 03NEW [13:00:59] (03open) 10jnuche: Release 4.161.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/791 [13:01:16] 10Phabricator: New Column Trigger: Assign unassigned task to... - https://phabricator.wikimedia.org/T393464#10795839 (10DSantamaria) [13:01:17] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream, 03Wikimedia-Hackathon-2025: New Column Trigger: When moving to a column, assign to the "current user" (user that is performing the move) - https://phabricator.wikimedia.org/T393268#10795838 (10DSantamaria) [13:01:27] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream, 03Wikimedia-Hackathon-2025: New Column Trigger: When moving to a column, assign to the "current user" (user that is performing the move) - https://phabricator.wikimedia.org/T393268#10795840 (10DSantamaria) Done! [13:01:56] 10Phabricator: Need Access to Facts (Charts) - https://phabricator.wikimedia.org/T393297#10795841 (10DSantamaria) Ok, not as trivial as I could imagine :( [13:02:16] (03merge) 10jnuche: Release 4.161.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/791 [13:05:18] 10Phabricator: Need Access to Facts (Charts) - https://phabricator.wikimedia.org/T393297#10795846 (10Aklapper) I'd be interesting though what you'd like to see regarding "Work distribution by tags" and "Visualize some queries". Not that I could quickly add such code at all but having specific use cases is always... [13:06:45] (03Merged) 10jenkins-bot: release: Quibble 1.14.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142584 (owner: 10Hashar) [13:08:34] (03Merged) 10jenkins-bot: release: Start 1.14.2 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1142585 (owner: 10Hashar) [13:10:54] 10Phabricator: Need Access to Facts (Charts) - https://phabricator.wikimedia.org/T393297#10795887 (10DSantamaria) **Work distribution by tags :** Similar to this chart: https://phabricator.wikimedia.org/fact/chart/IaTLVFKT1tnf/ But with Tasks tagged with some tags that you can choose. **Visualize some queries**... [13:33:44] 10Scap (SpiderPig πŸ•ΈοΈ): scap spiderpig-otp should not print a password that expires in 0 seconds - https://phabricator.wikimedia.org/T392815#10796040 (10Lucas_Werkmeister_WMDE) I just tried it out, it seems to work \o/ [13:34:27] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T393460#10796046 (10Jelto) [13:46:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:46:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T393470 (10phaultfinder) 03NEW [13:51:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:58:39] (03open) 10jnuche: housekeeping [repos/releng/scap3-dev] - 10https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/45 [13:59:30] (03merge) 10jnuche: housekeeping [repos/releng/scap3-dev] - 10https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests/45 [14:34:35] Yippee, build fixed! [14:34:35] Project ArticlePlaceholder-phpmetrics build #60: 09FIXED in 6.8 sec: https://integration.wikimedia.org/ci/job/ArticlePlaceholder-phpmetrics/60/ [14:38:09] 10Scap: SecurityPatchBot: Reopen security task if any of its patches is failing to apply - https://phabricator.wikimedia.org/T393361#10796317 (10jnuche) 05Openβ†’03Resolved a:03jnuche This is in prod now [14:40:22] hashar: We're waiting to drop buster (at your insistence) on the removal of PHP 7.4 support from MW CI, because prod 7.4 is still on Buster. [14:44:39] Hopefully SRE will complete the switch-off of 7.4 for the last remaining use case (maintenance scripts) in the next few days, and then we can do it! [15:03:16] James_F: yeah that is what I got (mwmaint being upgraded soon!) [15:03:55] I think what I was worried about is some mess up with libpcre / libicu or similar [15:06:53] !log Tag Quibble 1.4.1 @ 5247438621f802ba9878970b3b34b2d67cefa54c [15:06:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:09:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:12:45] 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481 (10bd808) 03NEW [15:14:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:25:03] maintenance-disconnect-full-disks build 699292 integration-agent-docker-1041 (/: 26%, /srv: 95%, /var/lib/docker: 40%): OFFLINE due to disk space [15:30:03] maintenance-disconnect-full-disks build 699293 integration-agent-docker-1041 (/: 26%, /srv: 41%, /var/lib/docker: 39%): RECOVERY disk space OK [15:48:58] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10796668 (10Jdforrester-WMF) [15:52:34] Not sure what's up with CI, I see stuff in gate-and-submit that's been in queue for 50 minutes and still no job running [15:52:44] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10796687 (10MSantos) 05Openβ†’03In progress p:05Triageβ†’03High [15:53:01] It seems overloaded but I don't get why it's testing stuff in the test pipeline [15:53:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:53:38] Also there's this job stuck in postbuild https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/90750/console [15:54:34] 10GitLab (Project Migration), 10BDC-Implementation, 06collaboration-services, 10fundraising-tech-ops: All fundraising analytics host to access gitlab in addition to gerrit - https://phabricator.wikimedia.org/T393423#10796699 (10Dwisehaupt) @Jelto Thanks for that info. Right now we have the restriction at t... [15:54:50] Also yeah gerrit seems to be struggling [15:54:53] hashar: Not an issue in practice due to the alignment between wmf builds of php74 and php81 binaries and dependencies, though? [15:55:37] Now the good news, beta is up and working at least! [15:56:56] :D [16:02:19] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485 (10Daimona) 03NEW [16:02:29] Filed ^ [16:03:28] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Wikidata, 13Patch-For-Review, 10Wikidata-Omega (The Board): Falling back to linear test runs appears to be broken (AbuseFilter, May 2025) - https://phabricator.wikimedia.org/T393266#10796750 (10Lucas_Werkmeister_WMDE) a:03Lucas_Werkm... [16:03:48] There's also this thing where our tests, especially browser tests, seem to become exponentially more fragile when CI is under load, which makes them fail much more often, which causes the gate-and-submit queue to reset, which causes more load to accumulate, etc. in a vicious circle [16:06:32] Oh wait, EVERYTHING is stuck in postbuild [16:06:48] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796765 (10Daimona) p:05Triageβ†’03Unbreak! [16:07:33] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796768 (10Daimona) On closer inspection, ALL running jobs are stuck in the postbuild phase, all the logs ending with something like: ` 17:35:46 Archiving artifacts 17:35:50 [PostBuildScri... [16:07:48] I'm trying to cancel https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/90750/consoleFull with no luck so far. [16:08:18] All jobs are stuck right there. Apparently verifying the npm cache or something [16:08:26] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796770 (10CDanis) It looks like this is also blocking test-prio CI for production deploy CI -- multiple `operations/puppet` patches have been queued for 20+ minutes. [16:08:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:09:47] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796778 (10Daimona) Checking integration-agent-docker-1050, which is where the above logs are from (https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/90750/consol... [16:11:14] `integration-agent-docker-1050` appears to be idle. [16:11:46] yep [16:11:50] some jobs are waiting on castor job which can't find an executor to run [16:12:23] https://integration.wikimedia.org/ci/computer/integration%2Dcastor05/ [16:12:34] it has a single executor [16:12:52] and somehow Jenkins is waiting for an executor to free up [16:12:58] OK I think something is borked [16:13:02] which would be a bug somewhere in the stack of Gearman/Jenkins scheduler etc [16:13:05] https://integration.wikimedia.org/ci/computer/integration-castor05/ it shows there's a job in progress [16:13:12] na it is pending [16:13:15] but if I click through, there's nothing [16:13:24] https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ [16:13:30] Ah that means pending, got it [16:13:31] that shows the queue [16:13:54] dropping them from the queue would cause the calling job to fail IIRC [16:14:11] let me try to disconnect and reconnect the agent [16:14:14] T353925 is related, except here it's 100% blocked and not just backlogged [16:14:14] T353925: gate-and-submit backlogged due to waiting for castor-save-workspace-cache - https://phabricator.wikimedia.org/T353925 [16:15:17] :( [16:15:32] I will restart Jenkins and clear the state [16:15:34] something is borked [16:16:16] !log restarting CI Jenkins due to a deadlock affecting castor-save-workspace which ends up blocking jobs # T353925 [16:16:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:17:37] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796822 (10Daimona) >>! In T353925#10796816, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng), href=https://sal.toolforge.org/log/ifpippYBffdvpiTrNTpW} [2025-05-... [16:17:48] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10796825 (10MSantos) I wonder if the script could support in the future the following call: ` ./branch.py REL1_44 --bundle ba... [16:18:34] it is not coming back :/ [16:18:58] ah no [16:18:59] here it is [16:19:15] https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ is processing again [16:19:36] Daimona: dancy: solved by restarting Jenkins. There is a deadlock somewhere between the Gearman plugin and the Jenkins executors :/ [16:19:47] Thanks! [16:20:02] Zuul retriggered the jobs [16:20:05] Looking forward to the random selenium failures coming back ;) [16:20:17] ahah [16:20:46] I never new "cancelled" was a status in Zuul. [16:21:12] I'd seen success/failure/queued/aborted, but I think cancelled is new to me. [16:21:31] https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php82-noselenium/32139/console <-- out of disk space? [16:21:33] Oh, no, now I think about it, I believe I've seen it also when restarting Zuul, it's just really rare. [16:21:42] 10Phabricator: New Column Trigger: Assign unassigned task to... - https://phabricator.wikimedia.org/T393464#10796840 (10Aklapper) [16:21:43] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream, 03Wikimedia-Hackathon-2025: New Column Trigger: When moving to a column, assign to the "current user" (user that is performing the move) - https://phabricator.wikimedia.org/T393268#10796839 (10Aklapper) [16:21:48] 10Phabricator: New Column Trigger: Assign unassigned task to... - https://phabricator.wikimedia.org/T393464#10796842 (10Aklapper) [16:21:50] 10Phabricator (Upstream), 06Release-Engineering-Team, 07Upstream, 03Wikimedia-Hackathon-2025: New Column Trigger: When moving to a column, assign to the "current user" (user that is performing the move) - https://phabricator.wikimedia.org/T393268#10796841 (10Aklapper) [16:21:51] I am off for an appointment! [16:22:39] 10Phabricator: New Column Trigger: Assign only unassigned tasks - https://phabricator.wikimedia.org/T393464#10796845 (10Aklapper) [16:23:08] I'm going to abort some jobs for stuff that's already failing [16:24:04] :-( [16:24:23] At least the REL1_44 cut job is now running. [16:25:52] It's catching up, slowly but surely [16:26:33] But then again, selenium is our slowest job and reports last, so we'll see how it goes for the gate-and-submit stuff. [16:29:10] Although TBH, now that success cache makes it more apparent, it's getting ridiculous. The first patch in line is already green for all jobs, except selenium, where we're just now starting to test AbuseFilter, which is obviously first in alphabetical order [16:33:46] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487 (10bd808) 03NEW [16:33:56] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10796896 (10bd808) p:05Triageβ†’03Medium [16:37:18] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10796929 (10ABreault-WMF) 05In progressβ†’03Resolved a:05Atienoβ†’03ABreault-WMF [16:38:07] 10Beta-Cluster-Infrastructure: Add allowlist to make poking holes in abuse_networks:blocked_nets:networks easier - https://phabricator.wikimedia.org/T393481#10796947 (10bd808) [16:38:07] 10Beta-Cluster-Infrastructure: High load on deployment-mediawiki14 and slow responses - https://phabricator.wikimedia.org/T392003#10796949 (10bd808) [16:38:08] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10796946 (10bd808) [16:38:09] 10Beta-Cluster-Infrastructure, 07Documentation: Improve documentation for Beta Cluster edge blocking - https://phabricator.wikimedia.org/T392643#10796948 (10bd808) [16:39:16] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10796963 (10bd808) [16:39:45] 10Continuous-Integration-Infrastructure: CI is overwhelmed, 2025-05-06 edition - https://phabricator.wikimedia.org/T393485#10796964 (10Daimona) 05Openβ†’03Resolved a:03hashar @hashar's restart fixed the deadlock and CI is now catching up. Having a single executor for castor jobs contributed to this outag... [16:39:55] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10796969 (10bd808) [16:41:30] 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10796974 (10bd808) [16:50:21] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10796991 (10A_smart_kitten) Probably a stupid question, but just to check, is it an issue that the `REL1_44` branch has a... [16:55:08] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10797009 (10Jdforrester-WMF) >>! In T390696#10796991, @A_smart_kitten wrote: > Probably a stupid question, but just to che... [16:58:54] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10797043 (10PatchDemoBot) Test wiki **created** on [[ https://pd.erebor.com | Patch demo ]] by usuario using patch(es) lin... [17:04:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:08:03] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10797089 (10PatchDemoBot) Test wiki on [[ https://pd.erebor.com | Patch demo ]] by usuario using patch(es) linked to this... [17:09:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:16:18] 10MediaWiki-Releasing, 10MW-1.44-notes (1.44.0-wmf.28; 2025-05-06), 05MW-1.44-release, 10MW-Interfaces-Team (MW-Sprint-8 (2025-04-23 to 2025-05-06)): Update CREDITS in MW master - https://phabricator.wikimedia.org/T390700#10797134 (10ABreault-WMF) 05Openβ†’03Resolved [17:17:30] 10MediaWiki-Releasing, 05MW-1.44-release: Add REL1_44 to on-wiki documentation as the development snapshot - https://phabricator.wikimedia.org/T390793#10797139 (10ABreault-WMF) 05Openβ†’03Resolved a:03ABreault-WMF https://www.mediawiki.org/w/index.php?title=Module%3AVersion&diff=7613508&oldid=7559814 [17:22:46] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎): Stand up Zuul 11 experiment environment in zuul3 cloud VPS project - https://phabricator.wikimedia.org/T391374#10797155 (10dduvall) 05In progressβ†’03Resolved [17:24:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:30:03] maintenance-disconnect-full-disks build 699309 integration-agent-docker-1050 (/: 25%, /srv: 97%, /var/lib/docker: 40%): OFFLINE due to disk space [17:31:49] ^gerrit still struggling. Is this known/expected or should I file a task? [17:35:03] maintenance-disconnect-full-disks build 699310 integration-agent-docker-1050 (/: 25%, /srv: 92%, /var/lib/docker: 40%): RECOVERY disk space OK [17:35:24] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.44-release: Add REL1_44 to ExtensionDistributor as the development snapshot - https://phabricator.wikimedia.org/T390794#10797205 (10MacFan4000) a:03MacFan4000 [17:35:35] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496 (10dduvall) 03NEW [17:39:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:45:17] Daimona: Please do file. [17:47:17] 10Gerrit: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498 (10Daimona) 03NEW [17:47:26] 10Gerrit: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498#10797286 (10Daimona) p:05Triageβ†’03Unbreak! [17:47:27] Done, thx [17:49:21] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10797297 (10ABreault-WMF) >>! In T390696#10797009, @Jdforrester-WMF wrote: >>>! In T390696#10796991, @A_smart_kitten wrote... [17:49:23] 10Gerrit: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498#10797298 (10Daimona) IRC logs for today, which include the various alerts that fired: https://wm-bot.wmcloud.org/browser/index.php?start=05%2F06%2F2025&end=05%2F06%2F2025&display=%23wikimedia-releng [17:54:55] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 13Patch-For-Review: Branch REL1_44 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T390696#10797339 (10Jdforrester-WMF) >>! In T390696#10797297, @ABreault-WMF wrote: >>>! In T390696#10797009, @Jdforrester-WMF wrot... [17:54:56] 10MediaWiki-Releasing, 05MW-1.44-release: Announce 1.44 "pencils down" and likely timeline to wikitech-l etc. - https://phabricator.wikimedia.org/T390692#10797341 (10ABreault-WMF) https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/XSEMSIVRI6G6AECG66LOK26I4YMUJFEA/ [17:55:08] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797342 (10bd808) [17:57:00] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797345 (10bd808) [17:57:21] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797346 (10bd808) +1 [17:59:34] 10Gerrit, 06collaboration-services: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498#10797357 (10Arnoldokoth) [18:01:09] 10Gerrit, 06collaboration-services: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498#10797381 (10Arnoldokoth) I restarted apache. The alerts recovered too. @Daimona Any improvement on your end. [18:06:39] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797398 (10bd808) [18:19:49] 06Release-Engineering-Team, 06Security-Team, 07Security: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10797436 (10sbassett) [18:19:58] 10Phabricator: New Column Trigger: Assign only unassigned tasks - https://phabricator.wikimedia.org/T393464#10797438 (10Aklapper) P75809 on top of https://we.phorge.it/D25991 seems to work but I don't understand the full call stack. So not sure if that one check is enough. Needs more testing: {P75809,lines=15} [18:26:47] 10MediaWiki-Releasing, 05MW-1.44-release: Send a branch announcement e-mail for MW REL1_44 - https://phabricator.wikimedia.org/T393503 (10ABreault-WMF) 03NEW [18:30:08] 10MediaWiki-Releasing, 05MW-1.44-release: Send a branch announcement e-mail for MW REL1_44 - https://phabricator.wikimedia.org/T393503#10797479 (10ABreault-WMF) [18:30:09] 10MediaWiki-Releasing, 05MW-1.44-notes, 05MW-1.44-release, 05Release: Release MediaWiki 1.44.0 - https://phabricator.wikimedia.org/T390686#10797480 (10ABreault-WMF) [18:38:03] 06Project-Admins, 06Release-Engineering-Team, 06Security-Team, 07Security: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10797489 (10Aklapper) Any [name](https://www.mediawiki.org/wiki/Naming_things) in mind? Just want to make... [18:39:17] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase zuul3 quotas for cpu/ram/disk/instances - https://phabricator.wikimedia.org/T393496#10797497 (10Andrew) 05Openβ†’03Resolved a:03Andre... [18:54:47] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.44-release, 13Patch-For-Review: Add REL1_44 to ExtensionDistributor as the development snapshot - https://phabricator.wikimedia.org/T390794#10797535 (10MacFan4000) 05Openβ†’03Resolved [18:58:10] (03close) 10addshore: go bump spf13 packages [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/614 [19:04:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:09:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:27:56] 10Continuous-Integration-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 07ci-test-error, 13Patch-For-Review: api-testing CI failure: Timeout of 5000ms exceeded in "before all" hook in "POST /campaignevents/v0/event_registration" - https://phabricator.wikimedia.org/T389863#10797661 (10Daim... [19:50:27] (03PS1) 10JHathaway: puppet: use bind mounts for local runs [integration/config] - 10https://gerrit.wikimedia.org/r/1142674 [19:55:33] 10MediaWiki-Releasing, 05MW-1.44-release: Send a branch announcement e-mail for MW REL1_44 - https://phabricator.wikimedia.org/T393503#10797720 (10ABreault-WMF) 05Openβ†’03Resolved https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/JT6QELB6V6HOT2T6QTVNGJVVNUJRVB2A/ [20:04:40] (03CR) 10JHathaway: "requires 1142675, testing instructions on that change" [integration/config] - 10https://gerrit.wikimedia.org/r/1142674 (owner: 10JHathaway) [20:06:37] Project beta-scap-sync-world build #205402: 04FAILURE in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/205402/ [20:07:28] 10MediaWiki-Releasing, 10LibUp, 05MW-1.44-release: Configure LibraryUpgrader for REL1_44 - https://phabricator.wikimedia.org/T393506 (10ABreault-WMF) 03NEW [20:09:02] (03open) 10arlolra: Add REL1_44 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/75 (https://phabricator.wikimedia.org/T393506) [20:09:31] (03merge) 10taavi: Add REL1_44 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/75 (https://phabricator.wikimedia.org/T393506) (owner: 10arlolra) [20:13:41] 10MediaWiki-Releasing, 10LibUp, 05MW-1.44-release, 13Patch-For-Review: Configure LibraryUpgrader for REL1_44 - https://phabricator.wikimedia.org/T393506#10797800 (10ABreault-WMF) 05Openβ†’03Resolved [20:16:41] Project beta-scap-sync-world build #205403: 04STILL FAILING in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/205403/ [20:25:07] I filed https://phabricator.wikimedia.org/T393507 for the beta-scap-sync-world problem. [20:26:58] Yippee, build fixed! [20:26:58] Project beta-scap-sync-world build #205404: 09FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/205404/