[00:07:36] hrm, looking at this it seems like /srv/mediawiki-staging/scap/image-build has a bunch of state files that point to the last image build which...if I'm reading things right...is probably the only thing that we need to save. [00:09:02] dancy: or dduvall ^ is that right? can we purge any docker images from the deploy host not listed in state files? [00:12:49] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598919 (10thcipriani) I know we use some of these images to build subsequent images. It //may be// that the only images we need to keep are the ones listed i... [00:38:19] 10WikimediaDebug: Extract and display hostname from server header - https://phabricator.wikimedia.org/T340727#10598981 (10bd808) >>! In T340727#10598839, @bd808 wrote: > We don't currently vary display on the popup.html add-on control screen based on the active tab's content other than toggling `hidden: true` fo... [00:40:08] 10WikimediaDebug: Extract and display hostname from server header - https://phabricator.wikimedia.org/T340727#10598983 (10bd808) >>! In T340727#10598839, @bd808 wrote: > We don't currently vary display on the popup.html add-on control screen based on the active tab's content other than toggling `hidden: true` fo... [00:40:42] 10WikimediaDebug: Evaluate single-tab mode for WikimediaDebug - https://phabricator.wikimedia.org/T269031#10598986 (10bd808) >>! In T269031#8173170, @kostajh wrote: > This sounds like it might be more complicated than it's worth. > > My original motivation for this request is that I sometimes forget that Wikime... [00:42:42] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598993 (10dduvall) I've removed a bunch of old images for now. I will talk with Tyler and Ahmon tomorrow about long term solutions. For future reference, sc... [00:43:34] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10598994 (10dduvall) Even after the deletion, there is still a lot of reclaimable space: ` dduvall@deploy2002:~$ docker system df TYPE TOTAL AC... [00:44:43] (03PS1) 10BryanDavis: README: Update Firefox add-on label [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124209 [00:47:05] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10599005 (10thcipriani) >>! In T387796#10598965, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://sal.toolforge.o... [00:47:17] (03CR) 10BryanDavis: [C:03+2] README: Update Firefox add-on label [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124209 (owner: 10BryanDavis) [00:47:40] (03Merged) 10jenkins-bot: README: Update Firefox add-on label [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124209 (owner: 10BryanDavis) [00:59:21] (03PS1) 10Jforrester: Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos [integration/config] - 10https://gerrit.wikimedia.org/r/1124212 (https://phabricator.wikimedia.org/T386108) [01:00:51] 10Continuous-Integration-Config, 07Epic, 13Patch-For-Review, 07PHP 8.4 support: Make PHP 8.4 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T386108#10599018 (10Jdforrester-WMF) [01:01:34] (03CR) 10Jforrester: [C:03+2] Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos [integration/config] - 10https://gerrit.wikimedia.org/r/1124212 (https://phabricator.wikimedia.org/T386108) (owner: 10Jforrester) [01:03:06] (03Merged) 10jenkins-bot: Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos [integration/config] - 10https://gerrit.wikimedia.org/r/1124212 (https://phabricator.wikimedia.org/T386108) (owner: 10Jforrester) [01:03:35] !log Zuul: Start testing in PHP 8.4 for 'mediawiki-php-library' repos, for T386108 [01:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:03:37] T386108: Make PHP 8.4 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T386108 [01:04:31] (03PS4) 10Hashar: Archive mediawiki/extensions/ExtJSBase [integration/config] - 10https://gerrit.wikimedia.org/r/1111611 (https://phabricator.wikimedia.org/T383857) (owner: 10Hslater) [01:04:44] (03PS5) 10Jforrester: Zuul: [mediawiki/extensions/ExtJSBase] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1111611 (https://phabricator.wikimedia.org/T383857) (owner: 10Hslater) [01:04:51] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ExtJSBase] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1111611 (https://phabricator.wikimedia.org/T383857) (owner: 10Hslater) [01:06:29] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ExtJSBase] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1111611 (https://phabricator.wikimedia.org/T383857) (owner: 10Hslater) [01:09:13] (03CR) 10Jforrester: [C:03+2] Zuul: Add WgevaertWikiBase to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1059926 (owner: 10Pppery) [01:10:37] (03Abandoned) 10Hslater: Remove ExtJSBase dependency from BlueSpiceFoundation [integration/config] - 10https://gerrit.wikimedia.org/r/1115423 (owner: 10Hslater) [01:10:53] (03Merged) 10jenkins-bot: Zuul: Add WgevaertWikiBase to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1059926 (owner: 10Pppery) [01:13:32] !log Zuul: Add WgevaertWikiBase to CI allowlist [01:13:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:15:06] (03CR) 10Jforrester: "Alternatively we could make this CI step non-voting, so whether it passes or fails has no effect on code's mergeability?" [integration/config] - 10https://gerrit.wikimedia.org/r/1123728 (https://phabricator.wikimedia.org/T387601) (owner: 10Eric Gardner) [01:31:52] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1124215 (https://phabricator.wikimedia.org/T386570) [01:32:58] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1124215 (https://phabricator.wikimedia.org/T386570) (owner: 10Jforrester) [01:39:20] (03PS2) 10Jforrester: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1124215 (https://phabricator.wikimedia.org/T386570) [01:41:10] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1124215 (https://phabricator.wikimedia.org/T386570) (owner: 10Jforrester) [01:42:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1124215 (https://phabricator.wikimedia.org/T386570) (owner: 10Jforrester) [01:42:29] 10WikimediaDebug, 06SRE, 06Traffic, 07Developer Productivity, 13Patch-For-Review: Let X-Analytics response header pass through with WikimediaDebug - https://phabricator.wikimedia.org/T305794#10599124 (10bd808) 05Open→03In progress a:03bd808 [01:42:58] !log Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Disable on PHP 8.4, for T386570 [01:43:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:43:00] T386570: phan-taint-check test failure for exit/die with php8.4 - https://phabricator.wikimedia.org/T386570 [02:47:29] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Automatically export and publish a list of WMF deployed code repositories in Bitergia's JSON format - https://phabricator.wikimedia.org/T385529#10599170 (10thcipriani) Currently our release tools has a [[https://gitlab.wikimedia.org/repos/... [03:57:26] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10599239 (10Dzahn) p:05High→03Medium [07:15:06] 10Release-Engineering-Team (Doing 😎): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10599372 (10hashar) 05Open→03Resolved The spammy processing is gone (except for the mediawiki/core deletion). **I claim a success** ` Systemd timer ran the following command:... [07:37:05] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822 (10Joe) 03NEW [07:40:36] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823 (10hashar) 03NEW [07:40:47] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10599459 (10hashar) p:05Triage→03Unbreak! [07:41:08] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10599460 (10hashar) [07:41:10] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10599461 (10hashar) [07:47:31] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10599476 (10fgiunchedi) Also note that standard partman recipes nowadays don't allocate all space on LVM for emergency response purposes, e.g. ` deploy2002:~$... [07:49:41] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10599484 (10hashar) ` $ systemctl status train-presync.service ● train-presync.service - Perform beginning-of-week train operations Loaded: loaded (/lib/systemd/system/train-presync.service; static)... [07:53:28] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10599525 (10hashar) [08:00:18] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10599559 (10hashar) That is because of the requested permission `declarativeNetRequest` is for any hosts, though its further restricted with `host_restrictions`. ` lang... [08:03:08] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10599570 (10hashar) Oh I can do: `scap stage-train -Dfull_image_build:True --yes auto` [08:45:19] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10599692 (10JMeybohm) 05Resolved→03Open >>! In T288629#10598113, @dancy wrote: >>>! In T288629#10582102, @JMeybohm wrote:... [08:45:49] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10599694 (10JMeybohm) a:05dancy→03None [08:50:28] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10599725 (10hashar) 05Open→03Resolved a:03hashar I cancelled the `scap stage-train` since the backport window was starting. @dcausse did a backport and a result ended up pushing the train, which... [10:18:09] jayme: looks like the new helm-lint job is slower than it used to: https://phabricator.wikimedia.org/T387781 :) [10:20:33] not really I guess...it's just that we almost never do full CI runs (because it's too slow). But when we change CI code, we obviously do - I'll reply on the task [10:23:23] 10Continuous-Integration-Config: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10600049 (10hashar) After we update the image, the job I was watching took 17 minutes. Jenkins does show a graphical histogram of the build durations ( https://integration.wikimedia.o... [10:25:22] jayme: I pasted some findings from yesterday so they are kept for the future selves :] [10:27:10] if we where to generate comparable pre-change numbers, we could run 'rake all' with the old image (on a commit prior to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1123006) [10:29:11] there is also a change that took 1m37 and then 18m which is a bit mysterious [10:29:26] well at least to me :) [10:30:28] ah that 1123006 got merged in between :) [10:30:31] same reason...the second one did a full ci run [10:32:38] 10Continuous-Integration-Config: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10600093 (10hashar) > That last change is interesting cause the previous build in test took just 1m37s seconds while the next one took 18m02s: > > * https://integration.wikimedia.org/... [10:32:49] yes I commented on the task [10:33:20] so when https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1124166 entered CI [10:33:30] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1123006 was already in the queue [10:33:43] so the job ran with master + 1123006 + 1124166 [10:33:57] possibly cause helm diff does a diff against origin/master rather than the parent commit [10:34:02] or whatever really :) [10:36:15] g.remote('origin').fetch [10:36:15] changes = g.changed_files('origin/master') [10:36:22] I think that is the reason :] [10:36:39] yeah [10:36:54] but it's also kind of correct when you test patch chains I guess [10:37:12] well if the patch behind passed it get merged [10:37:22] and once merged the next patch would not trigger a full run [10:37:41] the diff would only have the files changed by that last change [10:38:04] so instead of diffing against origin/master that should be against HEAD^ [10:38:10] anyway, it is probably quite rare [10:38:18] it is indeed [10:38:35] the puppet compiler has the same issue, it does the diff against production branch [10:38:58] which can be confusing when running the compiler against each changes of a chain ( production > A > B , then the compilation of B would have in the diff everything from A) [10:39:09] but is handy if you want to get the outcome of the whole chain [10:39:10] anyway [10:39:33] that explains why the same change took 1m and then 18m [10:40:01] thanks jayme ! [10:40:30] thank you for digging :) [10:56:26] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10600142 (10hashar) [11:54:27] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10600299 (10Clement_Goubert) Some of that is because we're building double the images we used to with the 8.1 migration. As a mitigation, I have lowered the ag... [11:57:43] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10600326 (10Clement_Goubert) Cleaned up all images older than 7 days: ` cgoubert@deploy2002:~$ df -h /srv Filesystem Size Used Avail Use% Mounted on... [12:43:13] 06Release-Engineering-Team, 06serviceops: train presync failed - https://phabricator.wikimedia.org/T387823#10600532 (10hashar) From the information @jnuche gave me, when we run `scap stage-train -Dfull_image_build:True --yes auto` the full image build causes the code to be in a single layer, that is to av... [12:58:07] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10600604 (10hashar) > @hashar For the next time we create a new doc machine, would you still request 2 separate disks or just a single disk that is large enough? See above T3... [13:04:33] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10600609 (10Mvolz) This is probably the same bug? But when I updated my browser today I was told the extension was turned off automatically because it "wasn't supported". [13:25:25] 10Phabricator (phabricator-next), 10Phabricator Antivandalism Extension, 10Release-Engineering-Team (Doing 😎), 07Performance Issue: More AVA performance improvements - https://phabricator.wikimedia.org/T386704#10600636 (10Aklapper) [13:33:42] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10600671 (10hashar) @Mvolz wrote: > This is probably the same bug? But when I updated my browser today I was told the extension was turned off automatically because it "... [14:15:07] 10WikimediaDebug: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10600851 (10hashar) If I change the permission in manifest.json from `declarativeNetRequest` to `declarativeNetRequestWithHostAccess`, then unload the extension and relo... [14:28:14] (03PS1) 10Hashar: Avoid block content on any page permission [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124442 (https://phabricator.wikimedia.org/T387822) [14:29:31] 10WikimediaDebug, 13Patch-For-Review: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10600899 (10hashar) a:03hashar [14:38:28] (03PS1) 10Hashar: v3.0.1: Update changelog [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1124448 [15:09:52] 06Project-Admins: Create project tag for SpamWatch - https://phabricator.wikimedia.org/T387869 (10TheresNoTime) 03NEW [15:20:59] 06Project-Admins: Create project tag for WMA-Hackathon-2025 - https://phabricator.wikimedia.org/T387871 (10Eugene233) 03NEW [15:23:15] 06Project-Admins: Create project tag for WMA-Hackathon-2025 - https://phabricator.wikimedia.org/T387871#10601180 (10Eugene233) [15:26:24] 06Project-Admins: Create project tag for WMA-Hackathon-2025 - https://phabricator.wikimedia.org/T387871#10601214 (10Eugene233) [15:28:49] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2025-03-04 - https://phabricator.wikimedia.org/T387873 (10brennen) 03NEW [15:29:48] (03open) 10brennen: update submodules for 2025-03-04 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/61 (https://phabricator.wikimedia.org/T387873) [15:56:26] (03merge) 10brennen: update submodules for 2025-03-04 release [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/61 (https://phabricator.wikimedia.org/T387873) [15:57:26] 10WikimediaDebug, 13Patch-For-Review: WikimediaDebug Firefox extension requires permission to block content on any page - https://phabricator.wikimedia.org/T387822#10601341 (10bd808) >>! In T387822#10600609, @Mvolz wrote: > This is probably the same bug? But when I updated my browser today I was told the exten... [16:02:01] 10Phabricator (2025-03-04), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-03-04 - https://phabricator.wikimedia.org/T387873#10601391 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d81974ec-6b2f-460c-89c6-1d3d03408c2c) set by je... [16:02:15] 10Phabricator (2025-03-04), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-03-04 - https://phabricator.wikimedia.org/T387873#10601393 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=46dc6e1e-4b34-4655-932e-4d8b16a9d175) set by je... [16:04:59] 10Phabricator (2025-03-04), 10Phabricator Antivandalism Extension, 10Release-Engineering-Team (Doing 😎), 07Performance Issue: More AVA performance improvements - https://phabricator.wikimedia.org/T386704#10601406 (10Malyacko) ...and I can still comment and even pass AVA. :P [16:08:01] 10Phabricator (2025-03-04), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-03-04 - https://phabricator.wikimedia.org/T387873#10601450 (10Aklapper) 05Open→03Resolved a:03brennen Thank you! Some non-working code removed (see T363337), plus... [16:11:20] 10Phabricator (2025-03-04), 10Phabricator Antivandalism Extension, 10Release-Engineering-Team (Doing 😎), 07Performance Issue: More AVA performance improvements - https://phabricator.wikimedia.org/T386704#10601476 (10Aklapper) 05Open→03Resolved Done and deployed. Latest code change should make the A... [16:18:16] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10601555 (10Ottomata) Ya makes sense! And we do that now. But, users of packages have to know and configure... [16:21:52] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: Jobs on Digital Ocean Cloud Runners are being OOM killed - https://phabricator.wikimedia.org/T387886 (10Jelto) 03NEW [16:24:25] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: Jobs on Digital Ocean Cloud Runners are being OOM killed - https://phabricator.wikimedia.org/T387886#10601621 (10brennen) [17:21:14] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849#10601958 (10bd808) 05In progress→03Resolved [17:23:19] 10Beta-Cluster-Infrastructure, 07OKR-Work: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10601963 (10bd808) 05Open→03Resolved [17:28:01] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10601981 (10brennen) [17:41:47] 10WikimediaDebug: WikimediaDebug no longer available in Chrome web store? - https://phabricator.wikimedia.org/T387899 (10colewhite) 03NEW [17:44:01] 10WikimediaDebug: WikimediaDebug no longer available in Chrome web store? - https://phabricator.wikimedia.org/T387899#10602085 (10Reedy) [17:44:02] 10Release-Engineering-Team (Priority Backlog 📥), 10WikimediaDebug: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10602086 (10Reedy) [17:44:15] 10WikimediaDebug: WikimediaDebug no longer available in Chrome web store? - https://phabricator.wikimedia.org/T387899#10602089 (10Reedy) Pretty sure this should be fixed by the WIP over at {T312694}... [17:48:27] 06Release-Engineering-Team, 06serviceops: deploy1003 reports helmfileAdminPendingChanges - https://phabricator.wikimedia.org/T387900 (10jijiki) 03NEW [17:51:25] 10WikimediaDebug: WikimediaDebug no longer available in Chrome web store? - https://phabricator.wikimedia.org/T387899#10602121 (10hashar) >>! In T387822#10601341, @bd808 wrote: > > Chrome seems to be A/B testing the Manifest v2 shutdown. A report from Roan of the app store showing the extension as blocked (T312... [18:01:51] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10602151 (10thcipriani) Got it. Well. The "instance" scope seems fully bought-in to the github model of each... [18:07:43] (03PS1) 10Bartosz Dziewoński: wmf-config-wg-vars: Don't match variables with suffixes [tools/code-utils] - 10https://gerrit.wikimedia.org/r/1124495 [18:08:59] (03CR) 10Bartosz Dziewoński: "This reports two new unused variables: `TrackGlobalJsonLinksNamespace` (previously matched `TrackGlobalJsonLinksNamespaces`) and `ReadingL" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/1124495 (owner: 10Bartosz Dziewoński) [18:32:23] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 06Data-Engineering: Create an instance-level npm package registry in Gitlab - https://phabricator.wikimedia.org/T384364#10602206 (10Ottomata) > Who is using these packages, currently? Any team that uses service-runner, or hopef... [18:34:27] (03open) 10dancy: Allow trainbranchbot to delete wmf/* branches [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/123 [18:34:29] (03update) 10dancy: Allow trainbranchbot to delete wmf/* branches [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/123 [18:35:26] (03merge) 10dancy: Allow trainbranchbot to delete wmf/* branches [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/123 [19:26:31] (03PS1) 10Bartosz Dziewoński: wmf-config-wg-vars: Improve display of multiple filenames in table [tools/code-utils] - 10https://gerrit.wikimedia.org/r/1124506 [19:30:12] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T386214#10602597 (10hashar) Some requests to CampaingEvent produce PHP undefined array key: T387914 . The page seems to be fine so I do... [20:03:44] (03open) 10dancy: Use a top-level Gerrit project to hold common access controls [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/124 [20:03:46] (03update) 10dancy: Use a top-level Gerrit project to hold common access controls [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/124 [20:05:36] (03update) 10dancy: Use a top-level Gerrit project to hold common access controls [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/124 [20:06:12] (03merge) 10dancy: Use a top-level Gerrit project to hold common access controls [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/124 [20:15:59] 10Scap, 06serviceops: scap leaves mediawiki main releases in the "forward" version after a canary rollback - https://phabricator.wikimedia.org/T375497#10602782 (10dancy) @Scott_French I like your idea (and patches are always welcome!). Don't worry too much about `--stop-before-sync`. It is used for automated... [20:47:42] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10602894 (10dduvall) Is there something else filling up `/srv`? It has filled back up and `docker system df` hasn't changed much. ` dduvall@deploy2002:~$ df -... [20:48:06] 06Project-Admins: Create project tag for SpamWatch - https://phabricator.wikimedia.org/T387869#10602899 (10Peachey88) Is this extension documented anywhere? https://www.mediawiki.org/wiki/Extension:SpamWatch doesn't currently exist [20:53:51] 06Project-Admins: Create project tag for WMA-Hackathon-2025 - https://phabricator.wikimedia.org/T387871#10602933 (10Peachey88) Is this hackathon likely to occur in future years? It might be worth considering making the project yearless, and then doing the 2025 as a subproject underneath it. [20:54:37] 10Continuous-Integration-Config, 07Browser-Tests: Remove wdio-video-reporter from all repositories - https://phabricator.wikimedia.org/T294341#10602947 (10Aklapper) [21:04:43] 10Release-Engineering-Team (Priority Backlog 📥): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927 (10dduvall) 03NEW [21:05:03] 10Release-Engineering-Team (Priority Backlog 📥): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#10602994 (10dduvall) @dancy thoughts on the implementation? [21:06:11] 10Release-Engineering-Team (Priority Backlog 📥): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#10603005 (10dduvall) [21:07:58] 10Release-Engineering-Team (Priority Backlog 📥): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#10603013 (10dduvall) [21:21:07] 10Release-Engineering-Team (Radar), 06serviceops-radar, 06SRE: deployment server - low disk space on /srv - https://phabricator.wikimedia.org/T387796#10603063 (10Dzahn) It's still /srv/docker/ (/srv/docker/overlay2) . Back to using about 164GB. [21:59:35] (03CR) 10Cwhite: helpers.ts: Include timestamp in phab description (032 comments) [releng/phatality] - 10https://gerrit.wikimedia.org/r/1116796 (https://phabricator.wikimedia.org/T385458) (owner: 10Reedy) [21:59:56] (03PS4) 10Cwhite: helpers.ts: Include timestamp in phab description [releng/phatality] - 10https://gerrit.wikimedia.org/r/1116796 (https://phabricator.wikimedia.org/T385458) (owner: 10Reedy) [22:06:45] (03PS1) 10Jforrester: Zuul: [mediawiki/services/example-node-api] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1124531 (https://phabricator.wikimedia.org/T387933) [22:06:47] (03PS1) 10Jforrester: jjb: Drop jobs for archived mediawiki/services/example-node-api [integration/config] - 10https://gerrit.wikimedia.org/r/1124532 (https://phabricator.wikimedia.org/T387933) [22:08:44] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/example-node-api] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1124531 (https://phabricator.wikimedia.org/T387933) (owner: 10Jforrester) [22:09:58] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/example-node-api] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1124531 (https://phabricator.wikimedia.org/T387933) (owner: 10Jforrester) [22:10:05] (03CR) 10Jforrester: [C:03+2] "Manually deleted." [integration/config] - 10https://gerrit.wikimedia.org/r/1124532 (https://phabricator.wikimedia.org/T387933) (owner: 10Jforrester) [22:10:14] !log Zuul: [mediawiki/services/example-node-api] Mark as archived, for T387933 [22:10:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:10:16] T387933: Archive the mediawiki/services/example-node-api repo - https://phabricator.wikimedia.org/T387933 [22:11:17] (03Merged) 10jenkins-bot: jjb: Drop jobs for archived mediawiki/services/example-node-api [integration/config] - 10https://gerrit.wikimedia.org/r/1124532 (https://phabricator.wikimedia.org/T387933) (owner: 10Jforrester) [22:12:48] (03CR) 10Cwhite: [C:03+2] helpers.ts: Include timestamp in phab description [releng/phatality] - 10https://gerrit.wikimedia.org/r/1116796 (https://phabricator.wikimedia.org/T385458) (owner: 10Reedy) [22:13:06] (03Merged) 10jenkins-bot: helpers.ts: Include timestamp in phab description [releng/phatality] - 10https://gerrit.wikimedia.org/r/1116796 (https://phabricator.wikimedia.org/T385458) (owner: 10Reedy) [22:16:37] Project beta-scap-sync-world build #196420: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196420/ [22:26:38] Project beta-scap-sync-world build #196421: 04STILL FAILING in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196421/ [22:29:22] (03open) 10lwatson: releases: Bump Codex to 1.20.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/64 (https://phabricator.wikimedia.org/T387757) [22:30:10] (03approved) 10volker-e: releases: Bump Codex to 1.20.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/64 (https://phabricator.wikimedia.org/T387757) (owner: 10lwatson) [22:30:14] (03merge) 10volker-e: releases: Bump Codex to 1.20.3 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/64 (https://phabricator.wikimedia.org/T387757) (owner: 10lwatson) [22:36:31] Project beta-scap-sync-world build #196422: 04STILL FAILING in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196422/ [22:46:34] Project beta-scap-sync-world build #196423: 04STILL FAILING in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196423/ [22:56:35] Project beta-scap-sync-world build #196424: 04STILL FAILING in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196424/ [23:04:07] ^ this is caused by https://gerrit.wikimedia.org/r/1120610 [23:06:29] Project beta-scap-sync-world build #196425: 04STILL FAILING in 1 min 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196425/ [23:08:48] 10Release-Engineering-Team (Doing 😎): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#10603431 (10dduvall) 05Open→03In progress p:05Triage→03Medium a:03dduvall [23:09:21] (03open) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:09:25] (03update) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:09:34] (03update) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:13:08] (03update) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:13:10] (03update) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:13:12] (03update) 10dduvall: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) [23:14:47] (03open) 10brennen: Draft: Alertmanager config for sending alerts [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/427 (https://phabricator.wikimedia.org/T373548) [23:16:37] Project beta-scap-sync-world build #196426: 04STILL FAILING in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196426/ [23:18:25] 10Beta-Cluster-Infrastructure, 07Developer Productivity: beta cluster down - Internal error - https://phabricator.wikimedia.org/T387938 (10Etonkovidova) 03NEW [23:21:54] 10Beta-Cluster-Infrastructure, 07Developer Productivity: beta cluster down - Internal error - https://phabricator.wikimedia.org/T387938#10603475 (10dancy) I wonder if this is related to T166010. [23:22:29] zabe: I'll try reverting [23:26:33] !log Disabling beta-scap-sync-world for noise reduction while dealing with T166010 [23:26:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:26:35] T166010: The Great Namespaceization Effort - https://phabricator.wikimedia.org/T166010 [23:26:41] Project beta-scap-sync-world build #196427: 04STILL FAILING in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/196427/ [23:33:35] (03approved) 10dancy: make-container-image: Support additional image labels [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/147 (https://phabricator.wikimedia.org/T387927) (owner: 10dduvall)