[01:55:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:00:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:05:19] (03PS1) 10Mhurd: dockerfiles: keep npm debug logs out of the castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/1338389 (https://phabricator.wikimedia.org/T437376) [02:05:26] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541 (10Ottomata) 03NEW [02:22:15] (03PS10) 10Phedenskog: Run SQLite tests in parallel by having one db per worker [integration/quibble] - 10https://gerrit.wikimedia.org/r/1289330 (https://phabricator.wikimedia.org/T426684) [02:29:36] (03CR) 10CI reject: [V:04-1] Run SQLite tests in parallel by having one db per worker [integration/quibble] - 10https://gerrit.wikimedia.org/r/1289330 (https://phabricator.wikimedia.org/T426684) (owner: 10Phedenskog) [02:33:39] (03CR) 10Pwangai: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1289330 (https://phabricator.wikimedia.org/T426684) (owner: 10Phedenskog) [02:41:35] (03CR) 10Mhurd: castor: load the cache without the delayed rename pass (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1330226 (https://phabricator.wikimedia.org/T436173) (owner: 10Phedenskog) [05:06:37] (03CR) 10Phedenskog: [C:03+2] jjb: remove dupe --reporting-url from Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337963 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [05:08:48] (03Merged) 10jenkins-bot: jjb: remove dupe --reporting-url from Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337963 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [05:20:10] !log Updated 134 Quibble Jenkins jobs to drop the duplicated --reporting-url T323750 [05:20:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:20:13] T323750: Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 [05:47:08] (03CR) 10Phedenskog: [C:03+2] "I've tested this in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1155330 and it works fine" [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [05:48:59] (03Merged) 10jenkins-bot: jjb: Run only database tests in the sqlite and postgres core jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) (owner: 10Phedenskog) [05:49:05] !log Updated quibble-for-mediawiki-core postgres and sqlite jobs to run only the phpunit-database stage T437134 [05:49:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:49:07] T437134: Run only the PHPUnit Database tests in the sqlite and postgres core jobs to speed up the gated pipeline - https://phabricator.wikimedia.org/T437134 [07:57:25] Hi folks, I deleted previous pipeline jobs but it still says the project is using "418.2 MiB Project Storage", do you know how I clean it up?? Not sure if there's a limit to Project Storage but we certainly don't need previous CI artifacts. https://gitlab.wikimedia.org/repos/sre/grafana-plugins [08:57:06] denisse: 👋 Where are you seeing 418.2 MiB? I'm looking at the usage quota of that project and I see a total usage of 186.8 MiB, which matches the pipeline usage [08:57:54] did you check the storage right after you deleted the old pipelines? I think there's a delay in displaying the updated value while the new usage is being calculated [09:16:19] 06Release-Engineering-Team, 06Infrastructure-Foundations: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12306788 (10elukey) Very high level breakdown for /restricted: ` elukey@registry1004:~$ curl http://localhost:5002/v2/_catalog {"repositories":["rest... [10:25:28] (03CR) 10Samtar: [C:03+1] zuul: add ORES and Wikibase as phan dependency of PersonalDashboard [integration/config] - 10https://gerrit.wikimedia.org/r/1338299 (https://phabricator.wikimedia.org/T436570) (owner: 10MusikAnimal) [11:23:42] (03CR) 10Phedenskog: [C:03+1] dockerfiles: keep npm debug logs out of the castor cache [integration/config] - 10https://gerrit.wikimedia.org/r/1338389 (https://phabricator.wikimedia.org/T437376) (owner: 10Mhurd) [11:38:26] 14Release-Engineering-Team (CI & Testing services), 10Cloud-VPS, 10Tools, 06tools-infrastructure-team, 07Wikimedia-Incident: Resolve keystone schema errors - https://phabricator.wikimedia.org/T235070#12307441 (10MLechvien-WMF) [11:38:29] 14Release-Engineering-Team (CI & Testing services), 10Cloud-VPS, 10Tools, 06tools-infrastructure-team, 07Wikimedia-Incident: Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07 - https://phabricator.wikimedia.org/T234834#12307443 (10MLechvien-WMF) [11:40:18] 10Continuous-Integration-Infrastructure, 14RelEng-Archive-FY201718-Q1, 10Cloud-VPS, 06tools-infrastructure-team, 07Wikimedia-Incident: contintcloud instance refuses to launch due to "Maximum number of fixed ips exceeded - https://phabricator.wikimedia.org/T171158#12307494 (10MLechvien-WMF) [11:40:47] (03CR) 10Phedenskog: "Oh that code is to complicated for my head :) I'm think there could be a simpler version if we can use the modified time directly? Somethi" [integration/config] - 10https://gerrit.wikimedia.org/r/1338086 (https://phabricator.wikimedia.org/T437377) (owner: 10Mhurd) [12:51:55] (03open) 10jforrester: releases: Bump stylelint-config-wikimedia to 0.19.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/147 [12:52:37] (03merge) 10jforrester: releases: Bump stylelint-config-wikimedia to 0.19.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/147 [14:02:29] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12308120 (10Aklapper) 05Open→03Resolved a:03Aklapper `Creating a project's first subproject moves all members to... [14:33:55] (03CR) 10Phedenskog: "Really looking forward to see this in production and see what impact it will have on the saves and the queue!" [integration/config] - 10https://gerrit.wikimedia.org/r/1338086 (https://phabricator.wikimedia.org/T437377) (owner: 10Mhurd) [14:58:47] 06Release-Engineering-Team, 06Infrastructure-Foundations: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12308543 (10elukey) I tried to delete restricted images via docker-registryctl on build nodes, but the required credentials (ci-restricted) are not de... [15:12:08] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:20:08] 06Release-Engineering-Team (Radar), 06ServiceOps, 10ServiceOps-Services-Oids, 06SRE-OnFire, 07Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162#12308680 (10Dzahn) After thinking about this some more: That last comment (only) makes sense if you lo... [16:44:17] 06Release-Engineering-Team (Radar), 06ServiceOps, 10ServiceOps-Services-Oids, 06SRE-OnFire, 07Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162#12309163 (10Raine) 05Resolved→03Open >>! In T309162#12308680, @Dzahn wrote: > After thinking about thi... [16:46:42] 06Release-Engineering-Team (Radar), 06ServiceOps, 10ServiceOps-Services-Oids, 06SRE-OnFire, 07Sustainability: deployment servers should have identical repo data (was: Remove old scap repositories from deploy1002) - https://phabricator.wikimedia.org/T309162#12309170 (10Dzahn) [17:01:03] FIRING: InstanceDown: Project deployment-prep instance deployment-acme-chief05 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:01:08] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-acme-chief05 is down - https://phabricator.wikimedia.org/T437629 (10wmcs-alerts) 03NEW [17:06:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-acme-chief05 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:08:54] (03update) 10bwang: Draft: Codex 2.7.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/146 (https://phabricator.wikimedia.org/T436714) [17:22:24] (03update) 10bwang: releases: Bump Codex to 2.7.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/146 (https://phabricator.wikimedia.org/T436714) [17:35:56] (03update) 10bwang: releases: Bump Codex to 2.7.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/146 (https://phabricator.wikimedia.org/T436714) [17:47:38] (03merge) 10volker-e: releases: Bump Codex to 2.7.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/146 (https://phabricator.wikimedia.org/T436714) (owner: 10bwang) [17:50:02] (03open) 10phedenskog: core: Show cache hit counts in the gate execution time tooltip [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/64 [18:12:49] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for fengri - https://phabricator.wikimedia.org/T436512#12309444 (10brennen) 05Open→03Resolved a:03brennen [18:13:36] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for fengri - https://phabricator.wikimedia.org/T436512#12309446 (10brennen) Added your user to #trusted-contributors based on on-wiki contributions. The GitLab account approval bot should pick it up shortly. [18:15:28] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309448 (10tchin) I moved eventstreams and eventgate to consume from main. Seems to work fine and @Sfaci inadvertently helped me test it. I complet... [18:17:08] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:19:03] (03CR) 10Pwangai: "Benchmarked this patch on a dedicated CI agent (10 CPUs) running the real" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1289330 (https://phabricator.wikimedia.org/T426684) (owner: 10Phedenskog) [18:25:44] (03PS1) 10Sbisson: Zuul: [mediawiki/extensions/ArticleGuidance] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1339054 (https://phabricator.wikimedia.org/T437586) [18:53:05] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ArticleGuidance] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1339054 (https://phabricator.wikimedia.org/T437586) (owner: 10Sbisson) [18:54:11] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/PersonalDashboard] Add ORES & Wikibase phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1338299 (https://phabricator.wikimedia.org/T436570) (owner: 10MusikAnimal) [18:54:15] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/PersonalDashboard] Add ORES & Wikibase phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1338299 (https://phabricator.wikimedia.org/T436570) (owner: 10MusikAnimal) [18:55:21] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ArticleGuidance] Add CommunityConfiguration for phan [integration/config] - 10https://gerrit.wikimedia.org/r/1339054 (https://phabricator.wikimedia.org/T437586) (owner: 10Sbisson) [18:55:39] !log Zuul: [mediawiki/extensions/ArticleGuidance] Add CommunityConfiguration for phan for T437586 [18:55:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:55:42] T437586: Allow configuring redirects via Community Configuration - https://phabricator.wikimedia.org/T437586 [18:56:01] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PersonalDashboard] Add ORES & Wikibase phan deps [integration/config] - 10https://gerrit.wikimedia.org/r/1338299 (https://phabricator.wikimedia.org/T436570) (owner: 10MusikAnimal) [18:56:18] !log Zuul: [mediawiki/extensions/PersonalDashboard] Add ORES & Wikibase phan deps for T436570 and T437491 [18:56:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:56:22] T436570: PersonalDashboard: Shared feed REST API for the dashboard feed scaffold - https://phabricator.wikimedia.org/T436570 [18:56:23] T437491: Ensure Wikidata descriptions are added to edits in all Review Changes sources - https://phabricator.wikimedia.org/T437491 [19:13:15] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12309688 (10apaskulin) 05Stalled→03Declined Declining as part of general task cleanup in preparation for Developer Portal version 2 [19:17:24] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Developer-Portal: Migrate wikimedia/developer-portal to GitLab - https://phabricator.wikimedia.org/T343261#12309707 (10apaskulin) 05Declined→03Open Decided to move these tasks instead of declining for now. Sorry for the noise! [19:31:03] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [19:37:41] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Kafka-Infrastructure: beta - Run a single Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309778 (10Ottomata) I think we can keep kafka-logging; we don't mirror from it into jumbo (by default) like we did from kafka main. Also kafka lo... [19:37:50] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Kafka-Infrastructure: beta - Run a single event platform Kafka cluster in beta - https://phabricator.wikimedia.org/T437541#12309782 (10Ottomata) [20:02:09] 06Release-Engineering-Team (Doing 😎): [FY26-27 ST5.1.2] Automated supervision - https://phabricator.wikimedia.org/T427968#12309870 (10thcipriani) a:05bd808→03thcipriani Claiming the moving the task forward conch while bd808 is away. [21:01:12] (03PS2) 10Mhurd: jjb: run npm cache verify at most once a day per cache [integration/config] - 10https://gerrit.wikimedia.org/r/1338086 (https://phabricator.wikimedia.org/T437377) [21:01:47] (03CR) 10Mhurd: "Done in PS2. Kept a comment that it relies on rsync --archive preserving mtime, with the file content as the fallback." [integration/config] - 10https://gerrit.wikimedia.org/r/1338086 (https://phabricator.wikimedia.org/T437377) (owner: 10Mhurd) [21:19:30] (03PS1) 10Kimberly Sarabia: Add MobileFrontend to ReaderExperiment dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1339162 (https://phabricator.wikimedia.org/T434919) [22:09:14] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469#12310236 (10Southparkfan) 05In progress→03Resolved a:03Southparkfan https://gerrit.wikimedia.org/r/c/operations... [22:11:03] FIRING: InstanceDown: Project deployment-prep instance deployment-mwlog02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:11:11] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-mwlog02 is down - https://phabricator.wikimedia.org/T437656 (10wmcs-alerts) 03NEW [22:16:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-mwlog02 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:35:21] (03CR) 10Elliottetzkorn: [C:03+1] Add MobileFrontend to ReaderExperiment dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1339162 (https://phabricator.wikimedia.org/T434919) (owner: 10Kimberly Sarabia)