[03:12:32] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10andrea.denisse) a:03andrea.denisse [04:07:56] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T330207 (10brennen) [09:02:22] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Urbanecm_WMF) >>! In T706#8711326, @MShilova_WMF wrote: > Hi There, I am a Sr. TPgM supporting the #Growth-Team. I need to modify two Phab task templates: > # [[ https://phabricat... [09:04:47] 10Scap: Improve behavior around global Scal lock + communicate changes - https://phabricator.wikimedia.org/T330756 (10Clement_Goubert) >>! In T330756#8710370, @dancy wrote: > I searched wikitech for references to `/var/lock/scap-global-lock`. These two pages need to be updated: > https://wikitech.wikimedia.org/w... [09:08:23] (03CR) 10Hashar: [C: 04-1] "It is not possible in JJB, it doesn't not process a tree of variables recursively. I have opened a discussion at https://groups.google.com" [integration/config] - 10https://gerrit.wikimedia.org/r/900311 (owner: 10Hashar) [09:09:44] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) a:03zeljkofilipin [09:09:50] 10Deployments, 10Release-Engineering-Team (Radar), 10SRE-Sprint-Week-Sustainability-March2023, 10serviceops-radar, and 2 others: Remove provisioning for 'mwscript', 'foreachwikiindblist' etc from deployment host - https://phabricator.wikimedia.org/T253822 (10Joe) [09:18:33] 10Phabricator, 10Release-Engineering-Team (Radar), 10SRE-Sprint-Week-Sustainability-March2023, 10serviceops-collab, and 2 others: Phabricator: Unable to view tasks in DB read-only mode - https://phabricator.wikimedia.org/T313879 (10Joe) [09:22:09] 10Phabricator, 10SRE, 10Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228 (10Aklapper) a:05mmodell→03None @mmodell: Removing task assignee as this open task has been assigned for more than two years - See the email sent to task assi... [09:23:09] 10Beta-Cluster-Infrastructure, 10SRE-OnFire: Add basic alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695 (10Joe) Incident followup tags should be reserved for production issues. [09:29:48] 10Beta-Cluster-Infrastructure: Add basic alerting to the Beta Cluster - https://phabricator.wikimedia.org/T315695 (10Volans) [09:30:42] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Rebase & merge or re-cherry-pick 668701 on deployment-puppetmaster04 - https://phabricator.wikimedia.org/T315395 (10Volans) Incident followup tags should be reserved for production issues. [09:34:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10SRE-OnFire, 10User-Ryasmeen, 10Wikimedia-Incident: Beta cluster Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T315350 (10taavi) [09:34:24] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Rebase & merge or re-cherry-pick 668701 on deployment-puppetmaster04 - https://phabricator.wikimedia.org/T315395 (10taavi) 05Open→03Declined I'm pretty sure this is not needed anymore after https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-pupp... [09:57:42] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) 05Open→03Resolved a:03dancy >>! In T331378#8673893, @dancy wrote: >>>! In T331378#8671563, @hashar wrote: >> find /srv/mediawiki-s... [10:08:15] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Move phabricator/deployments repository from Differential to Gerrit - https://phabricator.wikimedia.org/T321380 (10hashar) 05Declined→03Open >>! In T321380#8712149, @brennen wrote: > All but a couple of these have moved to GitLab at this point: >... [10:20:37] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10jnuche) @Clement_Goubert I will deploy the a new release with these changes this afternoon. There are two things: * `--stop-before-syn... [10:34:06] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10Clement_Goubert) >>! In T332187#8713994, @jnuche wrote: > @Clement_Goubert I will deploy a new release with these changes this afternoon.... [10:37:13] 10Release-Engineering-Team (Radar), 10SRE, 10observability, 10Epic, 10Sustainability (Incident Followup): Tracking: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942 (10Joe) [10:37:37] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10Clement_Goubert) Documentation at https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#How_to_deploy_MediaWiki_on_Kubernetes updated. [10:42:17] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 10SRE-OnFire, 10SRE-Sprint-Week-Sustainability-March2023, and 3 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10Volans) a:03Volans [10:51:21] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) @Krinkle apologies for the late reply, I almost forgot about this, but I was recently reminded about it because... [11:08:53] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) While debugging T332393 I've created a patch ([[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/866386 | 866... [11:19:02] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) I can't reproduce that error consistently on my machine. The only way I've noticed on how to reproduce it is to... [11:19:48] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) a:05zeljkofilipin→03None [11:26:31] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10kostajh) @zeljkofilipin I see in https://github.com/webdriverio/webdriverio/issues/8279 some reports about webdriver working i... [11:58:56] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10jnuche) > Documentation at https://wikitech.wikimedia.org/wiki/MediaWiki_On_Kubernetes#How_to_deploy_MediaWiki_on_Kubernetes updated. Th... [13:51:03] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob - https://phabricator.wikimedia.org/T325594 (10TheresNoTime) 05Resolved→03Open Nope, [[ https://beta-logs.wmcloud.org/app/dashboards#/doc/5f0c9be0-0b6f-11ec-9cde-3f4490e09a26/... [14:23:17] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10jnuche) 05Open→03Resolved Deployed now. [14:35:10] (03PS1) 10Ayounsi: Zuul: [operations/grafana-grizzly] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/901598 [14:37:20] (03PS2) 10Ayounsi: Zuul: [operations/grafana-grizzly] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) [14:38:48] (03CR) 10CI reject: [V: 04-1] Zuul: [operations/grafana-grizzly] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [15:08:20] 10Continuous-Integration-Infrastructure, 10OOUI, 10Patch-For-Review, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10hashar) [15:08:22] 10Continuous-Integration-Infrastructure, 10serviceops-collab: migrate doc hosts to bullseye - https://phabricator.wikimedia.org/T319477 (10hashar) [15:09:07] 10Continuous-Integration-Infrastructure, 10OOUI, 10Patch-For-Review, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10hashar) a:03hashar >>! In T322357#8468491, @hashar wrote: > doc.wikimedia.org is hosted on doc1002.eqiad.wmnet which has Buster. It gets... [15:47:37] (03PS3) 10Ayounsi: Zuul: [operations/grafana-grizzly] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) [17:04:06] (03CR) 10Hashar: Zuul: [operations/grafana-grizzly] Add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [17:08:59] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10brennen) [17:09:14] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: GitLab comments should come from a CodeReviewBot instead of gerritbot - https://phabricator.wikimedia.org/T327424 (10brennen) 05In progress→03Stalled [17:20:37] maintenance-disconnect-full-disks build 475560 integration-agent-docker-1026 (/: 28%, /srv: 97%, /var/lib/docker: 46%): OFFLINE due to disk space [17:22:11] (03CR) 10JMeybohm: Zuul: [operations/grafana-grizzly] Add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [17:25:35] maintenance-disconnect-full-disks build 475561 integration-agent-docker-1026 (/: 28%, /srv: 34%, /var/lib/docker: 45%): RECOVERY disk space OK [20:24:13] (03CR) 10Umherirrender: [C: 03+1] build: Updating eslint-config-wikimedia to 0.24.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/901696 (owner: 10Libraryupgrader) [21:25:50] (03CR) 10Hashar: Zuul: [operations/grafana-grizzly] Add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [21:50:54] 10Gerrit, 10Release-Engineering-Team (Priority Backlog 📥), 10SRE-OnFire, 10SRE-Sprint-Week-Sustainability-March2023, and 3 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10Dzahn) @hashar @Jelto Do you have any comments on the question by Volans above? [23:26:02] 10Continuous-Integration-Config: JJB: audit-resources job keeps running - https://phabricator.wikimedia.org/T332749 (10MarcoAurelio)