[03:42:00] (03CR) 10Hashar: [C: 03+2] dockerfiles: remove obsolete coverage utilities [integration/config] - 10https://gerrit.wikimedia.org/r/804587 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [03:44:23] (03Merged) 10jenkins-bot: dockerfiles: remove obsolete coverage utilities [integration/config] - 10https://gerrit.wikimedia.org/r/804587 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [03:51:36] Project mediawiki-core-phpmetrics-docker build #1296: 04FAILURE in 2 min 35 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/1296/ [04:06:57] I have managed to kill dockerd :( [04:20:51] Project mediawiki-core-phpmetrics-docker build #1297: 15ABORTED in 4 min 4 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/1297/ [04:26:25] Yippee, build fixed! [04:26:26] Project mediawiki-core-phpmetrics-docker build #1298: 09FIXED in 5 min 11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics-docker/1298/ [04:31:28] (03CR) 10Hashar: [C: 03+2] "I have wasted like 40 minutes trying to build this image to no availability. docker-pkg is somehow stuck when trying to pull the parent im" [integration/config] - 10https://gerrit.wikimedia.org/r/804587 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [04:31:50] cause of course [04:32:10] the first thing I hit at the beginning of a new week is DOCKER [04:33:05] !log Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore [04:33:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:40:07] looks like `docker-pkg` tries to download all known images of the parent image releng/quibble-buster bah [04:49:27] 10Release-Engineering-Team, 10docker-pkg: docker-pkg / docker downloads all versions of parent image upon building - https://phabricator.wikimedia.org/T310458 (10hashar) [04:49:42] (03CR) 10Hashar: [C: 03+2] dockerfiles: remove obsolete coverage utilities (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/804587 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [04:56:38] 10Release-Engineering-Team, 10docker-pkg: docker-pkg / docker downloads all versions of parent image upon building - https://phabricator.wikimedia.org/T310458 (10hashar) Once completed the build log shows the generated Dockerfile is using `FROM docker-registry.discovery.wmnet/releng/quibble-buster:1.4.5-s1` bu... [08:57:07] 10Phabricator (Upstream), 10Upstream: Add project-filtering to the Token Leader Board - https://phabricator.wikimedia.org/T302850 (10Samwilson) For a given project, if it were possible to see a summary of each task and the counts of each token types that it's been awarded, then it'd be possible to get a bit of... [10:30:21] 10Phabricator (Search): Change default search scope for Search field in upper right corner from Global to Open Tasks - https://phabricator.wikimedia.org/T252150 (10kostajh) >>! In T252150#7981239, @Dylsss wrote: > Not sure if this is a new feature, but when you change the filter it posts to /settings/adjust/?key... [10:31:36] 10Phabricator, 10Release-Engineering-Team (Yak Shaving 🐃đŸĒ’), 10User-MModell: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10kostajh) >>! In T215148#7981152, @Dylsss wrote: > Are people still interested in something like this? I was experimenti... [10:42:53] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, and 4 others: Move Termbox SSR for Beta Wikidata into deployment-prep project - https://phabricator.wikimedia.org/T304328 (10Lucas_Werkmeister_WMDE) Alright, sure. Though only the last of my changes actually creates a difference in... [13:17:43] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ale... [13:29:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ale... [14:18:49] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10Aklapper) For the records, as of today, the project tag #acl_WMF-FR can be edited by users DStrine, XenoRyet, greg, Ejegg. It would be great if someone changed the default to dis... [14:22:56] 10Diffusion, 10Toolforge: Renamed tool diffusion repository failed to be updated in toolsadmin.wikimedia.org - https://phabricator.wikimedia.org/T310493 (10Stang) [14:24:40] 10Diffusion, 10Toolforge: Renamed tool diffusion repository failed to be updated in toolsadmin.wikimedia.org - https://phabricator.wikimedia.org/T310493 (10Stang) [15:10:39] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10thcipriani) [15:38:34] (03CR) 10Ahmon Dancy: scap backport: deploy to mwdebug (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [15:47:45] (03CR) 10Ahmon Dancy: Add mwdebug container (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [16:02:06] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport --rollback command - https://phabricator.wikimedia.org/T287046 (10jeena) [16:02:29] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport --rollback command - https://phabricator.wikimedia.org/T287046 (10jeena) [16:04:10] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10jeena) 05Open→03Declined see T279322 [16:04:16] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10jeena) [16:06:43] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [16:07:02] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url: make gerrit plugin authorize with ssh - https://phabricator.wikimedia.org/T303755 (10jeena) 05Open→03Resolved a:03jeena [16:07:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:10:26] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10thcipriani) [16:20:57] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [16:32:59] CI gonna explode? [16:43:21] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10thcipriani) 05Declined→03Open [16:43:24] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10thcipriani) [16:46:59] Is anyone around to kill some jobs and ease the pressure on the CI infra? [16:47:23] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: scap backport --revert command - https://phabricator.wikimedia.org/T287046 (10thcipriani) [16:49:06] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ale... [16:54:19] Daimona: it will recover eventually [16:54:34] It's that time of day [16:54:35] Yeah, it's catching up now [16:55:14] Looks like a few are chained [16:57:01] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10thcipriani) [17:00:49] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Create alerts for automated Tuesday Train timer - https://phabricator.wikimedia.org/T310396 (10thcipriani) [17:03:45] 10Phabricator, 10Fundraising-Backlog: Please adapt Phabricator ACLs for "acl*WMF-FR" - https://phabricator.wikimedia.org/T246648 (10greg) Removed dstrine and other ex-staff: https://phabricator.wikimedia.org/project/manage/1070/#87066 Updated menu settings as well per above. [17:06:58] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10thcipriani) [17:20:28] Looks like it's hanging again [17:21:02] Well, it's monday, I guess CI has troubles starting the work week, too [17:30:35] Today has been a rubbish Monday [17:33:26] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) p:05Triage→03Medium [17:33:55] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) p:05Medium→03Unbreak! Making UBN! as a train blocker [17:34:55] 10Release-Engineering-Team, 10Web Team Visual Regression Framework, 10Readers-Web-Backlog (Kanbanana-FY-2021-22): Make visual regression tests run in CI (non-blocking) for the Vector repo - https://phabricator.wikimedia.org/T308194 (10Jdlrobson) p:05Triage→03Medium [17:36:19] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments, 10User-brennen: 1.39.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T308069 (10brennen) [17:38:07] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10RhinosF1) [17:38:21] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T308068 (10thcipriani) >>! In T308068#7995056, @JMeybohm wrote: > I see mcrouter get+gets increased significantly since the rollout to g... [17:40:26] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) [17:55:39] (03PS10) 10Ahmon Dancy: Add mwdebug container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:56:38] (03CR) 10Ahmon Dancy: Add mwdebug container (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [17:56:58] (03CR) 10Ahmon Dancy: [V: 03+1] "ps10 tested" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [18:02:53] (03CR) 10Jeena Huneidi: [C: 03+2] Add mwdebug container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [18:04:12] (03Merged) 10jenkins-bot: Add mwdebug container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi) [18:08:11] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10brennen) [18:11:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10brennen) [18:12:08] https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10&from=now-3h&to=now its getting there slowly :') [18:23:03] y'know what would be *nice*? The alert for ^ at say, 800 queued, auto-magically spinning up another `role::ci::slave::labs::docker` vm in horizon using the API, adding it to Jenkins & then binning it off once the queue is under 400 :3 [18:25:12] That would be nice [18:25:34] We might have more options when the move over to Gitlab is completed. [18:25:58] is all of CI moving over to Gitlab too? [18:26:20] s/gitlab/gitlab runners (or whatever they're called) [18:26:38] yeah, that is the ultimate plan. [18:27:30] whewww.. [18:59:44] 10Release-Engineering-Team (Radar), 10Release, 10Train Deployments: Investigate McRouter GET request spike from wmf.15 - https://phabricator.wikimedia.org/T310532 (10thcipriani) [19:36:44] 10Phabricator, 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10Reedy) [19:42:21] 10Phabricator, 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10TheresNoTime) Urgh, this took me waaay too long to figure out a while ago — great UX 😄 Does this help? {F35237326 size=full} [19:43:30] 10Phabricator, 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10TheresNoTime) Re-reading that, no probably not, as you say "manage //**a**// user" and you probably mean as a phab admin... ignore me! 🙃 [19:51:41] 10Phabricator, 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10Reedy) Yeah.. It seems to let you manage yourself easily... Not someone elses profile Manage is on the left hand side bar... But then it uses user IDs... So I could in theory find the ID to make the url m... [19:52:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:12:26] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [20:12:43] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:16:37] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [20:22:12] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [20:33:38] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [20:37:45] Daimona, TheresNoTime: it managed it ^ [20:37:53] \o/ [21:04:34] 10Phabricator (Upstream), 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10Aklapper) [21:18:31] 10Phabricator (Upstream), 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10RhinosF1) I think I mentioned this upstream before somewhere. Is the old forum still up? [21:19:29] 10Phabricator (Upstream), 10Mobile: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10RhinosF1) Nope :( [21:20:08] Reedy: uostream we're not very interested when they did support phab [21:21:21] There's the ban tool to disable [21:44:53] 10Phabricator (Upstream), 10Upstream: Add project-filtering to the Token Leader Board - https://phabricator.wikimedia.org/T302850 (10Aklapper) > If tokens don't have any meaning, what's the point of the feature? That's a good question to upstream - I don't know what's the point. :) (And I myself don't believe... [21:56:04] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10dancy) I think I heard @hashar express interest in this task. [22:04:19] !log cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up [22:04:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:34:34] 10Phabricator (Search): Change default search scope for Search field in upper right corner from Global to Open Tasks - https://phabricator.wikimedia.org/T252150 (10Aklapper) The search scope is "Current Application" if you're in a Phab task. The search scope is global if you're on e.g. the Phab front page. In ge...