[06:18:33] Short Gerrit maintenance starts in 15 minutes. It will take around 10 minutes. [06:32:06] Gerrit will restart now, should take less than 10 minutes [06:37:13] Project beta-code-update-eqiad build #458142: 04FAILURE in 4 min 13 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/458142/ [06:41:03] Gerrit maintenance done [06:43:16] Beta-code-update should recover anytime now [06:45:37] Yippee, build fixed! [06:45:37] Project beta-code-update-eqiad build #458143: 09FIXED in 2 min 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/458143/ [06:47:14] :) [06:49:07] :) [08:55:27] 10Scap, 10Data Products, 10Data-Engineering, 10Data-Platform-SRE: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 (10JAllemandou) >>! In T328472#9110768, @xcollazo wrote: > I would rephrase the problem as: Why do we need to keep generated artifacts inside our version cont... [09:19:09] 10GitLab (Pipeline Services Migration🐤), 10Research, 10collaboration-services, 10Patch-For-Review: Move research webpages to gitlab - https://phabricator.wikimedia.org/T334511 (10Jelto) a:05Jelto→03fkaelin Reassigning to @fkaelin > @fkaelin do you have experience deploying services to Kubernetes. The... [10:07:14] 10GitLab, 10Release-Engineering-Team, 10collaboration-services: Strange gitlab behavior - https://phabricator.wikimedia.org/T344156 (10Jelto) [10:10:23] 10Beta-Cluster-Infrastructure, 10Thumbor, 10Beta-Cluster-reproducible: There was a problem while contacting the thumbnailing service: [Errno -2] No address found - https://phabricator.wikimedia.org/T344605 (10hnowlan) There has definitely been an increase in the last few days based on some rather unscientifi... [10:23:53] 10Beta-Cluster-Infrastructure, 10Thumbor, 10Beta-Cluster-reproducible: There was a problem while contacting the thumbnailing service: [Errno -2] No address found - https://phabricator.wikimedia.org/T344605 (10hnowlan) Adding a hack to route requests for the old imagescaler hostname to a netcat session shows... [10:35:01] 10Beta-Cluster-Infrastructure, 10Thumbor, 10serviceops, 10Beta-Cluster-reproducible: deployment-prep needs a Thumbor instance - https://phabricator.wikimedia.org/T344605 (10hnowlan) [11:29:59] 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10noarave) >>! In T295601#7780925, @hashar wrote: > I supposed the code is somewhere... [11:39:44] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:44:03] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/14 JWT iss... [11:48:31] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:50:51] (03PS1) 10Hnowlan: Review access change [3d2png] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/951849 [12:52:45] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10kostajh) [12:53:50] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10kostajh) [12:56:00] 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10HasanAkgun_WMDE) >>! In T295601#9113046, @noarave wrote: >>>! In T295601#7780925,... [12:58:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/16 README:... [12:59:01] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10Tchanders) [13:00:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/16 README:... [13:13:36] (03CR) 10Hashar: [C: 03+2] "Jobs created:" [integration/config] - 10https://gerrit.wikimedia.org/r/951539 (https://phabricator.wikimedia.org/T336380) (owner: 10Hnowlan) [13:13:50] (03CR) 10Hashar: [C: 03+2] zuul: add media-analytics configuration [integration/config] - 10https://gerrit.wikimedia.org/r/951540 (https://phabricator.wikimedia.org/T336380) (owner: 10Hnowlan) [13:14:44] (03Merged) 10jenkins-bot: jjb: add media-analytics service [integration/config] - 10https://gerrit.wikimedia.org/r/951539 (https://phabricator.wikimedia.org/T336380) (owner: 10Hnowlan) [13:14:55] (03Merged) 10jenkins-bot: zuul: add media-analytics configuration [integration/config] - 10https://gerrit.wikimedia.org/r/951540 (https://phabricator.wikimedia.org/T336380) (owner: 10Hnowlan) [13:17:32] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/951540 # T336380 [13:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:17:34] T336380: AQS 2.0: Media Analytics Service - Deploy to staging and production - https://phabricator.wikimedia.org/T336380 [13:17:35] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/951540 (https://phabricator.wikimedia.org/T336380) (owner: 10Hnowlan) [13:22:00] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10jnuche) 05In progress→03Resolved The new id tokens changed the value in the issuer (`iss`) field which made th... [13:27:04] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10Jelto) [13:33:18] 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10HasanAkgun_WMDE) >>! In T295601#9113046, @noarave wrote: >>>! In T295601#7780925,... [14:46:37] 10Phabricator, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10brennen) [14:46:40] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation: Update [[mw:Phabricator/Code]] after migrating our upstream to Phorge - https://phabricator.wikimedia.org/T341808 (10brennen) [14:46:44] 10Phabricator: New post for Phabricating Phabricator - https://phabricator.wikimedia.org/T340223 (10brennen) [14:46:52] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10brennen) 05Stalled→03In progress Planning to go ahead in ~15 minutes. Deployment notes: https://et... [14:47:11] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) Deployment notes: https://etherpad.wikimedia.org/p/2023-08-23-phabricator [14:49:21] (03CR) 10Hashar: Provide a secondary database (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/949986 (https://phabricator.wikimedia.org/T342763) (owner: 10Urbanecm) [14:50:21] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) Where can we coordinate? [14:51:49] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10dancy) The problem is that the build is running inside of a container created by buildkitd while the mariadb service is running in a container created by the runne... [14:52:47] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) We'll be on `#wikimedia-operations` but also feel free to join https://meet.google.com/for-tnif-yku -... [15:03:12] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=00df6f2a-2b2a-4d40-81db-d959750d0622)... [15:04:14] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10kostajh) >>! In T344818#9113789, @dancy wrote: > The problem is that the build is running inside of a container created by buildkitd while the mariadb service is r... [15:05:38] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10dancy) Great work @jnuche ! [15:16:22] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Priority Backlog 📥): Archive the "wikimedia/production" repo - https://phabricator.wikimedia.org/T344763 (10Jdforrester-WMF) I've [[https://gerrit.wikimedia.org/r/admin/repos/wikimedia/production,general|set it to Read-Only]]; let's see if th... [15:23:01] 10Beta-Cluster-Infrastructure, 10User-ArielGlenn: Puppet broken on mediawiki instances in deployment-prep - https://phabricator.wikimedia.org/T344147 (10Urbanecm_WMF) This is caused by `wikifunctions` definition missing from `profile::mediawiki::mcrouter_wancache::shards` in `hieradata/cloud/eqiad1/deployment-... [15:30:59] brennen: If you need anything later today for phab/phorge feel free to ping me in IRC or on phone. [15:32:03] 10Phabricator (phabricator-next), 10CSS, 10Upstream: Incorrect height of class "phui-oi-frame" - https://phabricator.wikimedia.org/T307492 (10Aklapper) [15:32:18] jelto: thank you! [15:32:20] 10Phabricator (phabricator-next), 10Upstream: Add link to atom feed in each main blog page - https://phabricator.wikimedia.org/T205181 (10Aklapper) [15:32:50] 10Phabricator (2023-08-23), 10Upstream: Cannot create new Mock: RuntimeException: Illegal offset type in isset or empty - https://phabricator.wikimedia.org/T333089 (10Aklapper) 05Stalled→03Resolved This should be fixed by now. [15:33:03] ^ that one reminds me i wanted to do some work to add more feeds to phab generally, and now it seems actually practical. [15:33:04] 10Phabricator (phabricator-next), 10Upstream: "Unhandled Exception: Call to a member function getAppliedTransactionPHIDs() on bool" when viewing old Herald Transcript - https://phabricator.wikimedia.org/T294860 (10Aklapper) [15:33:06] 10Continuous-Integration-Infrastructure, 10GitLab: Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10Jdforrester-WMF) It sounds like you'd want something more like the #tool-ducttape tool that QTE are planning to re-build for general use on GitLab. [15:33:10] 10Phabricator (phabricator-next), 10CSS, 10Upstream: Upload popup overflows box with a long filename - https://phabricator.wikimedia.org/T316305 (10Aklapper) [15:33:35] 10Beta-Cluster-Infrastructure, 10User-ArielGlenn: Puppet broken on mediawiki instances in deployment-prep - https://phabricator.wikimedia.org/T344147 (10Jdforrester-WMF) It's fine for this to be blank and fall back to the main memcached servers. No need to provision a new set in Beta Cluster. [15:34:17] 10Phabricator (phabricator-next), 10Upstream: Give transparent embedded image files in Phabricator a checkered background - https://phabricator.wikimedia.org/T78644 (10Aklapper) [15:35:37] (03CR) 10Kosta Harlan: "I think we can abandon this one now" [integration/quibble] - 10https://gerrit.wikimedia.org/r/939634 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [15:38:01] 10Phabricator (2023-08-23), 10Security: Streamline process for uploading private files to public tasks - https://phabricator.wikimedia.org/T181317 (10Aklapper) 05Open→03Resolved Closing per last comment as phabricator.wikimedia.org now runs an upstream version from 2023 [15:40:08] 10Phabricator (2023-08-23): Less restrictive l image policy for drag and drop images for users or anons - https://phabricator.wikimedia.org/T333213 (10Aklapper) For testing after today's upgrade, dragging and dropping an image here: {F37622797} [15:41:02] 10Phabricator (2023-08-23): Less restrictive l image policy for drag and drop images for users or anons - https://phabricator.wikimedia.org/T333213 (10Aklapper) 05Open→03Resolved Image is visible when going to this task with my second (and standard) account. Resolving. [15:42:35] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Escape Goats🐐), 10Platform Engineering, 10Patch-For-Review: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10Eevans) >>! In T335691#9113942, @gerritbot wrote: > Change 913949 **merged** by Eevans: >... [15:44:13] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Escape Goats🐐), 10Platform Engineering, 10Patch-For-Review: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10dancy) Thanks! [15:44:17] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation: Update [[mw:Phabricator/Code]] after migrating our upstream to Phorge - https://phabricator.wikimedia.org/T341808 (10Aklapper) 05Stalled→03Open [15:44:33] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Escape Goats🐐), 10Platform Engineering, 10Patch-For-Review: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10dancy) [15:49:32] I've probably broken beta syncs for a little while; had to force merge a vendor patch to unblock some stuff [15:49:40] Should hopefully be resolved ASAP [15:57:39] 10Gitlab-Application-Security-Pipeline, 10Security, 10user-sbassett: osv-scanner: add support for finding nested lockfiles - https://phabricator.wikimedia.org/T342178 (10sbassett) 05Open→03Resolved p:05Triage→03Low [15:57:45] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [15:59:53] 10Scap, 10Data Products, 10Data-Engineering, 10Data-Platform-SRE: analytics/refinery: Stop using git-fat - https://phabricator.wikimedia.org/T328472 (10xcollazo) > Currently this is done through the refine-repo deployment (on stats machine and on HDFS). Before removing the artifacts from the refine repo o... [16:06:13] OMGFONTS [16:08:06] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): Captchas are broken in the beta cluster - https://phabricator.wikimedia.org/T344834 (10Urbanecm_WMF) [16:10:27] (03Abandoned) 10Hashar: (DO NOT SUBMIT) CI Mariadb error log with checkuser [integration/quibble] - 10https://gerrit.wikimedia.org/r/939634 (https://phabricator.wikimedia.org/T342088) (owner: 10Hashar) [16:16:58] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10Lucas_Werkmeister_WMDE) [16:18:21] Reedy: sync seems to be working, captchas not. if you have any ideas... :-D [16:18:22] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10Aklapper) [16:18:26] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Aklapper) [16:20:00] Project beta-update-databases-eqiad build #69703: 04FAILURE in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69703/ [16:22:24] 10Phabricator: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10Aklapper) Workaround: Clicking one of the tabs shows the content. [16:43:49] 10GitLab (Misc), 10Release-Engineering-Team (Radar), 10User-brennen: Investigate opendev.org hosting options and their costs, risks, & benefits - https://phabricator.wikimedia.org/T344372 (10brennen) [16:44:00] 10GitLab (Integrations), 10ExtensionDistributor: Add Gitlab Provider for ExtensionDistributor - https://phabricator.wikimedia.org/T340523 (10brennen) [16:44:11] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners): Allow GitLab CI containers to connect to services - https://phabricator.wikimedia.org/T344818 (10brennen) [16:50:14] 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥): Test Trusted Contributors group in devtools GitLab - https://phabricator.wikimedia.org/T344379 (10thcipriani) 05Open→03In progress p:05Triage→03Medium [16:50:37] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen): GitLab: find a way to search open merge-requests filtering out groups of people - https://phabricator.wikimedia.org/T344361 (10thcipriani) [16:50:43] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions, 10Patch-For-Review: Phabricator Project Reports have inaccurate counts for the age histogram - https://phabricator.wikimedia.org/T294998 (10brennen) [16:50:54] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen), 10Upstream: Disable GitLab snippets - https://phabricator.wikimedia.org/T332993 (10brennen) [16:50:58] 10Release-Engineering-Team, 10Wikimedia-Site-requests: Clarify whether config patches need reviews before being scheduled for deployment - https://phabricator.wikimedia.org/T344409 (10hashar) + #wikimedia-site-requests which is the tag used to update configuration. It has several watchers which I guess would b... [16:51:57] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap: scap backport sometimes crashes when running with two stacked patches - https://phabricator.wikimedia.org/T343967 (10thcipriani) [16:55:42] 10Gerrit, 10Release-Engineering-Team (Radar), 10Machine-Learning-Team: ML-Team will soon stop using LFS on Gerrit (for ORES deployment) - https://phabricator.wikimedia.org/T342765 (10thcipriani) 05Open→03Stalled Phabricator does the mirroring here so I think the things to fix on our side here: - [] Stop... [17:00:08] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Aklapper) Copying from **Etherpad: Notes for the 2023-08-23 Phabricator → Phorge deploy** **Background... [17:00:10] 10Release-Engineering-Team, 10sre-alert-triage: Alert triage: overdue critical alert - https://phabricator.wikimedia.org/T342755 (10thcipriani) Hrm. We get an email from the systemd timer for this, so the alert is probably not necessary. We're not very familiar with alertmanager. Can we just remove this alert? [17:03:30] 10Diffusion, 10Phabricator (Upstream), 10Upstream: Turn off 'blame' by default on Diffusion - https://phabricator.wikimedia.org/T198838 (10Aklapper) 05Open→03Declined No link shared; declining for the time being. [17:09:57] Yippee, build fixed! [17:09:57] Project beta-update-databases-eqiad build #69704: 09FIXED in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69704/ [17:22:05] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation: Update [[mw:Phabricator/Code]] after migrating our upstream to Phorge - https://phabricator.wikimedia.org/T341808 (10Aklapper) 05Open→03Resolved Done in https://www.mediawiki.org/w/index.php?title=Phabricator%2FCode&diff=607... [17:28:04] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Escape Goats🐐), 10Platform Engineering, 10Patch-For-Review: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10dancy) [18:55:10] If anyone has time, could you look at my access request for https://gitlab.wikimedia.org/repos/releng/dev-images ? Just need to update a dev image [18:57:31] ^^ ebernhardson is requesting access as well [19:12:43] inflatador: sure, one sec. :) [19:13:34] inflatador, ebernhardson: {{done}} [19:15:21] {◕ ◡ ◕} [19:15:24] Thanks brennen ! [19:46:38] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): Captchas are broken in the beta cluster - https://phabricator.wikimedia.org/T344834 (10Urbanecm_WMF) I've spent some time investigating this problem. The issue is that beta's swift server now does not have the right swift key in its `/etc/swift/... [19:49:10] 10Beta-Cluster-Infrastructure, 10SRE, 10SRE-swift-storage: Captchas are broken in the beta cluster - https://phabricator.wikimedia.org/T344834 (10Urbanecm_WMF) Adding some SRE tags. [19:49:57] 10Beta-Cluster-Infrastructure, 10SRE, 10SRE-swift-storage: Captchas are broken in the beta cluster - https://phabricator.wikimedia.org/T344834 (10RhinosF1) [20:23:00] 10Phabricator: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10Peachey88) [20:23:24] 10GitLab (Pipeline Services Migration🐤), 10Editing-team: Migrate Citoid/Zotero Pipeline Repos to GitLab - https://phabricator.wikimedia.org/T344736 (10Jdforrester-WMF) [20:25:05] 10GitLab (Pipeline Services Migration🐤), 10Platform Engineering: Migrate former Platform team service to GitLab - https://phabricator.wikimedia.org/T344739 (10Jdforrester-WMF) [20:25:08] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Escape Goats🐐), 10Platform Engineering: Migrate mediawiki/services/kask to GitLab - https://phabricator.wikimedia.org/T335691 (10Jdforrester-WMF) [20:29:32] TheresNoTime: any clue why is https://wikitech.wikimedia.org/w/index.php?diff=2101450 missing myself from the backport windows? [20:42:44] taavi: probably an old template [20:47:25] question on the dev-images workflow: https://gitlab.wikimedia.org/repos/releng/dev-images#deploying-images . Can/should I (SRE) run `fab deploy_devimages` to deploy a new dev image? Or is that something releng takes care of? [21:02:46] inflatador: it has been something we do, but you are welcome to do it. [21:03:00] i guess i think it will work as long as you are able to sudo to the correct user. [21:03:57] i like to make sure i've built changes successfully locally first [22:27:27] brennen I'll slow my roll for now and work on running it locally. I wouldn't even know which server to run fab on anyway ;P . [23:03:46] inflatador: it's just a script that sends some commands over ssh to contint [23:03:57] i'm out for a while after tomorrow, but i'd be happy to demo some time. [23:04:21] the Makefile should have what you need to build locally but the README could no doubt be better. :) [23:23:08] (03PS1) 10Krinkle: docker: Remove unused checksum files for Composer and Node.js [integration/quibble] - 10https://gerrit.wikimedia.org/r/952014 [23:48:29] 10Release-Engineering-Team (Priority Backlog 📥): Adopt gitlab-cloud-runner Digital Ocean based cluster as the default for untagged jobs - https://phabricator.wikimedia.org/T344874 (10dduvall) [23:50:32] (03PS1) 10Krinkle: Follows-up 8aabc136595 from 2018, which is interesting because Apache wasn't added until 2020 with 5439da36e5. [integration/quibble] - 10https://gerrit.wikimedia.org/r/952016 [23:55:25] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Adopt gitlab-cloud-runner Digital Ocean based cluster as the default for untagged jobs - https://phabricator.wikimedia.org/T344874 (10dduvall) p:05Triage→03Medium a:03dduvall