[05:04:57] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for **Daniel Kinzler** - https://phabricator.wikimedia.org/T304955 (10ArielGlenn) Daniel did indeed show up, get a refresher training, and do the deploy. All is well! [05:54:12] !log deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461 [05:54:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:54:14] T304461: Delete `growthexperiments-mentor-id` properties from user_properties - https://phabricator.wikimedia.org/T304461 [05:58:56] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) [06:01:50] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) Please hold off deploying to group 2 till I have verified that T305597 isn't a problem because this looks suspiciously related to... [06:07:30] !log deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204 [06:07:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:07:32] T304461: Delete `growthexperiments-mentor-id` properties from user_properties - https://phabricator.wikimedia.org/T304461 [06:52:58] good morning [07:00:22] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) I established that the bug predates the new parsoid release. So, I am going to remove it as a blocker and we'll debug and fix it s... [07:00:46] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) [07:03:10] PROBLEM - Host contint1001 is DOWN: PING CRITICAL - Packet loss = 100% [07:05:08] RECOVERY - Host contint1001 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [07:08:32] PROBLEM - Check systemd state on contint1001 is CRITICAL: CRITICAL - starting: Late bootup, before the job queue becomes idle for the first time, or one of the rescue targets are reached. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:13:48] RECOVERY - Check systemd state on contint1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:16:04] PROBLEM - Check systemd state on contint2001 is CRITICAL: CRITICAL - starting: Late bootup, before the job queue becomes idle for the first time, or one of the rescue targets are reached. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:16:33] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) [07:17:28] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) Sorry about the flip flop, but there is one other piece of the mystery that we need to resolve. Why is this bug now triggering on... [07:21:18] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10ArielGlenn) No training today because there were no patches, probably due to the gerrit maintenance window set for the same time. Let's reschedule this for next... [07:32:13] RECOVERY - Check systemd state on contint2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:44:20] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) **[[ https://office.wikimedia.org/w/index.php?title=User%3ASsastry%2FSandbox&type=revision&diff=310735&oldid=310734 | Test edit on... [07:51:25] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) [07:53:29] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10ssastry) Please look at T305597 for details, but I have verified that this issue doesn't impact non-private wikis, and even on private wiki... [08:08:37] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10hashar) [08:09:39] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T305213 (10hashar) [09:04:44] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10hashar) Now new MediaWiki log errors so far. It is rather quiet! [11:26:42] (03PS2) 10Jforrester: doc: [WikiLambda] Fix link typo and re-group/re-word other links [integration/docroot] - 10https://gerrit.wikimedia.org/r/777865 (owner: 10Krinkle) [11:26:49] (03CR) 10Jforrester: [C: 03+2] "Thanks." [integration/docroot] - 10https://gerrit.wikimedia.org/r/777865 (owner: 10Krinkle) [11:27:20] (03Merged) 10jenkins-bot: doc: [WikiLambda] Fix link typo and re-group/re-word other links [integration/docroot] - 10https://gerrit.wikimedia.org/r/777865 (owner: 10Krinkle) [11:35:26] (03CR) 10Zfilipin: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [11:40:53] https://danluu.com/monorepo/ [11:41:13] This isn't a secret, but it's good to see it publicly written about. [11:42:15] (He works at Twitter.) [12:18:45] 10Project-Admins: Create project tag for Moderator Tools - https://phabricator.wikimedia.org/T305145 (10Astuthiodit_1) Correct above staff profile is mine, is there a way I can edit this ticket such that I can change the Author to my Staff profile. Apologies for creating it under my volunteer profile, thank you [12:25:16] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10serviceops: Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10Joe) >>! In T303857#7818920, @dancy wrote: > I have confirmed that being in the `deployment` group will all... [12:41:15] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T305212 (10hashar) 05Open→03Resolved Looks like it is stable \o/ [12:41:53] 10Phabricator, 10Security-Team, 10SecTeam-Processed: How to categorise security tasks for beginners? - https://phabricator.wikimedia.org/T305579 (10Aklapper) (Note however the requirements listed on https://phabricator.wikimedia.org/project/profile/169/ ) [12:42:07] Krinkle: my stance on mono repo vs multiple repo or micro services vs monolith is the same I have regarding automobiles rear propulsion vs front tractions [12:42:30] they have pro and con which one needs to know about but in the end they do the same job :] [12:43:51] I am tempted to use an hybrid model which is a small independent repos with clear ownership, all integrated together via testing. The end results are repos known to work together and from that you can forge a stable monorepo [12:44:09] which is more or less what we do and ends up as mediawiki wmf branches [12:44:36] though it is really partial since the infra or micro services are not included [12:45:29] at least the good point with monorepos is you probably care less about back compatibility since a single change updates everything [12:46:38] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10MW-on-K8s, 10serviceops: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10Joe) [13:01:45] hashar: updating methods certainly was a lot easier in our monorepo in the past. [13:01:59] yup [13:02:11] But, the inconvenience also serves a purpose, it incentivises stability [13:02:22] Parsoid has the issue right now which is that when they change the parser output that causes extensions parser tests to break [13:02:27] Which is important for our scale now, inter team relations. Separation of concerns [13:02:36] Third parties as well [13:02:39] unless some backcompatibility / feature flagging is added [13:02:52] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10Joe) 05Open→03Resolved a:03Joe ` oblivian@deploy1002:~ $ sudo -u mwbuilder groups... [13:03:13] the multiple repositories definitely enforce clears ownership [13:03:28] (well when an owner ship is actually defined, but that is another topic) [13:03:47] what I am wondering is what tool Google / Facebook are using to manage their giant monorepo [13:03:54] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10Joe) 05Resolved→03Open a:05Joe→03None sorry for the mistake, I commented on the... [13:04:01] hashar: yeah.. though thats more testing related. We already get results from extensions in parser changes. We don't need monorepo to solve that anymore. [13:04:10] Those tests assert too many details [13:04:15] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Scap: Automate rebuild l10n cache for Train - https://phabricator.wikimedia.org/T245187 (10Joe) [13:04:29] We update them blindly more or less in most cases in the cross-repo use case [13:05:05] Could eg improve by better assertions (like the ones VE uses), or by disabling some extensions for some test cases. [13:05:33] afk a bit, having a coffee with james [13:05:35] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 2 others: Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10Joe) 05Open→03Resolved `lang=bash oblivian@deploy1002:~ $ sudo -u mwbuilder groups mwbuilder docker deplo... [13:05:36] Or if we want the fully integrated result stored then yeah we have to merge them at the same time. [13:07:19] But maybe showing the full diff in CI without failing on it is possible too. Like we do for config repos. We of course assume that because a monorepo makes it possible that therefore it is easy or useful. Arguably it would be better for devs if they didn't have to change the extension tests in the first place regardless where they are stored :) [13:07:27] (03CR) 10Awight: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/778287 (https://phabricator.wikimedia.org/T304813) (owner: 10Awight) [13:08:13] Krinkle: cscott told me they have some way for some back compatibility but I don't have the details [13:15:31] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Kartographer tests depend on FlaggedRevs [integration/config] - 10https://gerrit.wikimedia.org/r/778287 (https://phabricator.wikimedia.org/T304813) (owner: 10Awight) [13:20:18] Krinkle: are you a good person to ask about refreshLinks performance suddenly being awful [13:23:14] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10Joe) >>! In T299648#7829599, @dancy wrote: > Proposal for preparing for a transitiom fro... [13:33:13] RhinosF1: maybe, we could start at least; perhaps in #wikimedia-tech or #mediawiki-core for other's benefit [13:45:47] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10ItamarWMDE) Hello all, I am a staff engineer in WMDE, part of the Wikidata wikibase team. I would like to be able to create projects and boards so that I can track various tech relat... [14:35:00] 10Phabricator, 10SRE, 10SRE Observability (FY2021/2022-Q4), 10User-Ladsgroup: SRE access request to join #triagers for user lmata - https://phabricator.wikimedia.org/T305463 (10lmata) Thanks! [14:41:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: On CI Jenkins, audit worker labels and remove unused ones - https://phabricator.wikimedia.org/T225031 (10Jdlrobson) [14:41:28] thcipriani: coudl we make it part of the triage meeting to create the workboard column? One less thing to feel pressured under :) [14:41:53] also, do we have a runbook for the meeting? (I vaguelly recall it, but can't find it now) [14:41:57] going to unbreak the CI job for fresh :-\ [14:42:00] I messed up shell [14:47:31] (03PS1) 10Hashar: jjb: fix arguments passing in qemu-run.bash [integration/config] - 10https://gerrit.wikimedia.org/r/778297 [14:52:14] Krinkle: we don't have a runbook for that meeting. The closest thing is the demo you did that's currently NDA-only IIRC. [14:52:55] 10Phabricator, 10Project-Admins, 10DBA: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Krinkle) [14:53:04] 10Phabricator, 10Project-Admins, 10DBA, 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Krinkle) [14:53:15] thcipriani: ack, ok. [14:54:03] (but I'm pro-runbook for that meeting--raise the lottery-factor a bit :D) [15:04:32] 10Phabricator, 10Project-Admins, 10DBA, 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Marostegui) Not sure I get the rationale. The #mariadb-optimizer-bug is just about that. The optimizer bein... [15:15:29] 10Phabricator, 10Security-Team, 10SecTeam-Processed: How to categorise security tasks for beginners? - https://phabricator.wikimedia.org/T305579 (10sbassett) >>! In T305579#7838026, @Aklapper wrote: > (Note however the requirements listed on https://phabricator.wikimedia.org/project/profile/169/ ) As I much... [15:34:04] (03PS2) 10Jdlrobson: Add selenium daily job to Vector [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) [15:34:08] (03CR) 10Jdlrobson: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [15:53:06] (03CR) 10Zfilipin: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [15:56:47] (03CR) 10Zfilipin: "I have deployed the job, to see how it runs: https://integration.wikimedia.org/ci/view/selenium-daily/job/selenium-daily-beta-Vector/" [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [15:59:59] (03CR) 10Zfilipin: "It's failing with: Failed to launch the browser process!" [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [16:17:34] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Debug hosts sometimes Fatal error: "The UdpSocket to 127.0.0.1:10514 has been closed" - https://phabricator.wikimedia.org/T214734 (10Krinkle) [16:17:44] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Debug hosts sometimes Fatal error: "The UdpSocket to 127.0.0.1:10514 has been closed" - https://phabricator.wikimedia.org/T214734 (10Krinkle) @tstarling wrote a good summary of the issue at T285823: >... [16:23:53] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Debug hosts sometimes Fatal error: "The UdpSocket to 127.0.0.1:10514 has been closed" - https://phabricator.wikimedia.org/T214734 (10Krinkle) My hunch is that some code is (in)directly calling `logger->... [16:35:56] (03PS1) 10Ahmon Dancy: scap update-interwiki-cache: Don't overwrite interwiki.php in place [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) [16:38:08] (03PS2) 10Ahmon Dancy: scap update-interwiki-cache: Don't overwrite interwiki.php in place [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) [16:49:49] (03CR) 10Jdlrobson: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [16:53:58] 10Phabricator, 10Project-Admins, 10DBA, 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Ladsgroup) I understand the confusion but they are actually distinct issues. The most importantly, Slow-DB-Q... [17:18:06] 10Phabricator, 10Project-Admins, 10DBA, 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Krinkle) > The optimizer being buggy, which might result in weird query plans that might or might not produc... [17:18:22] 10Phabricator, 10Security-Team, 10SecTeam-Processed: How to categorise security tasks for beginners? - https://phabricator.wikimedia.org/T305579 (10Aklapper) I'd say it's also fine to add a clear sentence at the beginning of the task description which permissions are required to work on a task. [17:23:40] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10MediaWiki Train Development Environment: Add sample patches to train-dev deploy container - https://phabricator.wikimedia.org/T305647 (10dancy) [17:23:40] (03CR) 10Bernard Wang: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [17:30:31] (03CR) 10Hashar: [C: 03+2] "That has fixed fresh-test" [integration/config] - 10https://gerrit.wikimedia.org/r/778297 (owner: 10Hashar) [17:33:11] (03Merged) 10jenkins-bot: jjb: fix arguments passing in qemu-run.bash [integration/config] - 10https://gerrit.wikimedia.org/r/778297 (owner: 10Hashar) [17:37:17] (03CR) 10Ahmon Dancy: [C: 04-1] "Needs work" [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) (owner: 10Ahmon Dancy) [17:38:49] 10Phabricator, 10Project-Admins, 10DBA, 10Developer Productivity: Fold Phab tags for optimizer-bug and Slow-DB-Query tag under Wikimedia-database-error - https://phabricator.wikimedia.org/T305639 (10Marostegui) I personally don't use #Wikimedia-database-error, so I'd be fine with any decision. Same goes fo... [17:51:13] (03PS3) 10Ahmon Dancy: scap update-interwiki-cache: Don't overwrite interwiki.php in place [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) [17:51:52] (03CR) 10jerkins-bot: [V: 04-1] scap update-interwiki-cache: Don't overwrite interwiki.php in place [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) (owner: 10Ahmon Dancy) [17:53:04] (03PS4) 10Ahmon Dancy: scap update-interwiki-cache: Don't overwrite interwiki.php in place [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) [17:54:28] (03CR) 10Ahmon Dancy: "Good to go." [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) (owner: 10Ahmon Dancy) [18:43:12] (03CR) 10Krinkle: [C: 03+1] "Nice :)" [tools/scap] - 10https://gerrit.wikimedia.org/r/778318 (https://phabricator.wikimedia.org/T304529) (owner: 10Ahmon Dancy) [19:22:06] (03CR) 10Jdlrobson: "The job outputs files to docs/a11y. Is it possible to include those in the build as associated assets (similar to how we zip videos/images" [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [19:24:55] 10GitLab (CI & Job Runners), 10Airflow, 10Data-Engineering-Kanban: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10BTullis) Pausing this task, since we are not currently working on it. I think that it would still be useful to have a mee... [20:03:48] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10TheresNoTime) [20:05:49] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for TheresNoTime - https://phabricator.wikimedia.org/T305191 (10TheresNoTime) >>! In T305191#7837336, @ArielGlenn wrote: > No training today because there were no patches, probably due to the gerrit maintenance window set for... [20:42:54] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T305213 (10thcipriani) a:05brennen→03dancy [21:01:28] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10thcipriani) p:05Triage→03Medium [21:24:55] 10MediaWiki-Releasing, 10Security-Team, 10Security, 10user-sbassett: Write and send supplementary release announcement for extensions and skins with security patches (1.35.6/1.36.4/1.37.2) - https://phabricator.wikimedia.org/T297839 (10sbassett) [21:25:01] 10MediaWiki-Releasing, 10Security: Release MediaWiki 1.35.6/1.36.4/1.37.2 - https://phabricator.wikimedia.org/T297829 (10sbassett) [21:25:07] 10MediaWiki-Releasing, 10Security-Team, 10Security, 10user-sbassett: Write and send supplementary release announcement for extensions and skins with security patches (1.35.6/1.36.4/1.37.2) - https://phabricator.wikimedia.org/T297839 (10sbassett) 05Open→03Resolved [23:03:47] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T305213 (10thcipriani) [23:32:59] 10Release-Engineering-Team (Next), 10Security-Team: Determine a better way to track Wikimedia production security patches - https://phabricator.wikimedia.org/T295925 (10thcipriani) > 5. something else? I think I have a pretty simple proposal the falls into the "something else" camp. * * * Let's create a `/s...