[00:02:51] Krinkle: oh interesting. duesen and petr have been talking about how to handle per-extension config instances (an idea of config nodes iirc). i'm not totally up on that part of it, as i've mainly been focusing on the source loading parts, but i bet they will have thoughts [00:12:44] I think we can flatten it all, or possibly for editing convenience or size optimisation we could formalise the common prefix like ExtensionRegistry v2 does, but I don't think we're likely to see something like an extension that doesn't accept settings from the same source as the mw core config on the same wiki. I think we'd expect wmf-config and LocalSettings etc to be able to set any setting. [00:13:05] But yeah, now is a good time to think about want we want to do with the remnant ConfigRegistry [00:42:43] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator admin knowledge transfer - https://phabricator.wikimedia.org/T300693 (10brennen) [00:43:02] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator admin knowledge transfer - https://phabricator.wikimedia.org/T300693 (10brennen) 05Open→03In progress p:05Triage→03High [00:45:58] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10brennen) > Thinking out loud: how easy is it to move/rename and/or merge these groups if when the corresponding teams chang... [00:49:18] (03CR) 10NavinoEvans: [C: 03+2] Review access change [labs/tools/Isa] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/752276 (https://phabricator.wikimedia.org/T222291) (owner: 10Sebastian Berlin (WMSE)) [01:29:55] Project beta-update-databases-eqiad build #56349: 04FAILURE in 9 min 54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56349/ [01:53:10] looks like that just ran between the vendor patch landing and the core patch landing [02:30:06] Yippee, build fixed! [02:30:07] Project beta-update-databases-eqiad build #56350: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56350/ [04:24:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Patch-For-Review: contint1001 and contint2001 need a newer version of Docker installed - https://phabricator.wikimedia.org/T300682 (10Legoktm) [11:01:53] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: research - https://phabricator.wikimedia.org/T300074 (10Aklapper) @fkaelin: Thanks for the clarification! As @fab has no means of authentication anymore I've disabled that account. There's no recommendation to stick to e... [12:11:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [14:09:43] (03CR) 10NavinoEvans: [V: 03+2 C: 03+2] Review access change [labs/tools/Isa] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/752276 (https://phabricator.wikimedia.org/T222291) (owner: 10Sebastian Berlin (WMSE)) [14:41:55] 10Release-Engineering-Team (Radar), 10Scap, 10observability, 10Developer Productivity, 10Regression: Scap should set 'level' in Logstash messages - https://phabricator.wikimedia.org/T246793 (10mmodell) a:05mmodell→03None [16:12:07] !log Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta [16:12:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:30] hi everyone :) [16:14:42] jnuche: hiya! welcome [16:14:48] jnuche: hi! [16:14:54] thx! [16:15:00] welcome to the channel, to the team [16:15:20] for lurkers/folks who I haven't already told: jnuche is our new relenger :) [16:15:35] thanks Dan, looking forward to joining the fun [16:21:59] oh: you're voiced, it just didn't give me any confirmation :) [16:23:17] anyway: I now dub thee voiced, jnuche [16:26:45] I shall endeavor to make myself worthy, my lord [16:26:49] (thanks) [16:28:42] <3 [16:37:32] could a friendly phab admin delete a spam comment for me? https://phabricator.wikimedia.org/T205378#7671518 [16:39:18] (03PS1) 10Ahmon Dancy: scap sync-*: Avoid warnings abouut empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/759270 [16:40:36] (03CR) 10jerkins-bot: [V: 04-1] scap sync-*: Avoid warnings abouut empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/759270 (owner: 10Ahmon Dancy) [16:41:05] (03PS2) 10Ahmon Dancy: scap sync-*: Avoid warnings about empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/759270 [16:46:17] (03CR) 10Ahmon Dancy: [C: 03+2] scap sync-*: Avoid warnings about empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/759270 (owner: 10Ahmon Dancy) [16:46:59] (03Merged) 10jenkins-bot: scap sync-*: Avoid warnings about empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/759270 (owner: 10Ahmon Dancy) [16:50:27] !log Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta. [16:50:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:52:18] (03CR) 10Dduvall: [C: 04-1] "I like the change and the implementation looks good. I just wonder if we should embrace polymorphism since we're in JVM land already." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756029 (owner: 10Ahmon Dancy) [16:53:45] (03CR) 10Ahmon Dancy: Add PipelineRunner.runv2() (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756029 (owner: 10Ahmon Dancy) [16:56:42] (03CR) 10Dduvall: [C: 04-1] PipelineRunner.run(): Add removeContainer parameter (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/755830 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [16:59:25] jnuche: looks like you figured out the irc madness :D [17:00:43] (03CR) 10Dduvall: "A small non-blocking nit. Looks good once the dependency is merged." [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [17:18:43] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10Nikerabbit) I am not sure how Wikimedia wikis use `wgUseInstantCommons`, but maybe someone should check if group0 wikis a... [17:20:05] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) I don't believe wikimedia wikis use instant commons [17:20:56] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10Zabe) It is used on labswiki and labtestwiki. [17:21:12] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) And we can see the magic that is in place via https://www.mediawiki.org/wiki/MediaWiki seems to be working [17:22:04] zabe: wikitech is showing .19 [17:23:05] yep, wikitech is a group1 wiki. So InstantCommons might break when wmf.20 is pushed to group1 wikis. [17:24:23] yeah, https://labtestwikitech.wikimedia.org/wiki/User:Majavah_clouddev is broken [17:24:48] Is it worth blocking [17:24:55] Given it only affects wikitech [17:25:41] I'm going to let someone else make that call [17:26:41] I let WMCS folks now so they can scream if they would like it blocked [17:30:07] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) Most content on wikitech is apparently locally uploaded so it should be fine to proceed [17:30:33] dancy, brennen: fyi, ^ but don't take my word yet [17:37:32] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) [17:38:01] I moved it to a blocked [17:38:12] I think my Q to WMCS caused confusion [17:38:29] There's ~650 usages on wikitech including their logo [17:40:40] Amir1: it's a patch you +2'd [17:40:42] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/757120 [17:47:42] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10Nikerabbit) https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page shows a bunch of red image links. https://codesearch.wm... [17:48:26] I will debug it, let me see if I can spot the fix [17:49:18] Cool :) [18:02:24] (03CR) 10Ahmon Dancy: jjb/service-pipeline.groovy: Always remove test container (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [18:02:27] (03PS2) 10Ahmon Dancy: jjb/service-pipeline.groovy: Always remove test container [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) [18:09:49] (03PS3) 10Ahmon Dancy: PipelineRunner.run(): Add removeContainer parameter [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/755830 (https://phabricator.wikimedia.org/T290608) [18:09:51] (03PS2) 10Ahmon Dancy: Add PipelineRunner.run() method which takes a Map argument. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756029 [18:10:32] (03CR) 10Ahmon Dancy: PipelineRunner.run(): Add removeContainer parameter (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/755830 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [18:12:28] (03CR) 10Dduvall: [C: 03+2] PipelineRunner.run(): Add removeContainer parameter [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/755830 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [18:12:57] (03CR) 10Dduvall: [C: 03+2] Add PipelineRunner.run() method which takes a Map argument. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756029 (owner: 10Ahmon Dancy) [18:13:04] (03Merged) 10jenkins-bot: PipelineRunner.run(): Add removeContainer parameter [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/755830 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [18:13:34] (03Merged) 10jenkins-bot: Add PipelineRunner.run() method which takes a Map argument. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/756029 (owner: 10Ahmon Dancy) [18:16:58] (03PS3) 10Ahmon Dancy: jjb/service-pipeline.groovy: Always remove test container [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) [18:24:46] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) [18:25:00] (03CR) 10Dduvall: [C: 03+1] "Looks good. I'll update the jobs and then merge." [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [18:25:30] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10RhinosF1) Train is no longer blocked [18:26:08] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10dancy) Thanks everyone. [18:26:18] Np [18:32:47] (03PS1) 10Cwhite: bump dashboards version to 1.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/759293 [18:34:00] (03PS2) 10Cwhite: bump dashboards version to 1.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/759293 (https://phabricator.wikimedia.org/T299168) [19:11:24] (03CR) 10Dduvall: [C: 03+2] "Ahmon updated the jobs because my python 2.7 installation is all kinds of messed up right now. Merging." [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [19:13:44] (03Merged) 10jenkins-bot: jjb/service-pipeline.groovy: Always remove test container [integration/config] - 10https://gerrit.wikimedia.org/r/756030 (https://phabricator.wikimedia.org/T290608) (owner: 10Ahmon Dancy) [20:05:27] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10gmodena) Hey @brennen, This is great! Many thanks. Could you help me understand how repos and people namespaces differ f... [20:24:28] (03PS1) 10Ahmon Dancy: ProgressReporter: Report number of in-flight operations [tools/scap] - 10https://gerrit.wikimedia.org/r/759326 [21:39:53] (03PS1) 10Ahmon Dancy: Release 4.3.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/759342 [21:43:21] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.3.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/759342 (owner: 10Ahmon Dancy) [21:45:40] (03Merged) 10jenkins-bot: Release 4.3.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/759342 (owner: 10Ahmon Dancy) [21:51:19] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.3.0 - https://phabricator.wikimedia.org/T300804 (10dancy) [22:14:02] Krinkle: do you have a link to the logstash dashboard you shared in the postmortem? [22:14:29] dduvall: https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors [22:14:42] https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors [22:14:56] with the time ranges you selected? [22:15:28] ah, I don't have that one anymore. It was Mon-Fri of the wmf.16 and .17 weeks, and then clicking on the branch that's new that week. [22:15:44] ah, i see [22:15:49] I had two tabs open [22:16:05] ok, yeah. it didn't seem to reflect what i had seen during train, but that makes sense [22:16:32] during train, i and i think others usually narrow it down to ~ 15-30 minutes [22:19:11] ack, I didn't mean to imply this is how I review logstash during a train deploy, I don't. [22:20:31] I would view just mediawiki-wiki over the last 1h, and leave it unfiltered, maybe ocasionally filtering on the new-branch only to look at the ones that stand out there and then flip back to see if they are in the previous version as well. [22:20:37] mediawiki-errors* [22:31:36] Krinkle: no, i didn't think you'd implied that at all. i didn't recall seeing that deadlock error at all during train and i'm trying to figure out why. if it just got buried in the dashboard or what [22:32:52] the first instance of it i see is at Jan 14, 2022 @ 01:22:37.753 [22:34:17] but the group1 promotion that resulted in the rollback was the day before [22:34:47] i think that was a separate issue [22:35:18] Jan 13, 2022 @ 20:28 Deadlock found when trying to get lock; try restarting transaction (db1157) Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites [22:35:42] [2022-01-13T20:07:11Z] rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.17 refs T293958 [22:35:43] T293958: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 [22:35:59] about 20min after group0 went out [22:37:04] https://logstash.wikimedia.org/goto/7b2c9379a67c73f6e51cadb5caa79a58 [22:37:38] These instances have less boilerplate for some reason, so the normalized_message got trimmed in the variable part, so it didn't surface as "top" aggregated by count. [22:37:58] it also includes the dbhost name (e.g. db1118) for some reason [22:38:05] further spreading it out between other errors [22:41:16] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) Alright! the TLS cert issue should be fixed with the above ^ at least in this sense: ` Feb 02 22:36:55 gitlab-prod-1001 systemd[1]: certbot.service: Suc... [22:41:24] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10sbassett) I'm to call [[ https://gitlab.wikimedia.org/repos/security/gitlab-ci-security-templates/-/blob/65b8e1d0def28d... [22:41:41] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett) [22:42:09] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10sbassett) 05Open→03Resolved [22:42:40] Ah, this was the second roll out. The first roll out was Jan 11 [22:42:57] Krinkle: that it didn't surface makes sense. but that still seems to be another issue [22:43:10] the delete that caused corruption run against pagelinks, right? [22:43:17] *ran* against [22:43:54] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [22:44:02] Yes and no. The problem was in factorConds which is used by all of the new Linktable::doWrite calls, for all the link tables. [22:44:38] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [22:44:43] yes, but i don't see any missing parens in those statements [22:44:45] But the missing parenthesis was only a problem when there are three conditios in the WHERE clause. When there's two, it did the right thing still without the extra paran pair. [22:44:58] page_props has fewer columsn so it didn't cause corruption [22:45:08] but it was still subject to the same deletion churn and deadlocks [22:45:16] just without corruption [22:45:18] right. so that condition didn't even surface the bug quite yet [22:45:33] but it created more noise [22:46:08] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) this is not puppet-related anymore. Manually installing gitlab-ce also does this: ` Checking PostgreSQL executables: OK Checking if a newer PostgreSQL... [22:46:34] Thre's a few of these from Jan 12: [22:46:35] which is really the trouble with running the train most of the time. there's way too much noise and it's not easily parseable by RelEngers [22:46:46] > Jan 12, 2022 @ 20:23 Lock wait timeout exceeded; try restarting transaction (db1109) Function: MediaWiki\Deferred\LinksUpdate\LinksTable::doWrites Query: INSERT IGNORE INTO `pagelinks` (pl_from_namespace,pl [22:47:03] yeah, those were the ones i called out in the task [22:47:12] right, that as a good call, it was new that branch [22:47:33] but low frequency disproportionally small on small wikis because it's less likely to contend on small wikis in the first place [22:47:55] so the mental ratio of something that's an issue on small wikis translating to big wikis isn't the usual volume magnitude, but a lot more [22:47:55] but sadly it wasn't enough and i can't say i would have always called something like that out [22:48:06] right, not with current tolerance levels [22:48:20] it was only because the risky patch mentioned db performance that i thought i _might_ be relevant [22:49:58] speaking of db perf, the db perf warnigs (not fatals) aren't on the exception channels so anything that does come through here is a fatal error for a web request, so short of something that seems attributable to opcache randomness or slow parser timeouts, I do think we could work towards some kind of zero tolerance even for db errors if they are confirmable as 'new'. [22:50:25] my long term thinking re: train is: 1) make it easier (pretty much done thanks to dancy); 2) do it more frequently (start full group0/group1/all train deployment successions Tues-Thurs); 3) hand it off to folks that know the code being deployed [22:50:52] i think with #2 it stops looking like the train and looks closer to backport deployment [22:51:52] ack, that sounds good to me and aligns with my thinking as well. [22:52:26] Krinkle: re: perf, yeah, i know. i've tried to add some more perf dashboards to my train process but it just becomes too much to take in. that's why i'm wondering if we could figure out a good top level dashboard for all domains and have each link in to more specific dashboards with details [22:52:34] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [22:54:43] I've proposed cutting the train branches manually (push button that is, not more work, but not delayed longer than needed, e.g. after wrap up on Thursday, start the next one etc. For us to be responsible with non-fixed deployment dates though, we need to slightly more formalise our weekly cadence, hence my thinking in the past toward a "checklist" (aggregate branch changelog by component/owner team, require an ACK from relevant dev/qa [22:54:43] engineer to go to group1, then again for group2). This would mean we can go to the next group when everybody's done, possibly much sooner than 24 hours, e.g. after 8-10 hours depending on timezones and scheduling. [22:56:21] interesting. i've had an idea of a formalized deployment console that sounds pretty congruent with that proposal [22:58:47] and of course all this becomes easier when you don't need a lot of widespread and evolving knowledge of things to "ignore", i.e. lower base line noise, so that you can query "the" erros from group0 and expect little to nothing and go through them with the people on the task without even looking at whether they are new or not, more an attitude of this is happening and it shouldn't be, where are we on the fix, should it block/not, and [22:58:47] then proceed when team sign off and each aggregate entry is acknowledged in some way. We wouldn't need to remember many filters from week to week either. For indivudal team dashboards we already don't use filters that much since for a given team there's usually only 1 or 2 that they'd know about. [22:59:18] the basic idea is that a deployer (dev) would open the console and checkin to "claim a lane" for deployment. lanes would be a mutex basically and a 1:* abstraction above repos so, for example, all mediawiki/* repos could belong to the MediaWiki lane. at this point, for mediawiki, the branch could be prepped and synced. when the deployer clicks "deploy" it would scap sync-wikiversions [22:59:50] there's a big rollback button until the deployer has deemed the deployment a success and must explicitly release the lane [23:00:06] at which point, there's no more rollback [23:00:43] we could also have top-level metrics displayed in the console if that makes sense, or just a link to logstash [23:01:19] high level diff information, etc. one day, risk assessment :) dreaming now... [23:10:45] Wikilove for this conversation. [23:16:33] 10Project-Admins, 10Data-Engineering: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) * Changed H126 by replacing #Analytics with #Data-Engineering and removing some archived project tags, so that rule is now: ** When _all_ of these conditions are met: *** Project tags include a... [23:22:58] 10GitLab, 10SRE: gitlab: enable IPv6 for https - https://phabricator.wikimedia.org/T300816 (10Dzahn) [23:23:53] 10GitLab (Infrastructure), 10SRE: gitlab: enable IPv6 for https - https://phabricator.wikimedia.org/T300816 (10Dzahn) [23:24:22] 10GitLab (Infrastructure), 10SRE, 10serviceops: gitlab: enable IPv6 for https - https://phabricator.wikimedia.org/T300816 (10Dzahn) [23:24:32] 10Project-Admins, 10Data-Engineering: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) For open tasks tagged with #Analytics but not with `#Data-Engineering*`, see https://phabricator.wikimedia.org/maniphest/query/jex5HoXYMhEN/#R . There are some open tasks which only have an #An...