[00:01:35] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10BilalShirwani) [00:02:36] 10Continuous-Integration-Config, 10Shellbox: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6 - https://phabricator.wikimedia.org/T298008 (10BilalShirwani) [00:03:16] 10Project-Admins: Create OgvJS project as subproject of TimedMediaHandler - https://phabricator.wikimedia.org/T298016 (10BilalShirwani) [00:05:34] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10BilalShirwani) [00:05:50] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10BilalShirwani) [00:06:33] twentyafterfour_: if you're about, a vandal did a huge batch of merging things into T298058 that needs to be undone :/ [00:06:34] T298058: [Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools - https://phabricator.wikimedia.org/T298058 [00:07:01] so huge that apparently even though the account is blocked phab is still processing things they merged in [00:07:23] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10JJMC89) 05duplicate→03Open [00:11:06] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T293957 (10JJMC89) [00:11:47] 10Continuous-Integration-Config, 10phan: Add a phan job to gated extensions/skins and core testing each change with all of them - https://phabricator.wikimedia.org/T298053 (10JJMC89) 05duplicate→03Open [00:16:28] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10JJMC89) 05duplicate→03Open [00:16:30] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10JJMC89) [00:16:34] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10Zabe) 05duplicate→03Open [00:17:24] 10Continuous-Integration-Config, 10Shellbox: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6 - https://phabricator.wikimedia.org/T298008 (10JJMC89) 05duplicate→03Open [00:18:13] 10Project-Admins: Create OgvJS project as subproject of TimedMediaHandler - https://phabricator.wikimedia.org/T298016 (10JJMC89) 05duplicate→03Open [00:22:10] twentyafterfour_: looks like manual cleanup got it all this time [00:52:51] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) 05Open→03In progress [00:53:49] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar, 10User-revi: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) a:03revi [06:34:14] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar, 10User-revi: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10revi) 05In progress→03Resolved OOOOOOOOHHHHHHHHHH Was doing some cleanups for my now-defunct server (wh... [06:38:00] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar, 10User-revi: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10revi) (To be honest, I am very surprised that someone other than me was using the redirection, lol) [08:50:30] jayme: that charts patch reminds me: with helm3 migration mostly complete, is there anything else blocking upgrades to a more recent k8s cluster? can I help with any of that? [08:55:08] majavah: I'd need to revisit in detail, but I guess it's "only" the OS upgrade and potential docker upgrade that we're missing now. Or at least that is what we planned to do prior to the actual k8s upgrade [08:56:51] but there is no detailed plan for that layes out yet. Unfortunately serviceops is a bit understaffed so this might go slower as we would like it to :/ [09:13:03] 10Phabricator: Make sure anti-vandalism features are up to snuff - https://phabricator.wikimedia.org/T84 (10Aklapper) [09:26:04] 10Continuous-Integration-Config, 10Shellbox: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6 - https://phabricator.wikimedia.org/T298008 (10Legoktm) We can switch to the composer-package image I think. [09:39:06] 10GitLab (Project Migration), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10User-brennen, 10Voice & Tone: Delete temporary main branch in mediawiki/core.git until rename from master to main is ready - https://phabricator.wikimedia.org/T296205 (10Lucas_Werkmeister_WMDE) Thanks :) [10:48:54] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10Jdforrester-WMF) I think it's pretty seriously unlikely that we're going to want to support EOL node environments just for release branches. It's more likely that we'd just drop test... [10:54:35] !log Kick beta-scap-sync-world [10:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:05:00] (03CR) 10Jforrester: [C: 03+2] zuul: Make OAuth depend on AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/748788 (owner: 10Majavah) [11:07:02] (03Merged) 10jenkins-bot: zuul: Make OAuth depend on AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/748788 (owner: 10Majavah) [11:16:49] !log zuul: Make OAuth depend on AbuseFilter [11:16:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:21:10] hmm, that doesn't seem to work [11:38:03] majavah: As in it doesn't co-load the repo? [11:38:38] 00:00:06.895 INFO:quibble.cmd:Projects: mediawiki/core, mediawiki/extensions/Echo, mediawiki/extensions/OAuth, mediawiki/skins/Vector, mediawiki/vendor [11:38:40] Hmm, indeed. [11:39:19] * James_F re-syncs zuul config. [12:32:56] majavah: Works now. [13:18:01] 10Continuous-Integration-Config, 10Shellbox: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6 - https://phabricator.wikimedia.org/T298008 (10Daimona) 05Open→03Resolved a:03Legoktm Seems to be fixed now, thanks! [13:32:50] 10Continuous-Integration-Config, 10phan: Add a phan job to gated extensions/skins and core testing each change with all of them - https://phabricator.wikimedia.org/T298053 (10Daimona) I'm not sure if it's doable... The main potential issues I can think of are: - It might need A LOT of time and memory - We need... [13:51:10] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) I've done some more testing around managing the GitLab Runner configuration file `config.toml` using... [15:08:59] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10sbassett) [15:49:03] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10Jdlrobson) That was my thought too. Maybe as a starting point, we could drop CI from branches 1releases back, e.g. since 1.38 is now what we are currently working on, we would only s... [15:49:04] I'm running perf on two VMs, if it misbehaves, let me know [16:01:07] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10serviceops, 10Upstream: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Jelto) >>! In T296944#7561721, @brennen wrote: >> I am tempted t... [17:05:54] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10serviceops, 10Upstream: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Dzahn) >>! In T296944#7583225, @Jelto wrote: > I also couldn't f... [17:10:21] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar, 10User-revi: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) @revi Thank you very much! > (To be honest, I am very surprised that someone other than me was using... [18:06:31] hi releng! Does someone have time to push a new image to the docker dev-images repo? [18:06:33] 10GitLab, 10serviceops: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) a:03Dzahn [18:06:55] Looks like fundraising-smashpig-buster-php73-apache2 (0.0.1-1-s3) isn't there yet [18:07:10] (merged in this commit: https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/e9c958a36b0c2cf6ee47e1c3404d2e85fb343eb4 ) [18:21:41] o/ [18:22:54] Is that something that usually takes manual intervention? [18:28:16] It appears so. Trying. [18:32:28] hmm. [18:33:38] so the releng/dev-images repo is hosted on Gerrit but you provided a link to a gitlab commit above. [18:34:03] ejegg: Was there a corresponding commit made to Gerrit? [18:34:49] just reading scrollback - dev-images is officially on gitlab these days [18:35:27] although i may have failed to archive it on gerrit [18:35:34] ooh, thanks. [18:36:26] dancy - i'm on a somewhat flaky satellite connection at the moment, but if you don't mind running a couple commands in the background, instructions for publishing are in dev-images/README.md [18:36:26] * dancy makes an attempt to update the remote [18:37:29] i thought we had that handled - it should be set properly in puppet, and i manually updated it in place a while back, but maybe something went awry with the puppet change... [18:37:30] branch changed to main too, I presume. [18:37:40] yeah [18:37:46] I'm accessing from a checkout on my machine. [18:37:56] oh, right, gotcha. [18:38:48] ok.. my repo is up to date now. The next trick is getting docker-pkg to do what I want. [18:41:01] Is there an official host where these are normally built? [18:42:12] ooh, looks like I made progress but _not_ trying to use --select to limit which image is built. [18:46:42] dancy: for publishing, they get built on contint1001 - for local testing i usually just do it on my box. [18:46:54] thx. I'll log in there. [18:49:20] for deploying, `tox -e fabric -- deploy_docker` should do the the trick if you've got tox locally. [18:50:28] Did that. Looks like /srv/dev-images is owned by you and not fully group writeable so that's not working. [18:50:53] `error: cannot update the ref 'refs/remotes/origin/main': unable to append to '.git/logs/refs/remotes/origin/main': Permission denied` [18:50:58] I asssume it is logging in as me. [18:52:19] I'll try enabling core.sharedRepository [18:54:21] alright. that plus a sudo chmod got me moving along. [18:55:56] !log Updating dev-images docker-pkg files on primary contint [18:55:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:01:05] ejegg: Published [19:01:32] `Successfully published image docker-registry.discovery.wmnet/dev/fundraising-smashpig-buster-php73-apache2:0.0.1-1-s3` [19:04:25] thx dancy [19:13:01] thanks dancy ! [19:13:29] 👍🏾 [19:25:57] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 (10dancy) [19:31:18] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 (10dancy) [19:34:11] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Etonkovidova) [22:23:59] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 (10dancy) [22:28:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Ladsgroup) I ran this for an hour. The scary part is that it seems 85% of the time i... [22:30:14] Amir1: that is generated from perf_tools? is that wall clock or cpu time? [22:30:34] yeah, it's perf [22:30:35] wall [22:30:45] at least that's my understand of perf [22:31:20] I'm also trying to get excimer running on tests https://gerrit.wikimedia.org/r/748314 (and https://gerrit.wikimedia.org/r/748312) [22:32:00] Amir1: interesting, and that samples multiple cpus at the same time? [22:32:07] I suspect that swapper == idle. [22:32:09] or does it sample one of the cpus only? [22:32:09] yup it's -ag [22:33:37] dancy: no, I did what it's recommended to remove idle (grep -v cpu_idle) and still swapper was 85% [22:34:27] I think I need to run it like snapshot and let it run for a day and see how that goes [22:35:59] Anyone here know how to use phan-SecurityCheck properly? I'm running into an issue enabling it in wikimedia/cdb where it flags every use of file_ and _open etc as unsafe because $this->filename is "tainted" as being assigned to $argv user input. [22:36:01] the kallsysms syscalls is fishy there in the swapper. there is a small chance that the perf tool just overwhelmed the system being ran for an hour (but the data is not much 300MB) [22:36:07] I marked it as @param-taint none, but it seems to make no difference [22:37:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Ladsgroup) the kallsysms syscalls is fishy there in the swapper. there is a small ch... [23:05:11] details re taint issue - https://gerrit.wikimedia.org/r/c/cdb/+/749278/1 [23:20:35] oh perf has its own wiki https://perf.wiki.kernel.org/index.php/Tutorial [23:20:56] > set $wgLogo [23:21:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS, 10Wikidata, and 2 others: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs - https://phabricator.wikimedia.org/T281122 (10Ladsgroup) >>! In T281122#7426577, @nskaggs wrote: > I wan... [23:29:49] hmm, it's running mw 1.19.24 [23:46:32] dancy: actually you're right. That looks like the idle but for whatever reason the output does not include idle and instead includes long hex strings that decoding returns mojibakes [23:46:55] https://www.irccloud.com/pastebin/HPJUbjoV/ [23:51:06] e.g. in here https://www.brendangregg.com/perf.html [23:51:19] it supposed to look like this: [23:51:24] https://www.irccloud.com/pastebin/4LMTyk5A/ [23:52:33] glad to see I'm not the only person who immediately goes to Special:Version upon seeing MW in the wild [23:55:24] It's one of the first things I do when people post links in #mediawiki