[02:15:06] 10serviceops, 10MW-on-K8s, 10SRE-swift-storage, 10Shellbox, and 2 others: Support large files in Shellbox - https://phabricator.wikimedia.org/T292322 (10tstarling) There is openssl_digest() which presumably has hardware acceleration and can do SHA-256 in 2.1 seconds per gigabyte. But its input is a single... [10:22:30] 10serviceops, 10Toolhub, 10Patch-For-Review: Get mcrouter & prometheus-mcrouter-exporter tags for helmfile.d from upstream config - https://phabricator.wikimedia.org/T291530 (10Joe) 05In progress→03Resolved This task is resolved. I think it's time we write a style guide to writing a helm chart and managi... [10:33:33] 10serviceops, 10Prod-Kubernetes: Better scaffolding for helm charts / releases - https://phabricator.wikimedia.org/T292818 (10Joe) >>! In T292818#7431354, @akosiaris wrote: > To be honest, I am not understanding what the solution exactly is. I gather that the point is to not have a lot of boilerplate code in t... [12:46:22] Hi, in the #wikimedia-analytics channel we noticed that neither systemd nor rsync expand glob patterns like `*`, and so one such command was broken. We're currently looking into fixing it there. [12:46:30] Grepping through puppet/modules revealed two more usages of that: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/openstack/base/keystone/fernet_keys.pp#39 and https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/microsites/os_reports.pp#19 [12:46:41] They might be broken as well? [12:46:59] No idea who at wmf owns these tasks, but could you check the logs or let me know who to ping about it? [13:28:25] <_joe_> is there any task about this issue? [13:29:47] <_joe_> btw I'm pretty sure that systemd doesn't expand those, but the software might anyways [13:30:09] <_joe_> oh rsync again heh [13:31:00] for the analytics/dump one there is https://phabricator.wikimedia.org/T299358, but I have no idea what those places I found here are about [13:31:17] <_joe_> tbh, are you sure rsync doesn't expand glob patterns? [13:31:18] so I wouldn't know what tags to add to those tasks^^ [13:31:23] <_joe_> I was pretty sure of the opposite [13:31:48] afaik, the shell which runs rsync usually does it [13:33:51] <_joe_> anyways, the right place to ask is #wikimedia-sre [13:33:53] from the rsync man-page: [13:33:53] > Note that the expansion of wildcards on the command-line (*.c) into a list of files is handled by the shell before it runs rsync and not by rsync itself (exactly the same as all other Posix-style programs). [13:34:10] ok, I will bring it there :) [13:34:10] <_joe_> yeah I was reading as well [13:34:20] <_joe_> so yes, possible those are broken as well [13:34:25] and maybe I'll create some skeleton tasks [14:57:03] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [14:57:18] MichaelG_WMDE: _joe_: sorry din;t see this chatter, i have update the two tasks, [14:57:47] cool, thanks :) [14:57:59] In the examples above the source is an rsync:// protocol end point as such the '*' is sent to that end point and works as intended. [14:59:16] in the example from T299358, you are relying on shell expantion so this is causing issues. i have added aomment to the CR suggesting using --einclude --exclude rsunc params instead of processing the command through a shell [15:02:12] ah, much appreciated 🙏 [15:06:15] np :) [15:55:43] 10serviceops, 10MW-on-K8s, 10Release Pipeline: Pushes to docker-registry fail for images with compressed layers of size >1GB - https://phabricator.wikimedia.org/T288198 (10dancy) Noting for the record that pushes magically started working again around 12:28 UTC (2021/01/19). [16:54:06] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) a:03Dzahn [17:10:21] 10serviceops, 10MW-on-K8s, 10SRE: Make all httpbb tests pass on the mwdebug deployment. - https://phabricator.wikimedia.org/T285298 (10Joe) 05Open→03Resolved Right now we have just 3 tests not passing: * the nonexistent test, which cannot work without an exposed http endpoint, so we don't really care. *... [17:10:31] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review, 10User-jijiki: Create a mwdebug deployment for mediawiki on kubernetes - https://phabricator.wikimedia.org/T283056 (10Joe) [17:21:36] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) per the gitlab IC meeting we had today, the plan is: - give Jelto / Arnold / Brennen access to existing project "devtools" that has Gerrit and Phabricator... [17:22:53] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) 05Open→03In progress [17:35:37] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [x] added @Jelto @Arnoldokoth and @brennen to project `devtools`, with admin privileges After logging out and back in on Horizon you should see the new pr... [17:54:33] 10serviceops, 10GitLab: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) [17:54:47] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [17:58:57] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) 05In progress→03Stalled And yes, we are at quota limit. I think it's just "instance count" and not more specific about CPU/disk etc. Part of the reaso... [18:58:26] 10serviceops, 10Phabricator, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Aklapper) [18:58:34] 10serviceops, 10Phabricator, 10Release-Engineering-Team (Next): Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Aklapper) [19:23:59] 10serviceops, 10GitLab: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Dzahn) [19:24:15] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) [19:34:36] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install wtp10[49-72] - https://phabricator.wikimedia.org/T299573 (10RobH) [19:34:45] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install wtp10[49-72] - https://phabricator.wikimedia.org/T299573 (10RobH) [19:40:36] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install parse100[01-24] - https://phabricator.wikimedia.org/T299573 (10RobH)