[07:32:14] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10186599 (10elukey) The dry run took ~33 hrs and its log can be found on `registry1004:/home/elukey/docker_registry_dryrun_gc.log`. The blobs/l... [07:33:34] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10186597 (10JMeybohm) 05Open→03Resolved I don't see any unexpected out of sync events from calico anymore since m... [09:02:38] 06serviceops, 10Prod-Kubernetes, 10Data-Engineering (Q1 2024 July 1st - September 30th), 10Data-Platform-SRE (2024.09.28 - 2024.10.18), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10186930 (10gmodena) [10:27:15] 06serviceops, 10MediaWiki-Uploading, 13Patch-For-Review, 07Regression, 07Wikimedia-production-error: Large file uploads broken via Special:Upload - https://phabricator.wikimedia.org/T374436#10187200 (10hnowlan) p:05Unbreak!→03Medium [11:06:54] 06serviceops, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187271 (10Zabe) [11:09:27] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187272 (10Zabe) [11:27:09] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187300 (10Zabe) 05Open→03Resolved a:03jnuche scap got rolled back to a non-b... [11:27:13] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187303 (10jnuche) 05Resolved→03Open a:05jnuche→03None [11:28:32] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187308 (10jnuche) As a temporary measure, I rolled scap back to 4.104.0 [13:16:44] hello, I have to test a new spicerack release that has some changes on how to instantiate a confctl instance and I'd like to have a host that I could use to be able to test depool+pool on it. Do you have any suggestion? [13:19:34] volans: you can pick a random k8s worker (even in staging) I'd say [13:20:22] jayme: like kubestage1003.eqiad.wmnet ? [13:20:37] volans: tep [13:20:40] *yep [13:20:43] great, thanks a lot! [13:26:30] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10187584 (10elukey) Some stats: ` swift_account_stats_objects_total{account="AUTH_docker", cluster="swift", instance="ms-fe1009:9112", job="st... [13:28:17] hey folks! [13:28:36] qq - I/F is doing the hackathon and one project is to replace the stack for irc.wikimedia.org [13:28:57] we created irc1003, is it ok to deploy the new IP via https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1076730 ? [13:29:07] only network policies, I remember the scap command [13:33:25] elukey: +1ed [13:37:45] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10187610 (10akosiaris) Nice job! So, 75k is a not insignificant but also a not particularly large percentage. ~6%. I have my doubts it will su... [13:48:56] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187633 (10jnuche) p:05Unbreak!→03High [14:05:58] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10187719 (10elukey) Hi @akosiaris! I had a chat with Janis and there are some details that needs to be highlighted: * I didn't run it with `--... [14:06:13] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10187722 (10elukey) Started: ` elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect --dry-run /etc/dock... [14:48:25] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10187888 (10dancy) Previously docker commands were only run by the `mwbuilder` user... [14:53:51] 06serviceops: Update PHP 8.1 packages - https://phabricator.wikimedia.org/T376036 (10Scott_French) 03NEW [15:56:20] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10188254 (10bd808) Related to {T303450}. The current docker group membership for t... [16:04:40] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10188324 (10Jclark-ctr) [16:05:09] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install mc-misc100[12] - https://phabricator.wikimedia.org/T371987#10188325 (10Jclark-ctr) These are both setup and waiting for puppet / preseed to be updated to complete [16:11:02] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org: wikitech self-auth: Allow wikitech to use its own internal authentication - https://phabricator.wikimedia.org/T371588#10188343 (10bd808) [16:58:11] 06serviceops, 13Patch-For-Review: Update PHP 8.1 packages - https://phabricator.wikimedia.org/T376036#10188564 (10Scott_French) 05Open→03Resolved [22:07:07] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored mw-debug k8s deployment - https://phabricator.wikimedia.org/T372604#10189848 (10Scott_French) Summary of the current state: All of the necessary patches to turn up the "next" release are ready to go, but I'd like to wait until at least the "support mul... [22:21:19] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10189865 (10Scott_French) Summary of the current state: We'll have the first 8.1-based service images available soon, at which point we should be unblocked to start testing. I would propose that... [22:45:14] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10189894 (10bd808) [22:45:17] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10189895 (10bd808)