[09:36:32] 06serviceops, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10198903 (10Joe) >>! In T352650#10179932, @dr0ptp4kt wrote: > @Joe checking - is Q2 FY 24-25 still looking... [09:43:10] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10198925 (10elukey) I may have found some clue related to why the catalog contains so many stale things: https://github.com/distribution/distri... [09:45:30] 06serviceops, 06Infrastructure-Foundations, 06SRE: Timeout while retrieving the catalog from the Docker Registry - https://phabricator.wikimedia.org/T376285#10198926 (10elukey) I tried to disable the Redis cache `blobdescriptor` setting `inmemory` for eqiad registry nodes, and I didn't hit the timeout proble... [09:56:42] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10198967 (10jijiki) Thanks @taavi for pointing it out, we'll try to find a temporary bandaid to keep dumps running... [12:00:53] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: Build and add Mercurius to PHP base image - https://phabricator.wikimedia.org/T371699#10199153 (10hnowlan) Mercurius images for bookworm and bullseye are now building via CI (with some modifications for bullseye): https://gitlab... [12:50:54] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10199240 (10Ladsgroup) This probably needs a ticket to undeploy the extension altogether fro... [13:34:46] 06serviceops, 06Infrastructure-Foundations, 06SRE: Timeout while retrieving the catalog from the Docker Registry - https://phabricator.wikimedia.org/T376285#10199389 (10elukey) Current setup: * registry100* hosts using inmemory blobdescriptor cache * registry200* hosts using redis blobdescription cache The... [13:57:21] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10199496 (10Jhancock.wm) [14:02:30] 06serviceops, 06DC-Ops, 10ops-codfw: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10199506 (10Jhancock.wm) @jijiki hi, we got the servers in this week and are going to be racking them today. Could you update the operations and puppet repo for us? I'm hoping to get them ins... [16:09:06] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10199975 (10jijiki) 05Open→03In progress p:05Triage→03Unbreak! [16:20:31] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200015 (10jijiki) 05Resolved→03In progress [16:20:47] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200019 (10jijiki) p:05Unbreak!→03Low [16:27:43] 06serviceops, 06cloud-services-team, 10MW-on-K8s, 10wikitech.wikimedia.org, 13Patch-For-Review: Review/update wikitech-static syncing after wikitech moves to Kubernetes - https://phabricator.wikimedia.org/T374114#10200009 (10jijiki) 05In progress→03Resolved a:03jijiki @Ladsgroup and I believe this... [16:33:28] 06serviceops, 06cloud-services-team, 06Infrastructure-Foundations, 10wikitech.wikimedia.org, 13Patch-For-Review: LdapAuthentication: Disable extension from Wikitech - https://phabricator.wikimedia.org/T371592#10200066 (10Pppery) {T376097} was already created. [17:34:58] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200401 (10cmooney) To make progress here while we work on automating the sub-zone delegation I have manually delegated the required zones covering the k... [18:04:46] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200521 (10ssingh) Since `gdnsd` is fine and `pdns-recursor` is not, could it be because of this? https://doc.powerdns.com/recursor/settings.html#settin... [18:07:52] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200529 (10ssingh) If we want to confirm the above, we can depool a DNS host, disable Puppet, manually edit `recursor.conf`, restart it, test it and then... [18:21:19] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200581 (10cmooney) @ssingh yes that I think is probably it! Totally makes sense for the recursor to have that rule. People often mess up and put those... [18:30:06] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200622 (10ssingh) Are the kubectls IP in some particular subnet? If so, we can exclude them from above, such as `!10.64.0.0/12` or something -- you get... [18:44:30] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10200686 (10cmooney) @ssingh you were correct!!! awesome <3 We depooled dns1005 and then modified `/etc/powerdns/recursor.conf`, adding a //dont-query//... [20:13:56] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10200859 (10Scott_French) The changes to support `routed_via: main` appear to work as expected: ` swfrench@deploy2002:~$ curl -v 'https://staging.svc.eqiad.wmnet:4014/healthz' ... snip ... < HTT... [21:20:03] 06serviceops, 10MW-on-K8s: Migrate mwmaint server functionality to mw-on-k8s - https://phabricator.wikimedia.org/T341560#10200963 (10EBernhardson) [21:37:48] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10200997 (10EBernhardson) Could --follow also exit with the same return code as the script that was executed? The most notable example of this on our side is the CirrusSearch Upda... [22:31:11] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: parsoidtest1001 implementation tracking - https://phabricator.wikimedia.org/T363402#10201192 (10ABreault-WMF) > There's a couple of followup patches to remove scandium and decom it, but I 'll defer those for the week of Sept 7th. I've updated https://w...