[07:25:31] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11188174 (10elukey) I don't have big concerns on this, but I am going to add some historical context for completeness. The first bump was done in T359067, and the re... [07:45:34] 06serviceops, 06Infrastructure-Foundations: Increase RAM and nginx tmpfs on docker registry hosts - https://phabricator.wikimedia.org/T404742#11188202 (10MoritzMuehlenhoff) >>! In T404742#11188174, @elukey wrote: > Having a 12GB compressed limit is almost sort-of like not having it, but it is also good to... [08:26:06] 06serviceops, 06Traffic, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11188346 (10kostajh) @jijiki suggested that we consider using an approach similar to [[ https://codesearch... [08:28:05] 06serviceops, 06Traffic, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11188350 (10kostajh) [10:11:09] 06serviceops, 06Infrastructure-Foundations: Upgrade k8s deployments with prometheus-statsd-exporter 0.0.10 - https://phabricator.wikimedia.org/T404368#11188648 (10elukey) 05Open→03Resolved a:03elukey Reuven did most of the eqiad deployments (thanks!) and I did mobileapps this morning, so the rollout... [12:55:10] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11189287 (10Clement_Goubert) You can set `shell -$SHELL` in your `.screenrc` to make screen use a login shell. [14:07:00] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11189661 (10Jhancock.wm) [14:27:13] 06serviceops, 10envoy, 06SRE, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade Envoy to v1.29.12 on wcqs and wdqs hosts - https://phabricator.wikimedia.org/T404867 (10bking) 03NEW [14:58:47] 06serviceops, 06Content-Transform-Team: Migrate parsoidtest1001 to PHP 8.3 - https://phabricator.wikimedia.org/T403772#11189988 (10Scott_French) 05Open→03Resolved Many thanks to @cscott for starting a new RT test run to validate parsoidtest1001 post-upgrade, which seems to have completed without issue.... [16:06:50] 06serviceops, 06Infrastructure-Foundations, 06Machine-Learning-Team: Migrate the ownership of Docker images in production-images repo to mailing lists - https://phabricator.wikimedia.org/T373526#11190507 (10elukey) 05Open→03Resolved [16:54:45] 06serviceops, 10envoy, 06SRE, 10Data-Platform-SRE (2025.09.05 - 2025.09.26): Upgrade Envoy to v1.29.12 on wcqs and wdqs hosts - https://phabricator.wikimedia.org/T404867#11190745 (10bking) [17:21:09] 06serviceops, 06collaboration-services, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11190821 (10Dzahn) [17:24:00] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11190838 (10Scott_French) A bit more than 90 minutes on, no issues encountered for shellbox-timeline on 8.3, going by the usual set of service health and generic Shellbox-related error... [17:24:49] 06serviceops, 13Patch-For-Review: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11190840 (10Scott_French) [18:16:41] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11191019 (10Scott_French) A bit more than 30 minutes on from migrating shellbox-video, we're still waiting for new `webVideoTranscode*` jobs to confirm things work as expected ([[ https://grafana.wikimedia.o... [18:48:03] 06serviceops: Migrate production Shellbox services to PHP 8.3 - https://phabricator.wikimedia.org/T403284#11191184 (10Scott_French) [19:34:36] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11191364 (10Scott_French) Now that we're after the renewal horizon for the first batch of cfssl-based certs, I've confirmed that they have indeed been renewed and the notify functional... [20:49:59] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11191639 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host wikikube-ctrl2006.codfw.wmnet with OS bookworm [21:19:55] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11191736 (10Jhancock.wm) didn't pxe. will check nic in the morning. [22:10:18] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install wikikube-ctrl2006 - https://phabricator.wikimedia.org/T400661#11191983 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host wikikube-ctrl2006.codfw.wmnet with OS bookworm executed with errors: -... [22:18:57] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11192025 (10Krinkle) Brilliant. It hadn't ocurred to me that that could even be a thing. Thanks! This does mean if I want to keep `screen -DR` in my `.bash_profile` and automatically enter screen, I will n... [22:19:03] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11192026 (10Krinkle) 05Open→03Resolved a:03Krinkle [22:19:21] 06serviceops: "kube-env: command not found" when in GNU screen - https://phabricator.wikimedia.org/T404739#11192028 (10Krinkle) a:05Krinkle→03Clement_Goubert [23:12:51] 06serviceops, 06collaboration-services, 10envoy, 06SRE: Upgrade Envoy to v1.29.12 - https://phabricator.wikimedia.org/T403663#11192161 (10Dzahn) 05Open→03In progress