[06:49:12] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10205622 (10JMeybohm) >>! In T344171#10196688, @CDanis wrote: > OK, one weird issue I've found which is confounding but not fatal: the NodePort isn't work... [06:58:26] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10205624 (10JMeybohm) > ====Providing certs to pods==== > * Secrets > ** The certificates could be secrets which we can then mount as files As per our initial discussion I would rath... [09:01:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10205937 (10JMeybohm) It seems that this is a temporary thing. During kubelet startup, cadvisor has not yet build up it's in memory cache of filesystem... [09:24:31] 06serviceops, 07Datacenter-Switchover: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10205994 (10Clement_Goubert) I vote we revert as well, we can potentially leave in the values for single-DC and a link to T371273 as comments in the config so we have a reference poin... [09:38:17] 06serviceops, 06Infrastructure-Foundations, 06SRE, 07Datacenter-Switchover, 13Patch-For-Review: sre.discovery.datacenter should support switching the active/passive services to the other datacenter - https://phabricator.wikimedia.org/T335364#10206034 (10Clement_Goubert) The code hasn't been reviewed and... [09:47:08] 06serviceops, 07Kubernetes: sextant update should support a minimal change mode - https://phabricator.wikimedia.org/T369119#10206050 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert [10:35:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10206233 (10JMeybohm) [10:48:46] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, 10wmde-wikidata-tech: Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604 (10Lucas_Werkmeister_WMDE) 03NEW [10:48:53] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, 10wmde-wikidata-tech: Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604#10206276 (10Lucas_Werkmeister_WMDE) I assume that the [input on stdin](https://wikitech.wikimedia.org... [11:19:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Various k8s releases use docker-registry.wikimedia.org (pull images through CDN) - https://phabricator.wikimedia.org/T376608#10206393 (10JMeybohm) [12:20:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10206521 (10jijiki) >>! In T365687#10205624, @JMeybohm wrote: >> ====Providing certs to pods==== >> * Secrets >> ** The certificates could be secrets which we can then mount as files... [12:22:48] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10206528 (10jijiki) [13:00:02] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616 (10Lucas_Werkmeister_WMDE) 03NEW [13:00:56] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616#10206654 (10Lucas_Werkmeister_WMDE) (For all I know, this might not actually be an issue in `mwscript-k8s` – perhaps it’s somewhere in `MWScript.php` or elsewhere in the multiversion machiner... [13:28:58] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Improve calico-typha firewall rules - https://phabricator.wikimedia.org/T365687#10206736 (10JMeybohm) >>! In T365687#10206521, @jijiki wrote: > That was poorly phrased, I meant, how often would we want the certificates to be renewed I would suggest to go with... [14:09:15] 06serviceops, 07Datacenter-Switchover: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10206882 (10kamila) +1 to reverting and leaving the pointer to T371273 in the comments, for the already mentioned reasons. [14:16:31] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes: Degraded RAID on wikikube-worker2092 - https://phabricator.wikimedia.org/T374409#10206920 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=d5aed8b0-eaca-4555-b388-ad989b1c0dd9) set by kamila@cumin1002 for 7 days, 0:00:00 on 1 host(s... [14:18:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10206923 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm [14:25:45] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10206929 (10JMeybohm) [14:26:05] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10206930 (10JMeybohm) [14:37:03] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10206962 (10JMeybohm) [14:47:29] 06serviceops, 07Datacenter-Switchover: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10206998 (10jijiki) > In the immediate term, we should determine whether to either accept this as the new normal (i.e., being "ready" for single-DC serving at steady state) or scale b... [15:43:31] 06serviceops, 10decommission-hardware: decommission scandium - https://phabricator.wikimedia.org/T376632 (10akosiaris) 03NEW [15:49:55] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: parsoidtest1001 implementation tracking - https://phabricator.wikimedia.org/T363402#10207226 (10akosiaris) 05Open→03Resolved We 'll be tracking decom of scandium in {T376632}, I 'll resolve this, feel free to reopen if something weird comes up w... [15:57:45] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616#10207247 (10RLazarus) a:03RLazarus If this changed in mwscript-k8s, it's unintended (but definitely not impossible) -- I'll dig into it today and get back to you. [16:16:44] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10207280 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestage2002.codfw.wmnet with OS bookworm completed: - kubestage200... [16:16:51] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616#10207281 (10RLazarus) This is MWScript.php behavior, and it's actually unchanged: ` rzl@deploy2002:~$ echo 'https://wikitech.wikimedia.org/wiki/User:RLazarus_(WMF)' | mwscript-k8s --attach -... [16:18:40] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616#10207286 (10Lucas_Werkmeister_WMDE) > In the meantime, the workaround is to continue including ".php" when not mentioning a wiki. Hm, I could swear I tried that earlier today and still got t... [16:23:51] 06serviceops, 10MW-on-K8s: mwscript-k8s no longer supports wikiless scripts - https://phabricator.wikimedia.org/T376616#10207309 (10Lucas_Werkmeister_WMDE) > The "maintenance/" prefix is added before that check I’m tempted to say `MWScript` should actually stop doing this – AFAIK it’s no longer necessary for... [17:00:16] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10207459 (10Jhancock.wm) [17:17:05] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored mw-debug k8s deployment - https://phabricator.wikimedia.org/T372604#10207511 (10Scott_French) mw-debug next is now up in eqiad and codfw - appears healthy and successfully serves Special:BlankPage on port 4453 [17:52:58] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored mw-debug k8s deployment - https://phabricator.wikimedia.org/T372604#10207667 (10Scott_French) Remaining steps for the initial turn-up: [ ] Basic service configuration: service catalog entry, conftool entities for discovery, and realserver IPs on the k8... [20:19:50] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10208261 (10EBernhardson) >>! In T341553#10203241, @RLazarus wrote: > > We could do this, but I'd want to think more about the implications -- I think we'd want to be able to ind... [20:24:14] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q1:rack/setup/install mc-misc200[12] - https://phabricator.wikimedia.org/T372800#10208277 (10jijiki) >>! In T372800#10199506, @Jhancock.wm wrote: > @jijiki hi, we got the servers in this week and are going to be racking them today. Could you... [21:23:39] 06serviceops, 07Datacenter-Switchover: Steady-state sizing of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T376519#10208502 (10Scott_French) Thanks, all, for weighing in! +1 to leaving the "valid as of September 2024" sizes around in the values file, commented out with details on when / where the... [23:23:04] 06serviceops, 10Parsoid (Tracking), 13Patch-For-Review: parsoidtest1001 implementation tracking - https://phabricator.wikimedia.org/T363402#10208977 (10ABreault-WMF) 05Resolved→03Open >>! In T363402#10207226, @akosiaris wrote: > We 'll be tracking decom of scandium in {T376632}, I 'll resolve this, feel...