[06:53:15] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Joe) we also need to add wikifunctions to our internal certs [07:13:06] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Joe) [07:34:27] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Jelto) [07:45:25] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MoritzMuehlenhoff) [07:52:55] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10fgiunchedi) [08:28:54] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Gehel) [08:29:12] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10Gehel) [08:42:05] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) [09:08:49] 10serviceops, 10Machine-Learning-Team, 10SRE, 10Language-Team (Language-2023-April-June), and 2 others: New Service Deployment Request: NNLB-200 for machine translation - https://phabricator.wikimedia.org/T329971 (10Pginer-WMF) [09:19:59] hello folks [09:20:36] I am in the process of adding the first non-kserve service to ml-serve and ml-staging, namely ores-legacy [09:21:33] while doing so I re-thought about what's best to keep wikikube and ml-serve more in sync, and maybe having an ml-staging endpoint could make sense [09:21:51] (even if we don't have a lot of plans for more services that are not kserve) [09:22:12] does it make sense? In theory the changes to make all work with the current settings and templates should be minimal [09:22:20] jayme: --^ [09:27:45] elukey: with endpoint you mean a dedicated entry in service::catalog for ingress? [09:29:35] jayme: yes, basically what you have done with staging.svc.eqiad.wmnet (I'd have only codfw) [09:29:49] with the DNS record etc.. [09:30:14] it would require a change in the helmfile templates to allow "ml-staging" too, but shouldn't be too hard [09:30:29] so that our clusters would have the same configs for ingress/mesh/etc.. in staging [09:31:04] k8s-ingress-staging you mean? staging.svc.eqiad.wmnet is just a CNAME [09:33:05] and the mesh config as well (the single cert etc..) [09:33:49] ah right I see, staging.svc points to kubestage1003 [09:33:52] okok [09:34:05] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10BTullis) [09:34:28] basically what I'd like to avoid is to have a VIP + configs + ingress etc.. every time that I add a service to ml-staging [09:34:50] we don't plan to have a lot more than ores-legacy but I am sure more will come in the future [09:34:54] yeah, understood. [09:35:02] you can tell me if I am crazy [09:35:13] otherwise I'll start working on it [09:35:14] :) [09:35:17] Nono. I think it makes sense [09:36:44] super, I'll try to send some changes today, let's see how it goes [09:55:19] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10BTullis) [09:59:49] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ops-monitoring-bot) akosiaris@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in codfw: codfw row C switches upg... [10:53:02] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ops-monitoring-bot) akosiaris@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all active/active services in codfw: codfw row C switches upg... [11:44:05] I'll go reboot the poolcounters in codfw now that codfw is depooled [11:59:28] these are done [12:11:21] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MoritzMuehlenhoff) [12:20:38] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Eevans) [12:24:15] moritzm: thanks :) [12:25:59] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ssingh) [12:27:05] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Eevans) [12:42:10] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Eevans) [12:46:00] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Eevans) [13:00:40] 10serviceops, 10SRE, 10CommRel-Specialists-Support (Apr-Jun-2023), 10Datacenter-Switchover, 10User-notice: CommRel support for April 2023 Datacenter Switchback - https://phabricator.wikimedia.org/T334671 (10Trizek-WMF) [13:03:29] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 11 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=21224f03-d3c2-4431-accb-64fcadd01a0f) set by ayounsi@cumin1001 for 2:00:00 on 185 host(s)... [13:24:38] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10klausman) [13:25:28] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MoritzMuehlenhoff) [13:37:21] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), 10HTTPS: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) a:03Clement_Goubert [13:47:55] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10Andrew) [14:02:35] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10ayounsi) [14:02:53] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) [14:10:51] 10serviceops, 10SRE, 10Traffic, 10Abstract Wikipedia team (Phase λ – Launch), and 2 others: Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) 05Open→03In progress [14:57:12] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Jelto) [15:00:06] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: codfw row C switches upgrade -... [15:07:03] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) 05Open→03Resolved a:03ayounsi Upgrade went fine! Thanks everybody. [15:17:08] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ops-monitoring-bot) jiji@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: codfw row C switches upgrade -... [20:16:28] 10serviceops, 10SRE, 10Traffic-Icebox, 10conftool: confd's watch functionality appears to be partially broken when interacting with etcd 3.x - https://phabricator.wikimedia.org/T260889 (10BCornwall) 05Stalled→03Resolved [20:19:09] 10serviceops, 10DBA, 10Data-Engineering, 10Data-Platform-SRE, and 10 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10colewhite)