[07:10:28] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 8 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) p:05Triage→03Medium [07:13:02] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 8 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) [07:13:52] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 8 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Marostegui) [07:15:59] hello folks [07:16:16] if everybody agrees I'd merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/905251 and proceed slowly to migrate kafka main brokers [07:16:36] kafka test/jumbo/logging are already on PKI, main eqiad/codfw are the last one [07:16:39] *ones [07:21:17] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 8 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Marostegui) @ayounsi to confirm, codfw will be depooled before this maintenance right? @akosiaris @Joe ? [07:30:45] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10akosiaris) Yes, we 'll have to depool codfw. [07:31:03] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10ayounsi) >>! In T334049#8757732, @Marostegui wrote: > @ayounsi to confirm, codfw will be depooled before this maintenance right? @akosiaris @Joe ? That's m... [07:31:31] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MoritzMuehlenhoff) [07:43:55] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10netops, 10Patch-For-Review: Optimize k8s same row traffic flows - https://phabricator.wikimedia.org/T328523 (10ayounsi) 05Open→03Resolved a:03ayounsi This has been rolled to all k8s clusters. [07:57:43] ok I take the liberty to break everything :) [08:00:03] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Agree strategy for Kubernetes BGP peering to top-of-rack switches - https://phabricator.wikimedia.org/T306649 (10ayounsi) 05Open→03Stalled Marking it as stalled until the cookbook is reviewed/merged. [08:02:03] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Marostegui) [08:03:35] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10Marostegui) @jcrespo kindly check backup servers needs. Thanks [08:19:11] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10jcrespo) [08:54:37] 10serviceops, 10MW-on-K8s: Migrate linkrecommandations to mw-api-int - https://phabricator.wikimedia.org/T334060 (10Clement_Goubert) [08:55:50] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-on-K8s: Migrate linkrecommandations to mw-api-int - https://phabricator.wikimedia.org/T334060 (10Clement_Goubert) p:05Triage→03Medium [08:59:06] 10serviceops, 10MW-on-K8s, 10Product-Infrastructure-Team-Backlog-Deprecated, 10Push-Notification-Service: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) [08:59:34] 10serviceops, 10MW-on-K8s, 10Product-Infrastructure-Team-Backlog-Deprecated, 10Push-Notification-Service: Migrate push-notifications to mw-api-int - https://phabricator.wikimedia.org/T334061 (10Clement_Goubert) p:05Triage→03Medium [09:01:36] 10serviceops, 10MW-on-K8s, 10Recommendation-API: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) [09:01:51] 10serviceops, 10MW-on-K8s, 10Recommendation-API: Migrate recommendation-api to mw-api-int - https://phabricator.wikimedia.org/T334062 (10Clement_Goubert) p:05Triage→03Medium [09:04:07] 10serviceops, 10MW-on-K8s, 10Maps: Migrate tegola-vector-tiles to mw-api-int - https://phabricator.wikimedia.org/T334063 (10Clement_Goubert) [09:04:22] 10serviceops, 10MW-on-K8s, 10Maps: Migrate tegola-vector-tiles to mw-api-int - https://phabricator.wikimedia.org/T334063 (10Clement_Goubert) p:05Triage→03Medium [09:05:46] 10serviceops, 10MW-on-K8s, 10Wikidata-Termbox: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) [09:06:10] 10serviceops, 10MW-on-K8s, 10Wikidata-Termbox: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) p:05Triage→03Medium [09:07:20] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:15:10] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:15:21] 10serviceops, 10MW-on-K8s, 10Maps: Migrate tegola-vector-tiles to mw-api-int - https://phabricator.wikimedia.org/T334063 (10Clement_Goubert) 05Open→03Invalid [09:15:31] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:21:12] 10serviceops, 10MW-on-K8s: Migrate api-gateway to mw-api-int - https://phabricator.wikimedia.org/T334065 (10Clement_Goubert) [09:21:36] 10serviceops, 10MW-on-K8s: Migrate api-gateway to mw-api-int - https://phabricator.wikimedia.org/T334065 (10Clement_Goubert) p:05Triage→03Medium [09:22:43] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:27:22] 10serviceops, 10Add-Link, 10Growth-Team, 10MW-on-K8s: Migrate linkrecommandations to mw-api-int - https://phabricator.wikimedia.org/T334060 (10Clement_Goubert) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/905941 [09:43:51] 10serviceops, 10Machine-Learning-Team, 10SRE: Import and deploy istio 1.15.7 - https://phabricator.wikimedia.org/T334068 (10elukey) [10:58:07] 10serviceops, 10MW-on-K8s, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) Deploying to the staging-test environment yields errors https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-... [10:58:29] 10serviceops, 10MW-on-K8s, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech: Migrate termbox to mw-api-int - https://phabricator.wikimedia.org/T334064 (10Clement_Goubert) Reverting the change until we can investigate those differences further. [11:33:40] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10hnowlan) [14:34:30] folks kafka main eqiad is on PKI! [14:34:34] codfw still to be done [14:34:40] I'll keep an eye for errors [14:34:46] but so far it looks good [14:35:06] nice! Thanks you [14:36:36] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10Ottomata) > Could you please share resource requirements for the operator from your experiments on DSE here so that... [14:36:38] \o/ [14:36:55] thank yoUuu! [14:40:17] elukey: nice work! [14:40:44] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10Ottomata) BTW, IIUC {T331283} is an app specific configuration. The flink app (not the flink k8s operator) stores th... [14:40:53] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10Ottomata) [14:41:21] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) [15:05:58] 10serviceops, 10DBA, 10Data-Engineering, 10Discovery-Search, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10colewhite) [15:44:15] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, and 2 others: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Michaelcochez) Hi @akosiaris , apologies for the slow progress. We have been updating several things, but I didn't follow up with the...