[04:53:14] 10serviceops, 10Kubernetes: deployment-charts CI should allow opting out of default fixture injections - https://phabricator.wikimedia.org/T337359 (10Joe) [04:55:05] 10serviceops, 10Kubernetes: deployment-charts CI should allow opting out of default fixture injections - https://phabricator.wikimedia.org/T337359 (10Joe) 05Open→03In progress p:05Triage→03High a:03Joe [04:55:30] 10serviceops, 10Kubernetes: deployment-charts CI should allow opting out of default fixture injections - https://phabricator.wikimedia.org/T337359 (10Joe) [04:55:32] 10serviceops, 10Observability-Tracing: Helmchart for OpenTelemetry Collector - https://phabricator.wikimedia.org/T324117 (10Joe) [07:38:00] 10serviceops, 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure: kafka_mirror_maker TLS cert about to expire - 2023 - https://phabricator.wikimedia.org/T337248 (10elukey) @jbond thanks for checking! I think that the main question mark is what a client cert for kafka mirror maker (and potentially also... [07:39:55] hello folks! [07:40:15] I am going to deploy API Gateway in eqiad/codfw (Hugh knows, just giving an heads up here too) [07:41:08] 10serviceops, 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure: kafka_mirror_maker TLS cert about to expire - 2023 - https://phabricator.wikimedia.org/T337248 (10JMeybohm) >>! In T337248#8875545, @elukey wrote: > @jbond thanks for checking! I think that the main question mark is what a client cert f... [07:45:37] 10serviceops, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) I've added a v1.26 branch to the envoyproxy repo with the upstream code removed and packaging the upstream binary instead: https://gerrit.wikimedia.org/r/plugins... [08:45:34] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: hw troubleshooting: CPU error for mw2448.codfw.wmnet - https://phabricator.wikimedia.org/T334429 (10Clement_Goubert) >>! In T334429#8833688, @Jhancock.wm wrote: > The recommended fix for this one (according to Dell) is a reboot and see if the error comes back.... [08:57:53] Heads up, I am rebooting kafka-main @ codfw hosts [09:02:48] any clue as to who might update php7.4-pcov package for Buster? The current version (1.0.6) has some poor issue notably one causing large memory usage when generating coverage report https://phabricator.wikimedia.org/T243847#8080614 :] [09:10:36] 10serviceops, 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure, 10Patch-For-Review: kafka_mirror_maker TLS cert about to expire - 2023 - https://phabricator.wikimedia.org/T337248 (10jbond) > We can create multiple certs with the same CN on different machines (or even on the same machine). Thats us... [09:23:15] 10serviceops, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10akosiaris) > As with kubernetes and isio I choose a branch per minor version instead of "envoy-future" to make it more clear and to allow for easier upgrades of older vers... [12:47:15] 10serviceops, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) >>! In T300324#8875975, @akosiaris wrote: > * Is there a specific reason that debian/source/format says 1.0 instead of 3.0 (quilt) ? > * debian/changelog should... [14:24:44] jayme: i'm getting ready to submit a patch to deploy flink-operator in eqiad and codfw wikikube, might be easier to submit that patch if we merge and do https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922138 first [14:24:55] (undeploy from staging codfw) [14:25:01] does that patch look right to you? [14:25:35] I'm in an argument with Alex about that :) [14:25:51] oh haha [14:26:11] okay i'll keep it as is for now then, we can resolve the conflict with that later if we have to [14:26:50] ack. I'll try to resolve the conflict with Alex tomorrow ;-) [14:51:55] flink operator in wikikube eqiad and codfw patch: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922874 [15:04:55] 10serviceops, 10SRE, 10Traffic, 10Platform Team Initiatives (API Gateway): Handle edge cache invalidation for the api gateway - https://phabricator.wikimedia.org/T324200 (10kamila) [15:05:31] 10serviceops, 10SRE, 10Traffic, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Create Benthos docker image - https://phabricator.wikimedia.org/T336658 (10kamila) 05In progress→03Resolved Image built and published. [15:17:00] 10serviceops, 10Traffic, 10envoy: Refactor envoy.filters.http.router and envoy.filters.listener.tls_inspector - https://phabricator.wikimedia.org/T337405 (10JMeybohm) [15:21:09] 10serviceops, 10SRE-swift-storage, 10Wikimedia-Site-requests: Cleanup cirrus keys in $wmfSwiftEqiadConfig - https://phabricator.wikimedia.org/T199220 (10MatthewVernon) @dcausse no worries; the account looks (`search:backup` in ms-swift) to have been created in 2014, and is using some storage: ` root@ms-fe100... [15:23:56] 10serviceops, 10Traffic, 10envoy, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [15:27:34] 10serviceops, 10SRE-swift-storage, 10Wikimedia-Site-requests: Cleanup cirrus keys in $wmfSwiftEqiadConfig - https://phabricator.wikimedia.org/T199220 (10MatthewVernon) The equivalent account in codfw is empty: ` root@ms-fe2009:~# swift stat Account: AUTH_search Containers: 0 Objects: 0... [15:59:25] hello folks [15:59:39] I am moving the kafka mirror maker instances to pki, including the kafka main ones [15:59:47] so far test and jumbo went fine [16:02:01] nice! [16:12:42] 10serviceops, 10Data-Engineering, 10SRE, 10Shared-Data-Infrastructure: kafka_mirror_maker TLS cert about to expire - 2023 - https://phabricator.wikimedia.org/T337248 (10elukey) Rolled out the new keystores to all clusters! Next steps: * Clean up kafka mirror's classes as suggested in https://gerrit.wikime... [16:18:04] all good, rolled out, everything seems fine [16:18:09] ping me if anything looks weird :) [16:30:57] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Continuous-Integration-Config, 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) [16:31:35] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Continuous-Integration-Config, 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) >>! In T243847#828... [16:33:07] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) FYI: I just added swift access key to wikikube main mw-pa... [16:34:05] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) [16:52:54] 10serviceops, 10Release-Engineering-Team (Priority Backlog 📥): PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10thcipriani) [19:26:55] 10serviceops, 10SRE, 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10Test-Coverage: Add pcov PHP extension to wikimedia apt (and upgrade from 1.0.6-4+wmf1~buster1 to 1.0.11) so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10hashar) [20:33:31] 10serviceops, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review, 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata)