[02:13:31] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.4 unicode changes - https://phabricator.wikimedia.org/T292552 (10tstarling) [02:14:28] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) [06:40:15] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) [06:40:33] 10serviceops, 10MW-1.40-notes (1.40.0-wmf.6; 2022-10-17), 10PHP 7.4 support, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11 - https://phabricator.wikimedia.org/T292552 (10tstarling) The list of affected pag... [07:11:36] 10serviceops, 10Observability-Tracing, 10Epic: Package OpenTelemetry Collector as a .deb - https://phabricator.wikimedia.org/T320551 (10Clement_Goubert) Sounds good. Will get on the repackaging. [09:12:59] 10serviceops, 10Prod-Kubernetes: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) [09:13:45] 10serviceops, 10Prod-Kubernetes: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) 05Open→03In progress p:05Triage→03Medium [09:28:24] 10serviceops, 10Observability-Tracing, 10Epic: Package OpenTelemetry Collector as a .deb - https://phabricator.wikimedia.org/T320551 (10Clement_Goubert) Actually, if we're starting from `otelcol-contrib` I can probably just import the existing `.deb` as third party in order to kickstart the v0, and we can ch... [09:38:05] 10serviceops, 10Observability-Tracing, 10Epic: Package OpenTelemetry Collector as a .deb - https://phabricator.wikimedia.org/T320551 (10Clement_Goubert) @MoritzMuehlenhoff What are your thoughts on importing the raw `otelcol-contrib_0.62.1_linux_amd64.deb` from https://github.com/open-telemetry/opentelemetry... [09:58:10] hello folks [09:58:17] I got this weird error from helm lint [09:58:18] https://integration.wikimedia.org/ci/job/helm-lint/8051/consoleFull [09:58:31] it is not related to my change, but it seems more a go error [09:58:53] "sweep increased allocation count" [09:59:07] not sure if related to the new helm version or not [09:59:11] Cc: claime: --^ [09:59:27] it may be transient but reporting it anyway just in case [10:00:17] <_joe_> claime: did we update the docker image to use the newest helm version [10:00:19] <_joe_> ? [10:00:52] I think so yes, give me a sec [10:02:02] https://gerrit.wikimedia.org/r/c/integration/config/+/841042 yep [10:02:02] <_joe_> elukey: if so, can you try to run your change locally through an older version of the image? that will tell us if your hunch is right [10:03:36] <_joe_> the error seems to come from helmfile though? [10:03:57] The jenkins log is making my firefox beachball >< [10:04:27] <_joe_> claime: search for "sweep increased" [10:04:43] Yeah, once it's stopped crashing I will [10:04:44] :p [10:05:06] <_joe_> yep, it's helmfile [10:11:31] 10serviceops, 10SRE, 10observability, 10Maps (Kartotherian): Get Kartotherian SLO metrics into Prometheus - https://phabricator.wikimedia.org/T320748 (10fgiunchedi) With my Observability/Prometheus hat on: to bridge the statsd/prometheus gap we've been deploying `profile::prometheus::statsd_exporter` e.g.... [10:11:32] That error message seems to be gc related [10:15:59] sorry I was afk [10:16:18] no idea why admin is showing that much of a diff though [10:16:24] we do have quite a few versions difference with latest helmfile release [10:16:41] elukey: That's because of the way the linter works [10:17:04] If it breaks for some reason, it'll print out 11:27:03 +Template did not render correctly (HEAD of local branch). instead of the diff [10:17:21] Well, instead of the new chart, to feed to the diff [10:17:52] ahhh okok makes sense [10:18:02] I am re-running it in https://integration.wikimedia.org/ci/job/helm-lint/8052/console [10:18:15] just to see if it reproduces [10:18:49] if it doesn't reproduce, we should still probably look at the changelogs in-between what we have deployed and latest [10:19:00] even if* [10:19:45] my change will generated a big diff for sure, even if the changes themselves are trivial [10:19:54] maybe I crossed a certain tipping point for helm lint [10:20:42] Nah it went ok [10:20:49] I blame cosmic rays :p [10:21:51] let's keep this occurrence in mind if it re-happens, maybe it was just a one off weird corner case [10:21:55] thanks for the support :) [10:22:53] I'll check out the changelogs for helmfile just in case [10:27:46] <_joe_> elukey: I would open a task maybe [10:30:49] _joe_ if it doesn't reproduce or re-occur in the future I fear that the task will get only a lot of dust during the coming months, this is why I wasn't planning on creating one now (there are already 100 million things in our backlog) [10:41:08] 10serviceops, 10Prod-Kubernetes: chartctl: tool for helm chart management - https://phabricator.wikimedia.org/T320793 (10Joe) [10:41:40] 10serviceops, 10Prod-Kubernetes: chartctl: tool for helm chart management - https://phabricator.wikimedia.org/T320793 (10Joe) 05Open→03In progress p:05Triage→03Medium [10:41:43] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) [10:55:20] 10serviceops, 10Kubernetes: Remove kubeyaml from deployment-charts CI - https://phabricator.wikimedia.org/T316348 (10Clement_Goubert) a:03Clement_Goubert [11:20:22] 10serviceops, 10Data Engineering Planning, 10SRE, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10Clement_Goubert) Just for confirmation before diving into it on Monday, the list of services to re-de... [12:29:35] 10serviceops, 10Prod-Kubernetes, 10Patch-For-Review: Better management for helm charts - https://phabricator.wikimedia.org/T320782 (10Joe) [12:29:52] 10serviceops, 10Prod-Kubernetes: sextant: tool for helm chart management - https://phabricator.wikimedia.org/T320793 (10Joe) [12:30:28] 10serviceops, 10Prod-Kubernetes: sextant: tool for helm chart management - https://phabricator.wikimedia.org/T320793 (10Joe) [13:02:11] 10serviceops, 10Data Engineering Planning, 10SRE, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10Ottomata) Correct! [13:54:45] 10serviceops, 10Kubernetes, 10Patch-For-Review: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10Clement_Goubert) [13:54:47] 10serviceops, 10Kubernetes, 10Patch-For-Review: Remove kubeyaml from deployment-charts CI - https://phabricator.wikimedia.org/T316348 (10Clement_Goubert) 05Open→03In progress [15:00:11] Just a heads-up/RFC: Platform are beginning to talk about moving the first AQS 2.0 service to k8s. AQS 2 isn't going to be one service but a series of services that provide different endpoints. Would anyone be up for having a chat some time next week to flesh out details once I have more info> [15:00:25] <_joe_> yes [15:01:01] I'll be doing my best to ensure that we have as many reusable components as is possible of course [15:01:11] Ideally we could reuse the cassandra-http-gateway stuff [15:01:37] <_joe_> hnowlan: on the topic of reusability [15:01:55] <_joe_> yes, we should extract as many common templates as we can [15:02:23] <_joe_> if these services all look alike, we might get away with a single chart btw [15:02:49] <_joe_> also see https://phabricator.wikimedia.org/T320782 and the patch therein :P [15:04:13] oh neat [15:04:44] I am hoping/inclined to strongly suggest that we can use https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/cassandra-http-gateway with little to no modification [15:04:45] <_joe_> please poke holes :) [15:05:03] <_joe_> yeah that'd be great [15:05:25] <_joe_> given all this stuff doesn't need to go outside of k8s either, we can consider not having lvs either [15:08:50] <_joe_> just point the api gateway to the k8s service name [15:09:08] +1 to anything that bypasses LVS in useful ways! :) [15:10:28] oh neat, and still use ingress [15:10:55] <_joe_> we don't even need ingress if no other service needs to access them [15:11:16] <_joe_> there's only one limitation: to depool aqs in one dc, we'd need to depool the gateway [15:13:57] <_joe_> hnowlan: to be continued on monday! [15:26:00] sounds good :) [15:26:33] claime: still needs some work but hack a starting point https://phabricator.wikimedia.org/P35489 [15:26:35] yeah depooling the gateway is NOT GREAT. Probably will have to depool via config pushes [15:27:12] shows resources that changed in a specific time window [19:10:47] 10serviceops, 10Release-Engineering-Team (Seen): contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) Today we tried to upgrade the docker version on contint* hosts and we had to ask for the package to be imported into buster. We should use this opportunity, the hardware refresh, to u... [19:11:19] 10serviceops, 10serviceops-collab, 10Release-Engineering-Team (Seen): contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [19:11:32] 10serviceops, 10Release-Engineering-Team (Seen): contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10Dzahn) [20:54:42] 10serviceops, 10Kubernetes, 10Patch-For-Review: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10Jdforrester-WMF) [20:55:04] 10serviceops, 10Kubernetes, 10Patch-For-Review: Remove kubeyaml from deployment-charts CI - https://phabricator.wikimedia.org/T316348 (10Jdforrester-WMF) 05In progress→03Resolved All deployed.