[08:26:58] 10serviceops, 10iPoid-Service: Deploy ipoid to staging on Kuberenetes - https://phabricator.wikimedia.org/T341326 (10kostajh) [08:32:48] 10serviceops, 10iPoid-Service: Deploy ipoid to staging on Kuberenetes - https://phabricator.wikimedia.org/T341326 (10kostajh) Current status: The staging pod seems to run without issue and there is no log output. (Once it receives requests, we should see some output.) ` [kharlan@deploy1002 ~]$ kube_env ipoid... [08:41:32] godog: I'm trying to debug https://gerrit.wikimedia.org/r/c/operations/alerts/+/936070 and not really getting anywhere [08:41:55] godog: I fixed the le expression to +Inf but that isn't it [08:55:45] claime: ack, taking a look [08:56:01] ty <3 [09:20:40] claime: I went with a slightly different approach otherwise I think we have to simulate the entire histogram buckets, did rate(..._sum) i.e. get the seconds added to the whole histogram irrespective of buckets [09:20:46] uploaded a new PS [09:20:53] godog: <3 [09:21:29] sure np! afaics that works, and checked with existing data [09:40:18] ty [10:19:04] 10serviceops, 10iPoid-Service: Deploy ipoid to staging on Kuberenetes - https://phabricator.wikimedia.org/T341326 (10jijiki) When the chart was created, it was missing a template, which was not evident until we were able to successfully start ipoid containers in the staging environment. After including the mis... [10:19:17] 10serviceops, 10iPoid-Service: Deploy ipoid to staging on Kuberenetes - https://phabricator.wikimedia.org/T341326 (10jijiki) 05Open→03Resolved a:03jijiki [11:02:33] 10serviceops, 10iPoid-Service: Deploy ipoid to staging on Kuberenetes - https://phabricator.wikimedia.org/T341326 (10kostajh) I [updated the documentation](https://wikitech.wikimedia.org/w/index.php?title=Service%2FIPoid&diff=2091134&oldid=2089441) on Wikitech for how to deploy to staging and verify that the s... [11:13:38] 10serviceops, 10SRE, 10observability, 10Patch-For-Review: stop using $::site in description field of service.yaml - https://phabricator.wikimedia.org/T258697 (10akosiaris) 05Open→03Resolved a:03akosiaris PCC at https://puppet-compiler.wmflabs.org/output/936062/42341/ says 0 diff for alert hosts, lvs... [11:34:58] 10serviceops, 10Thumbor, 10Patch-For-Review, 10Platform Team Workboards (Platform Engineering Reliability): Upgrade Thumbor to bullseye - https://phabricator.wikimedia.org/T336881 (10MoritzMuehlenhoff) >>! In T336881#8984098, @Ladsgroup wrote: > Bookworm is not really tested enough to be released to a mass... [12:44:32] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [12:57:13] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10TomerLerner) @MSantos @akosiaris thanks for your help with this! We call /mobile-sections-lead on the server side and ha... [13:02:32] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) 05Open→03Declined [13:12:32] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) >>! In T338471#8948416, @daniel wrote: > Wuld it be possible to implement a compatibility layer, so the app can use the new service without any chang... [13:33:44] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10Jgiannelos) Lets avoid using `MWOffliner` as it is a different API consumer and we wont be able to track the deprecation. [13:34:16] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Urbanecm) I tried the new REPL prompt per @joe's request, and I have two suggestions for consideration: * My first command to run was `sudo mw-debug-repl cswiki` (an... [13:46:41] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Joe) @Urbanecm sadly the last request isn't something we can actually do, as it would complicate quite a bit how the sudo rules would work. I'll add a check for the u... [13:50:20] 10serviceops, 10MW-on-K8s: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Urbanecm) >>! In T341197#8997366, @Joe wrote: > @Urbanecm sadly the last request isn't something we can actually do, as it would complicate quite a bit how the sudo r... [13:56:54] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Allow deployers to get a php REPL environment inside the mw-debug pods - https://phabricator.wikimedia.org/T341197 (10Joe) >>! In T341197#8997371, @Urbanecm wrote: >>>! In T341197#8997366, @Joe wrote: >> @Urbanecm sadly the last request isn't something we can ac... [14:30:08] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10SCherukuwada) @elukey Do we have another idea on the table aside from asking a team of Android devs (3) to maintain a recommendations service? While I have... [14:44:32] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) >>! In T338471#8997492, @SCherukuwada wrote: > @elukey Do we have another idea on the table aside from asking a team of Android devs (3) to maintain... [14:45:48] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10Aklapper) >>! In T338471#8997492, @SCherukuwada wrote: > I worry about the precedent it sets around moving ownership of backend services to frontend teams th... [14:45:57] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10akosiaris) Just to add my 2 cents as a generic observation. If we can't find any kind of an owner for this, it will eventually have to be undeployed and wh... [15:35:58] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [15:40:57] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [16:11:25] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10TomerLerner) It seems "Wikiwand/0.1 (https://www.wikiwand.com; admin@wikiwand.com)" is blocked on some (if not all) end... [16:11:44] Added new kafka metrics and commented in T338357 [16:12:07] still a little confused but we'll see [17:09:36] 10serviceops, 10SRE, 10TimedMediaHandler: Upgrade Wikimedia production's ffmpeg to 4.4 or later so we can use the fpsmax flag - https://phabricator.wikimedia.org/T318419 (10TheDJ) BTW. it seems that stable is now at 5.1.3-1. Our current versions are: - MW servers: 4.1.11 - Thumbor: 3.2.18 - Docker: 4... [19:56:04] 10serviceops, 10SRE, 10TimedMediaHandler: Upgrade Wikimedia production's ffmpeg to 4.4 or later so we can use the fpsmax flag - https://phabricator.wikimedia.org/T318419 (10brion) Note I've worked around this in the related cleanup on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+...