[02:50:34] 10serviceops, 10Thumbor, 10Patch-For-Review, 10Platform Team Workboards (Platform Engineering Reliability): Upgrade Thumbor to bullseye - https://phabricator.wikimedia.org/T336881 (10AntiCompositeNumber) [09:16:31] 10serviceops, 10Machine-Learning-Team: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10kevinbazira) [10:13:50] 10serviceops, 10Shellbox, 10SyntaxHighlight, 10MW-1.41-notes (1.41.0-wmf.13; 2023-06-13), and 2 others: Pages with Pygments or Timeline intermittenly fail to render (Shellbox server returned status code 503) - https://phabricator.wikimedia.org/T292663 (10jijiki) Due to a typo, the patch that fixed this iss... [11:07:09] 10serviceops, 10Documentation: Create template on Wikitech for documenting production services - https://phabricator.wikimedia.org/T336354 (10jijiki) [11:42:05] 10serviceops, 10iPoid-Service, 10Kubernetes: Create helm chart for iPoid - https://phabricator.wikimedia.org/T336163 (10jijiki) Helm chart is ready. We will be exposing the following environmental variables, some of which will hold sensitive information: ` MYSQL_HOST MYSQL_PORT MYSQL_DATABASE MYSQL_RW_USER... [11:53:31] 10serviceops, 10Thumbor, 10Patch-For-Review, 10Platform Team Workboards (Platform Engineering Reliability): Upgrade Thumbor to bullseye - https://phabricator.wikimedia.org/T336881 (10Jdforrester-WMF) [12:39:20] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking), 10Patch-For-Review: Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10akosiaris) I 've noticed erratic and spiky max memory use in [changeprop-jobqueue](https://grafana-rw.wikimedia.org/d/LSeAShkGz/jobqueue?o... [13:17:42] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10akosiaris) >>! In T338471#8914650, @elukey wrote: > Thanks for the info! > >>>! In T338471#8914520, @akosiaris wrote: >> Oh I forgot t... [13:26:58] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) >>! In T338471#8927496, @akosiaris wrote: >>>! In T338471#8914650, @elukey wrote: >> Thanks for the info! >> >>>>! In T338471#... [14:13:40] 10serviceops, 10SRE, 10Traffic, 10Datacenter-Switchover: Figure out what changes are needed in the traffic layer for having codfw be the r/w DC for half a year - https://phabricator.wikimedia.org/T337535 (10akosiaris) 05Open→03Resolved a:03akosiaris Makes sense. Added in [Phase 9 of the Switchover](h... [14:23:25] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Replace the current recommendation-api service with a newer version - https://phabricator.wikimedia.org/T338471 (10elukey) a:03elukey [14:48:53] hello folks [14:49:14] I'd like to try to re-assign partitions in kafka like stated in https://phabricator.wikimedia.org/T338357#8927833 [14:49:23] one topic in kafka-main, ok to proceed? [14:54:14] fine by me [14:58:53] go for it! [14:59:41] doing it :) [15:06:19] one partition still in progress [15:09:35] all done [15:10:21] ran a preferred-replica-election [15:10:37] it worked :) [15:12:54] kafka-by-topic (the dashboard) is showing some change as well [15:17:47] Amir1: ^ wanna try regenerating the flamegraph? we took 2 different correcting actions (one apparently useless), let's see if they helped [15:18:30] akosiaris: sure. Outside rn. Will do it soon [15:25:20] no rush [16:21:57] akosiaris: for the last hour (15:00 UTC) it is now at 5.9% from 30%: https://performance.wikimedia.org/arclamp/svgs/hourly/2023-06-13_15.excimer-wall.RunSingleJob.reversed.svgz?x=555.0&y=1301 [16:22:08] which is good but still high tbh [18:47:44] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: mw1492.eqiad.wmnet is down - https://phabricator.wikimedia.org/T338566 (10Jclark-ctr) 05Open→03Resolved Replaced main board. Server is back up now @elukey