[06:08:09] 10serviceops, 10Growth-Team, 10Growth-Team-Filtering, 10MW-on-K8s, 10Notifications: Broken (empty) cross-wiki notification when using $wgLocalHTTPProxy (e.g. on Kubernetes) - https://phabricator.wikimedia.org/T223413 (10Tgr) >>! In T223413#9240943, @matmarex wrote: > I am not sure how this related to T34... [10:00:24] folks I am going to start deploying the new version of eventstreams [10:00:31] on node18 + bookworm [10:00:36] will start from es-internal [10:07:03] aaand of course crashloop in staging [10:07:05] * elukey sighs [10:50:07] should be better now [10:53:04] staging seems to work fine without insulting me in the logs [10:53:11] will do prod this afternoon [10:57:11] nice [11:08:52] btw regarding our nodejs stuff [11:09:32] either we find a team that wants to evolve our nodejs "platform" (node-service-template) with a roadmap [11:09:55] or we should consider using stuff like https://platformatic.dev/ [11:10:26] they're open core but the right type of open core, mostly what the free software lacks are vendor integrations [11:41:01] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, and 2 others: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [12:55:43] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [13:03:59] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) 05In progress→03Resolved All wikifunctions.org traffic from the edge as well as from function-orchestrator is now served by th... [13:56:32] 10serviceops, 10SRE, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Patch-For-Review: Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Aklapper) I updated https://www.mediawiki.org/wiki/Extension:Graph/Graphoid accordingly. [14:16:47] joe: +1 definitely [14:23:27] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Patch-For-Review: Improve log readability of kubernetes applications when logging in debug level - https://phabricator.wikimedia.org/T347717 (10MSantos) a:03Jgiannelos [14:37:51] eventstreams internal deployed [14:38:04] eventstreams (external) deployed only in codfw, will monitor [14:38:30] one interesting thing is that I don't see any traffic for -internal in the past 3 months: [14:38:33] https://grafana.wikimedia.org/d/znIuUcsWz/eventstreams?orgId=1&refresh=1m&var-dc=eqiad%20prometheus%2Fk8s&var-service=eventstreams-internal&from=now-90d&to=now [14:38:49] it doesn't seem an issue with metrics, so we may want to start the process to undeploy [14:38:57] I'll open a task to event platform [14:58:44] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF) Thank you! [15:05:39] 10serviceops, 10Data-Engineering, 10Event-Platform: Traffic for eventstreams-internal seems to be zero for the past months - https://phabricator.wikimedia.org/T348763 (10elukey) [15:10:23] elukey: yeah we had 1-2 rps one year ago, I asked back then and was assured it would rampup [15:10:32] well sorry, 2 years ago [15:13:45] opened a task to undeploy :) [15:14:15] I'll also complete the eventstreams (external) rollout tomorrow in eqiad [16:34:16] 10serviceops, 10Abstract Wikipedia team, 10SRE, 10Traffic, 10Wikifunctions: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF) [16:34:26] 10serviceops, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 4 others: Migrate functions-orchestrator service to mw-api-int - https://phabricator.wikimedia.org/T347397 (10Jdforrester-WMF) [16:34:36] 10serviceops, 10Abstract Wikipedia team, 10Wikifunctions, 10Wikimedia-production-error: Wikifunctions functions that require a lookup on wikifunctions.org timing out in the orchestrator, UX instead showing 'http' - https://phabricator.wikimedia.org/T344998 (10Jdforrester-WMF) [16:34:40] 10serviceops, 10Abstract Wikipedia team: Sandboxing Strategy for Wikifunctions - https://phabricator.wikimedia.org/T343829 (10Jdforrester-WMF) [16:34:44] 10serviceops, 10Abstract Wikipedia team, 10function-evaluator: Split the monolithic function-evaluator service up in production so we have differently-scalable pods for python vs. node - https://phabricator.wikimedia.org/T343388 (10Jdforrester-WMF) [17:19:01] 10serviceops, 10Abstract Wikipedia team, 10function-evaluator: Split the monolithic function-evaluator service up in production so we have differently-scalable pods for python vs. node - https://phabricator.wikimedia.org/T343388 (10Jdforrester-WMF) 05In progress→03Resolved [17:19:04] 10serviceops, 10Abstract Wikipedia team, 10function-evaluator: Split the monolithic function-evaluator service up in production so we have differently-scalable pods for python 3.7 vs. python 3.8 vs. … - https://phabricator.wikimedia.org/T343389 (10Jdforrester-WMF) [17:49:12] 10serviceops, 10Observability-Metrics, 10Patch-For-Review: Decide on default histogram buckets for MediaWiki timers - https://phabricator.wikimedia.org/T344751 (10colewhite) I propose we implement a mixed approach. 1. Statsd-exporter will translate timing metrics into native histograms 2. MediaWiki will imp... [20:16:00] 10serviceops, 10MediaWiki-libs-Stats, 10Observability-Metrics, 10Patch-For-Review: Decide on default histogram buckets for MediaWiki timers - https://phabricator.wikimedia.org/T344751 (10colewhite)