[10:18:02] hnowlan, kamila_ o/ - I created https://gerrit.wikimedia.org/r/c/mediawiki/services/change-propagation/+/942476 to decouple the changeprop's upgrade [10:18:27] a complete revert is weird since there seems to be problems installing stretch packages :D [10:45:05] elukey: LGTM, but that doesn't actually mean anything, as I do not have enough clue about any of this to help with debugging if^Wwhen it breaks [10:46:57] kamila_: I am pretty sure that nobody has a real clue about changeprop :D [10:47:10] fair :D [10:47:12] 💯 [10:47:22] but the more we are checking it the better, so we can share the pain when it breaks :D [11:03:07] 10serviceops, 10Data Products, 10RESTbase Sunsetting, 10Code-Health-Objective, 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10hnowlan) Currently we're deploying AQS 2 services to the api-gateway rather than the rest-gateway - would it be suitab... [11:07:47] will test https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/942634/ in staging if ok [11:07:51] (changeprop) [11:10:55] cool! [12:08:43] 10serviceops, 10Content-Transform-Team-WIP, 10RESTbase Sunsetting, 10Wikifeeds, and 3 others: Switchover plan from restbase to api gateway for wikifeeds - https://phabricator.wikimedia.org/T339119 (10Jgiannelos) [12:45:39] 10serviceops, 10ChangeProp, 10WMF-JobQueue: Check if node-rdkafka's version on changeprop can be upgraded from 2.8.1 - https://phabricator.wikimedia.org/T341140 (10elukey) Reverted the os + node-rdkafka upgrade, and went for OS upgrade only (stretch to buster). In staging the CPU dropped and the network usag... [12:45:45] hnowlan: https://phabricator.wikimedia.org/T341140#9050963 - it is definitely node-rdkafka [12:46:25] I propose to upgrade to buster anyway, and keep checking what's the issue [13:30:53] also all unit tests pass with nodejs-12 on bullseye :D [13:31:24] I'll also try node14 [13:46:44] ok we don't have nodejs14-devel, filed a code change [13:47:01] I'd be curious to test the new node-rdkafka package with nodejs >= 12 [13:47:36] worst case we keep the current version of node-rdkafka and bring changeprop to bullseye [13:47:39] thoughts? [14:18:05] elukey: yeah I'd say go to buster on monday [14:18:17] the network traffic going up is particularly odd though [14:18:24] not like there's a whole load of stuff happening on staging yknow? [14:18:38] yeah I agree [14:19:17] it may be that the new client runs ok on recent versions of node [14:19:25] meanwhile on 10 it has a huge penalty [14:19:43] yeah, some kind of polling/callback mechanism that 10 can't handle [14:19:57] the major time (from perf) seems spent on core libs [14:20:16] what do you think about trying node12 after buster? [14:20:21] all unit tests pass [14:20:34] even 14 could be on the table [14:21:21] (lol nodejs 10 was released 5 years ago, 12 4 and it is already EOLed [14:21:24] ) [14:29:09] yeah, very in favour [14:29:30] Petr had some uncertainty about it at first but I don't think it was rooted in anything specific [14:53:52] 10serviceops, 10Data Products, 10RESTbase Sunsetting, 10Code-Health-Objective, 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10Miriam) [15:08:51] 10serviceops, 10ChangeProp, 10WMF-JobQueue, 10Patch-For-Review: Check if node-rdkafka's version on changeprop can be upgraded from 2.8.1 - https://phabricator.wikimedia.org/T341140 (10elukey) Next steps: * roll out changeprop on buster and node10 * test changeprop on bullseye + node12 in staging, and roll... [17:33:26] 10serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10SRE, and 5 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10lmata) [17:44:12] 10serviceops: Allow for multiple confd instances in puppet - https://phabricator.wikimedia.org/T341669 (10Kappakayala)