[07:43:26] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10elukey) [07:43:50] changeprop upgrade finished! [07:44:05] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 9 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10elukey) [07:44:31] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10elukey) 05Open→03Resolved a:03elukey Beta deployed by @Ottomata as part of T351247 (thanks!) [08:15:08] 10serviceops, 10MediaWiki-Documentation, 10Documentation, 10Patch-Needs-Improvement, 10User-Dereckson: Repair "svn.wikimedia.org/doc/" redirect for doc.wikimedia.org - https://phabricator.wikimedia.org/T109950 (10Aklapper) a:05Dereckson→03None @Dereckson: Per emails from Sep18 and Oct20 and https://w... [08:18:51] 10serviceops, 10SRE: Update grafana link for mediawiki-error-rate-$cluster in icinga check - https://phabricator.wikimedia.org/T281261 (10Aklapper) a:05jijiki→03None @jijiki: Per emails from Sep18 and Oct20 and https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup , I am resetting the assignee of... [08:19:21] 10serviceops, 10SRE, 10User-jbond, 10User-jijiki: Refactor memcached modules - https://phabricator.wikimedia.org/T284454 (10Aklapper) a:05jijiki→03None @jijiki: Per emails from Sep18 and Oct20 and https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup , I am resetting the assignee of this task... [08:19:27] 10serviceops, 10envoy, 10observability, 10Patch-Needs-Improvement: Envoy should listen on ipv6 and ipv4 - https://phabricator.wikimedia.org/T255568 (10Aklapper) a:05jijiki→03None @jijiki: Per emails from Sep18 and Oct20 and https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup , I am resetting... [08:20:25] 10serviceops, 10SRE, 10Datacenter-Switchover: Document communication expectations around planning a DC switchover - https://phabricator.wikimedia.org/T285806 (10Aklapper) a:05Legoktm→03None @Legoktm: Per emails from Sep18 and Oct20 and https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup , I am... [08:29:00] 10serviceops, 10SRE, 10Datacenter-Switchover: Document communication expectations around planning a DC switchover - https://phabricator.wikimedia.org/T285806 (10Joe) 05Open→03Resolved a:03Joe Given we now have switchovers at regular intervals, we can resolve this task. There is no need to do a lot of c... [10:51:55] 10serviceops, 10Prod-Kubernetes: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) p:05Triage→03High [10:52:07] 10serviceops, 10Prod-Kubernetes: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) [10:54:16] 10serviceops, 10Prod-Kubernetes: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) [11:11:37] 10serviceops, 10Prod-Kubernetes: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) [11:18:20] 10serviceops, 10Prod-Kubernetes, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) Hey DCOps, this looks suspiciously like a cable might have been pulled. Could you please take a look? [11:18:42] 10serviceops, 10Prod-Kubernetes, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) a:05JMeybohm→03Papaul [11:23:18] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10JMeybohm) KSM in staging-eqiad was in a half installed state (probably due to prematurely terminated helmfile/helm command): ` (HelmReleaseBadStatus)... [11:32:37] 10serviceops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10JMeybohm) [11:38:05] 10serviceops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=771e4f70-9348-49e4-9f8a-1228c0c3d3dc) set by jayme@cumin1001 for 1 day, 0:00:00 on 1 host(s) and their ser... [13:54:15] 10serviceops, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Upgrade change propagation to nodejs18 - https://phabricator.wikimedia.org/T348950 (10Ottomata) I only deployed changeprop, not cpjobqueue. [14:13:41] 10serviceops, 10Prod-Kubernetes, 10SRE, 10ops-codfw: kubernetes2041.codfw.wmnet NotReady - https://phabricator.wikimedia.org/T351704 (10Jhancock.wm) a:05Papaul→03Jhancock.wm the network cable was still attached but loose. I reseated it and pulled on it to make sire it wouldn't come loose again. it did... [14:57:52] if you have a min https://gerrit.wikimedia.org/r/c/operations/puppet/+/976241/ [14:57:57] (move kafka-main1001 to puppet 7) [15:58:58] elukey: fine by me - don't break to many things :p [16:12:26] done! [16:13:03] tomorrow I'll migrate the rest [16:33:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review, 10User-jijiki: Deploy kube-state-metrics - https://phabricator.wikimedia.org/T264625 (10kamila) >>! In T264625#9348475, @JMeybohm wrote: > KSM in staging-eqiad was in a half installed state (probably due to prematurely terminated helmfile... [17:01:42] elukey: do you know 'who' from serviceops I should get to review? I've asked around and so far not had any takers [17:16:04] 10serviceops, 10MediaWiki-Documentation, 10Documentation, 10Patch-Needs-Improvement, 10User-Dereckson: Repair "svn.wikimedia.org/doc/" redirect for doc.wikimedia.org - https://phabricator.wikimedia.org/T109950 (10Dzahn) This is still blocked on https://gerrit.wikimedia.org/r/c/operations/puppet/+/631888....