[08:22:49] !log tools reboot tools-sgeweblight-10-14, 24 T349425 [08:22:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:22:54] T349425: CephSlowOps Ceph cluster in has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T349425 [09:49:30] !log deployment-prep turn off deployment-prometheus05 - T344974 [09:49:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [09:49:37] T344974: De-provision beta-specific Prometheus - https://phabricator.wikimedia.org/T344974 [09:54:55] I have switched alerting from cloudmetrics to prometheus hosts for cloud prometheus, please let me know if anything is amiss [09:56:01] taavi: I can't recall ATM, how is prometheus on cloudmetrics hosts accessed e.g. via which grafana ? [09:56:47] godog: only grafana.wikimedia.org has access to that prometheus instance [09:57:14] so that'd be https://gerrit.wikimedia.org/g/operations/puppet/+/f0b544b2c5832efa54f2f72b58d1ae5b5cb1a0a3/modules/profile/files/grafana/production-datasources.yaml#107 [09:57:46] oh wow labmon, blast from the past [09:57:53] that explains why I couldn't find it [09:58:01] thank you, I'll send a followup patch [09:58:56] :-) I have a bunch of code that I get to clean up now that the cloudmetrics boxes will be unused [09:59:21] feels good [10:19:34] godog: https://phabricator.wikimedia.org/T349490 [10:20:53] taavi: ack, taking a look [10:23:28] looks like it was transient, updating the task [10:36:14] !log admin merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config [10:36:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:18:06] !log tools release toolforge-builds-cli 0.0.4 [14:18:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:09:02] Hello! In https://wikitech.wikimedia.org/wiki/Incidents/2023-09-29_CloudVPS_vms_losing_network_connectivity#Actionables there's one task missing an assignee: https://phabricator.wikimedia.org/T347681 - Would someone like to take it or shall I move the "all actionables assigned?" to "no"? [15:10:32] brett: I'll claim it, one sec [15:10:43] Many thanks [15:12:08] brett: I got a question though, if I understand correctly, the incident review will/might create more actionables right? [15:12:32] It might! [15:12:44] Just didn't want this one to slip through the cracks [15:13:05] ack [18:35:00] !log paws deploy new cluster/jupyterhub chart T349545 [18:35:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [18:35:05] T349545: update z2jh chart to 3.1.0 - https://phabricator.wikimedia.org/T349545 [19:40:29] !log wikisp Deleted ceres-01 - T349555 [19:40:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [19:40:33] T349555: Desmantelar ceres-01 - https://phabricator.wikimedia.org/T349555