[00:02:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add
[00:02:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
[00:03:09] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add
[00:03:10] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
[00:03:25] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add
[00:03:33] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
[00:05:31] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[00:05:44] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[00:06:20] <wmcs-alerts>	 FIRING: PrometheusK8sCertExpirySoon: Prometheus k8s certificate is about to expire - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/PrometheusK8sCertExpirySoon  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusK8sCertExpirySoon
[00:06:42] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[00:06:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[00:07:08] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.drain_node
[00:07:56] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
[00:08:36] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[00:08:46] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[00:10:09] <jinxer-wm>	 FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[00:26:54] <jinxer-wm>	 RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[03:19:19] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[03:43:19] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790680 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by andrew@cumin1002 from cloudcontrol2007-dev to cloudrabbit2001-dev complete...
[03:46:30] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790682 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudrabbit2001-dev.codfw.wmnet with OS boo...
[03:50:14] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790685 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by andrew@cumin1002 from cloudcontrol2008-dev to cloudrabbit2002-dev complete...
[03:53:30] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790686 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by andrew@cumin1002 from cloudcontrol2009-dev to cloudrabbit2003-dev complete...
[03:54:48] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790688 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudrabbit2003-dev.codfw.wmnet with OS boo...
[03:54:49] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790687 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudrabbit2002-dev.codfw.wmnet with OS boo...
[04:29:09] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790694 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudrabbit2001-dev.codfw.wmnet with OS bookwor...
[04:31:42] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw: Update labels on cloudcontrol200[789]-dev.codfw - https://phabricator.wikimedia.org/T393347 (10Andrew) 03NEW
[04:35:05] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790706 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudrabbit2003-dev.codfw.wmnet with OS bookwor...
[04:38:37] <wikibugs>	 06cloud-services-team, 13Patch-For-Review: Rename cloudcontrol200[789]-dev.codfw to cloudrabbit200[123]-dev.codfw - https://phabricator.wikimedia.org/T392539#10790707 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudrabbit2002-dev.codfw.wmnet with OS bookwor...
[07:44:00] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[07:49:46] <jinxer-wm>	 FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[07:59:45] <jinxer-wm>	 FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:09:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:15:00] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:19:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:19:43] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
[08:19:56] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:24:41] <jinxer-wm>	 FIRING: [4x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:34:41] <jinxer-wm>	 FIRING: [3x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[08:49:41] <jinxer-wm>	 RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted  - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[10:19:58] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy
[10:21:11] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:18:28] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[11:44:00] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[11:45:41] <jinxer-wm>	 FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[12:00:41] <jinxer-wm>	 FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[12:25:41] <jinxer-wm>	 RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[12:40:41] <jinxer-wm>	 FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[12:41:11] <icinga-wm>	 PROBLEM - nova-compute proc minimum on cloudvirt1046 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/pytho[n].* /usr/bin/nova-compute https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
[12:42:11] <icinga-wm>	 RECOVERY - nova-compute proc minimum on cloudvirt1046 is OK: PROCS OK: 1 process with regex args ^/usr/bin/pytho[n].* /usr/bin/nova-compute https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
[12:46:02] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
[12:50:28] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025-05-02 - 2025-05-23): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10791880 (10Gehel)
[12:50:41] <jinxer-wm>	 FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[12:55:56] <wikibugs>	 (03merge) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 (https://phabricator.wikimedia.org/T390056)
[13:05:41] <jinxer-wm>	 FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[13:15:41] <jinxer-wm>	 RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted
[13:44:27] <wikibugs>	 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Set up a bitu instance for codfw1dev - https://phabricator.wikimedia.org/T360795#10792270 (10SLyngshede-WMF) 05Open→03Resolved To avoid having to maintain another installation of Bitu, we have opted for a small t...
[13:44:59] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: 1 VM for codfw1dev bitu deployment - https://phabricator.wikimedia.org/T362128#10792278 (10SLyngshede-WMF) 05Open→03Invalid
[14:15:09] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Add an option to cookbooks to specify which hosts should be targeted - https://phabricator.wikimedia.org/T393387 (10fnegri) 03NEW
[14:18:33] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Alert when views are out of sync - https://phabricator.wikimedia.org/T393388 (10fnegri) 03NEW
[14:18:55] <wikibugs>	 10Data-Services: add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#10792498 (10fnegri)
[14:18:56] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Alert when views are out of sync - https://phabricator.wikimedia.org/T393388#10792497 (10fnegri)
[14:22:41] <wikibugs>	 (03open) 10chuckonwumelu: Revert "Importing tools dns resources" [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/15
[14:37:00] <wikibugs>	 (03open) 10chuckonwumelu: Bug: T390056 Fixing service account permissions [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/16
[14:39:29] <wikibugs>	 (03update) 10chuckonwumelu: Bug: T390056 Fixing service account permissions [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/16
[14:49:31] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Update labels on cloudcontrol200[789]-dev.codfw - https://phabricator.wikimedia.org/T393347#10792749 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm We just need to change the external labels on the server. This has been done. Thank you for t...
[14:56:50] <wikibugs>	 (03approved) 10chuckonwumelu: toolsbeta: add toolsbeta.org zone and a record pointing to the bastion [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/14 (https://phabricator.wikimedia.org/T362253) (owner: 10andrew)
[14:57:15] <wikibugs>	 (03approved) 10chuckonwumelu: Bug: T390056 Fixing service account permissions [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/16
[14:57:34] <wikibugs>	 (03merge) 10chuckonwumelu: Bug: T390056 Fixing service account permissions [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/16
[15:00:13] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add
[15:03:49] <wikibugs>	 (03update) 10chuckonwumelu: toolsbeta: add toolsbeta.org zone and a record pointing to the bastion [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/14 (https://phabricator.wikimedia.org/T362253) (owner: 10andrew)
[15:06:31] <wikibugs>	 (03merge) 10chuckonwumelu: toolsbeta: add toolsbeta.org zone and a record pointing to the bastion [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/14 (https://phabricator.wikimedia.org/T362253) (owner: 10andrew)
[15:13:28] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[15:14:53] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10792837 (10fnegri) Thanks @BTullis, I think there is e...
[15:16:29] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10792846 (10fnegri)
[15:16:58] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Alert when views are out of sync - https://phabricator.wikimedia.org/T393388#10792851 (10fnegri)
[15:17:00] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10792850 (10fnegri)
[15:17:02] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Add an option to cookbooks to specify which hosts should be targeted - https://phabricator.wikimedia.org/T393387#10792852 (10fnegri)
[15:25:50] <wikibugs>	 06cloud-services-team: deployment-prep displaying error message on all pages - https://phabricator.wikimedia.org/T393393 (10Andrew) 03NEW
[15:26:20] <wikibugs>	 06cloud-services-team, 10Beta-Cluster-Infrastructure: deployment-prep displaying error message on all pages - https://phabricator.wikimedia.org/T393393#10792896 (10Andrew)
[15:36:01] <wikibugs>	 06cloud-services-team, 10Beta-Cluster-Infrastructure: deployment-prep displaying error message on all pages - https://phabricator.wikimedia.org/T393393#10792949 (10Andrew) It seems to fail only for certain users. @dancy can see pages just fine, but all I can see is  {F59701002}
[15:44:00] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[17:32:42] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10793398 (10VRiley-WMF) a:03VRiley-WMF
[17:41:14] <wikibugs>	 (03PS1) 10Herron: add dummy write group for testing [labs/private] - 10https://gerrit.wikimedia.org/r/1141944
[17:43:15] <wikibugs>	 (03CR) 10Herron: [V:03+2 C:03+2] add dummy write group for testing [labs/private] - 10https://gerrit.wikimedia.org/r/1141944 (owner: 10Herron)
[17:45:19] <wikibugs>	 (03open) 10chuckonwumelu: Creating records for new dns zone [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/17
[18:12:47] <wikibugs>	 (03merge) 10chuckonwumelu: Creating records for new dns zone [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/17
[18:25:51] <wikibugs>	 (03open) 10chuckonwumelu: Removed whitespace causing syntax error [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/18
[18:28:16] <wikibugs>	 (03merge) 10chuckonwumelu: Removed whitespace causing syntax error [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/18
[18:29:27] <wikibugs>	 06cloud-services-team, 10Toolforge: Check for non-libre vscode-server installs/processes on Toolforge bastions - https://phabricator.wikimedia.org/T390885#10793594 (10taavi) I was hoping one way to do this would be to null route the domain name where VS Code downloads the server binary from. https://code.visua...
[18:34:22] <jinxer-wm>	 FIRING: HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[19:17:56] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10793728 (10VRiley-WMF)
[19:18:27] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudcephosd100[1-3] - https://phabricator.wikimedia.org/T393196#10793730 (10VRiley-WMF) 05Open→03Resolved These have been decommed
[19:32:55] <wmcs-alerts>	 FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[19:37:55] <wmcs-alerts>	 RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown
[19:38:46] <wikibugs>	 (03update) 10ilanen1: Ilanapi [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/4
[19:42:34] <wikibugs>	 (03merge) 10ilanen1: Ilanapi [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/4
[19:44:01] <jinxer-wm>	 RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:44:30] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:49:30] <jinxer-wm>	 RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:50:30] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:54:15] <jinxer-wm>	 RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[19:55:30] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[20:04:16] <jinxer-wm>	 RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[22:10:01] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759#10794122 (10Andrew)
[22:19:13] <wikibugs>	 (03open) 10chuckonwumelu: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19
[22:31:47] <wikibugs>	 (03update) 10chuckonwumelu: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19
[22:34:22] <jinxer-wm>	 FIRING: HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[22:39:22] <jinxer-wm>	 RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[23:22:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-69 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[23:50:00] <wikibugs>	 (03update) 10chuckonwumelu: Bug: T390056 importing tools volumes to Tofu [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/19