[07:40:39] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11101029 (10Raymond_Ndibe) I believe status messages should be as uniform as possible. If we need to covey extra information it's better to pu...
[07:48:59] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): Disable tools.maintain-harbor - https://phabricator.wikimedia.org/T397933#11101034 (10Raymond_Ndibe)
[07:49:07] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): Disable tools.maintain-harbor - https://phabricator.wikimedia.org/T397933#11101036 (10Raymond_Ndibe) 05Open→03Resolved
[08:18:12] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11101063 (10dcaro)
[08:30:36] <wikibugs>	 (03update) 10dcaro: runtime.k8s.diff_with_running_job: use internal get_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/197
[08:36:15] <wikibugs>	 (03update) 10dcaro: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201
[08:39:15] <wikibugs>	 (03approved) 10dcaro: wmcs-k8s-metrics: update to support k8s v1.30 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/929
[08:39:21] <wikibugs>	 (03update) 10dcaro: wmcs-k8s-metrics: update to support k8s v1.30 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/929
[08:40:06] <wikibugs>	 (03merge) 10dcaro: wmcs-k8s-metrics: update to support k8s v1.30 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/929
[09:01:06] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.ceph.reboot_node (T401319)
[09:01:13] <stashbot>	 T401319: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319
[09:04:23] <icinga-wm>	 PROBLEM - Host cloudcephosd1013 is DOWN: PING CRITICAL - Packet loss = 100%
[09:04:53] <icinga-wm>	 RECOVERY - Host cloudcephosd1013 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[09:05:20] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) (T401319)
[09:10:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 07IPv6: Refresh Cloud VPS Puppet ENC servers to run on Trixie and enable IPv6 - https://phabricator.wikimedia.org/T401986#11101252 (10taavi) 05Open→03Resolved
[09:10:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 07IPv6: Refresh Cloud VPS NTP servers to run on Trixie and enable IPv6 - https://phabricator.wikimedia.org/T401848#11101270 (10taavi) 05Open→03Resolved
[09:15:39] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319#11101301 (10fnegri) 05Resolved→03Open @Jclark-ctr the new drive does not show up in `lsblk`, I tried rebooting but that didn'...
[09:16:28] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on cloudinfra-internal-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[09:26:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-etcd-22 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:27:27] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11101339 (10dcaro)
[09:29:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:29:47] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 (10dcaro) 03NEW
[09:30:04] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11101375 (10dcaro)
[09:31:28] <wmcs-alerts>	 FIRING: [31x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:32:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-prometheus-3 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:32:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-app13 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:32:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-mail-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:32:36] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378 (10dcaro) 03NEW
[09:33:08] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11101400 (10dcaro)
[09:33:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:34:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance runner-1037 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:36:28] <wmcs-alerts>	 FIRING: [50x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:37:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:37:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-controller-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:37:28] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-mail-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:37:32] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:38:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-cloudvps-puppetserver-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:39:28] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:39:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance project-proxy-acme-chief-02 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:40:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:41:28] <wmcs-alerts>	 FIRING: [70x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:41:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance quarry-bastion on project quarry   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:42:28] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:42:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:42:28] <wmcs-alerts>	 FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-mail-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:42:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudvirt2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:43:00] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudvirt2005-dev:9100 - https://phabricator.wikimedia.org/T402380 (10phaultfinder) 03NEW
[09:43:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:44:28] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:44:28] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance project-proxy-acme-chief-02 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:45:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:46:22] <wikibugs>	 (03open) 10vriaa: fix: Reactivity issues in useUpdateSide composable [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/10
[09:46:28] <wmcs-alerts>	 FIRING: [90x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:47:28] <wmcs-alerts>	 FIRING: [9x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:47:28] <wmcs-alerts>	 FIRING: [15x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-docker-imagebuilder-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:47:38] <wikibugs>	 (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe)
[09:47:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cloudvirt2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:48:00] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T402381 (10phaultfinder) 03NEW
[09:48:28] <wmcs-alerts>	 FIRING: [12x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:49:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:49:28] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-6 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:50:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:51:28] <wmcs-alerts>	 FIRING: [102x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:52:10] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11101455 (10Raymond_Ndibe) For some reason we don't seem to be discussing the possibility of making one `toolforge` object store and havi...
[09:52:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:52:28] <wmcs-alerts>	 FIRING: [19x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:52:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cloudvirt2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:53:28] <wmcs-alerts>	 FIRING: [16x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:54:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:56:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance k8s-nfs on project quarry   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:56:28] <wmcs-alerts>	 FIRING: [116x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:57:14] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11101465 (10Raymond_Ndibe) >>! In T358496#11101455, @Raymond_Ndibe wrote: > For some reason we don't seem to be discussing the possibilit...
[09:57:28] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:57:28] <wmcs-alerts>	 FIRING: [24x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-cumin-1 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:57:48] <jinxer-wm>	 RESOLVED: [2x] PuppetFailure: Puppet has failed on cloudvirt2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:58:28] <wmcs-alerts>	 FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:59:28] <wmcs-alerts>	 FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:59:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:00:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:00:35] <wikibugs>	 (03open) 10vriaa: fix: BorderProperty toggle state not syncing with parent updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/11
[10:01:28] <wmcs-alerts>	 FIRING: [116x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:02:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:02:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:02:28] <wmcs-alerts>	 FIRING: [34x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:03:28] <wmcs-alerts>	 FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:04:28] <wmcs-alerts>	 FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:04:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:05:28] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:06:28] <wmcs-alerts>	 FIRING: [116x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:07:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:07:28] <wmcs-alerts>	 FIRING: [34x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:08:28] <wmcs-alerts>	 FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:09:28] <wmcs-alerts>	 FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:11:13] <wikibugs>	 (03open) 10vriaa: fix: MarginProperty and PaddingProperty not syncing with parent state [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/12
[10:11:28] <wmcs-alerts>	 FIRING: [116x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:11:28] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance k8s-nfs on project quarry   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:11:46] <wikibugs>	 (03approved) 10samtar: Migrate CI from GitHub to GitLab [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/1 (https://phabricator.wikimedia.org/T395398) (owner: 10samwilson)
[10:12:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:12:28] <wmcs-alerts>	 FIRING: [34x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:12:33] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11101479 (10dcaro) >>! In T358496#11101465, @Raymond_Ndibe wrote: >>>! In T358496#11101455, @Raymond_Ndibe wrote: >> For some reason we d...
[10:13:28] <wmcs-alerts>	 FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:14:28] <wmcs-alerts>	 FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:14:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:16:28] <wmcs-alerts>	 FIRING: [103x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:17:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:17:28] <wmcs-alerts>	 FIRING: [34x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:18:28] <wmcs-alerts>	 FIRING: [17x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:19:28] <wmcs-alerts>	 FIRING: [10x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:19:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:21:28] <wmcs-alerts>	 FIRING: [83x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:22:28] <wmcs-alerts>	 FIRING: [7x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:22:28] <wmcs-alerts>	 FIRING: [6x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:22:28] <wmcs-alerts>	 FIRING: [29x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:22:32] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:23:28] <wmcs-alerts>	 FIRING: [11x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:23:58] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11101505 (10Raymond_Ndibe) >>! In T358496#11101479, @dcaro wrote: >>>! In T358496#11101465, @Raymond_Ndibe wrote: >>>>! In T358496#111014...
[10:24:28] <wmcs-alerts>	 RESOLVED: [9x] PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:24:28] <wmcs-alerts>	 RESOLVED: [5x] PuppetAgentNoResources: No Puppet resources found on instance maps-proxy-5 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:25:28] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:26:28] <wmcs-alerts>	 RESOLVED: [71x] PuppetAgentNoResources: No Puppet resources found on instance abogott-nstesting on project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:26:28] <wmcs-alerts>	 RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance k8s-nfs on project quarry   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:27:03] <wikibugs>	 (03open) 10vriaa: fix: Sync toggle with colorModel changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/13
[10:27:28] <wmcs-alerts>	 RESOLVED: [7x] PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-alertmanager-2 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:27:28] <wmcs-alerts>	 RESOLVED: [6x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache11 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:27:28] <wmcs-alerts>	 FIRING: [25x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:28:28] <wmcs-alerts>	 RESOLVED: [8x] PuppetAgentNoResources: No Puppet resources found on instance chartmuseum-2 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:32:28] <wmcs-alerts>	 RESOLVED: [18x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-acme-chief-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[10:35:19] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2025.08.16 - 2025.09.05): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11101682 (10BTullis)
[10:35:41] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 2 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11101690 (10BTullis)
[10:37:43] <wikibugs>	 (03open) 10vriaa: fix: ColorPicker component ignoring modelValue updates [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/14
[10:48:26] <wikibugs>	 (03open) 10vriaa: fix: BasicDropdown not updating when modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/15
[10:50:14] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#11101783 (10Jclark-ctr) @cmooney  Would you be able to assist with setting up eth1 link? servers are already imaged on eth0. I believe this ticket will be...
[11:10:12] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#11101852 (10cmooney) @Jclark-ctr that is done now and all four host's second ports are connected and running at 25G now.  @Andrew you can proceed to set th...
[11:10:56] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#11101863 (10Jclark-ctr) 05Open→03Resolved
[11:12:56] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319#11101871 (10Jclark-ctr) 05Open→03Resolved
[11:27:29] <wikibugs>	 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: hw troubleshooting: disk sdj failure for cloudcephosd1013.eqiad.wmnet - https://phabricator.wikimedia.org/T401319#11101921 (10Jclark-ctr) Removed new drive it was causing errors
[11:42:51] <wikibugs>	 (03open) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[11:56:30] <wikibugs>	 (03open) 10vriaa: fix: AddLink not syncing with parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/16
[12:05:43] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11101999 (10dcaro) > What am I missing? why is this a bad approach? the upside is that all the problems of managing multiple auth tokens...
[12:07:37] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudvirt2005-dev:9100 - https://phabricator.wikimedia.org/T402380#11102003 (10fgiunchedi) 05Open→03Invalid That was me re: https://phabricator.wikimedia.org/T355145
[12:07:43] <wikibugs>	 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T402381#11102008 (10fgiunchedi) 05Open→03Invalid That was me re: https://phabricator.wikimedia.org/T355145
[12:11:01] <wikibugs>	 (03open) 10vriaa: fix: useSelectIconButtonOption not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/17
[12:13:05] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[12:24:51] <wikibugs>	 (03open) 10vriaa: fix: TextTransform dropdown not responding to parent changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/18
[12:29:21] <wikibugs>	 (03open) 10vriaa: fix: BannerLink not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/19
[12:34:05] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102063 (10fgiunchedi) Update from a meet yesterday: the VMs for trove, magnum, etc are indeed expected to be unreachable. Followup is T402317
[12:35:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102066 (10fgiunchedi)
[12:53:12] <wikibugs>	 (03merge) 10samtar: Migrate CI from GitHub to GitLab [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/1 (https://phabricator.wikimedia.org/T395398) (owner: 10samwilson)
[13:16:29] <wikibugs>	 (03Abandoned) 10Hashar: docs: clarify callback URL setup for production and development [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/1179253 (owner: 10Rehan_khan_78)
[13:17:20] <wikibugs>	 (03Abandoned) 10Hashar: Added some unit tests for functions in utils.js in server folder. [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/891983 (owner: 10Kartik7120)
[13:17:23] <wikibugs>	 (03Abandoned) 10Hashar: Added Unit Tests for UrlBox [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/896416 (owner: 10Pratham Vaidya)
[13:17:28] <wikibugs>	 (03Abandoned) 10Hashar: Add a Back To Home button on Results page [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/897942 (owner: 10KunalKatara12)
[13:17:31] <wikibugs>	 (03Abandoned) 10Hashar: Add maximum limit for each option [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/900832 (https://phabricator.wikimedia.org/T283415) (owner: 10Reputation22)
[13:17:34] <wikibugs>	 (03Abandoned) 10Hashar: Added Notification in UrlBox Validation and updated Unit Test for it [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/896454 (owner: 10Pratham Vaidya)
[13:17:37] <wikibugs>	 (03Abandoned) 10Hashar: Added some unit tests for Sidebar component [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/896044 (owner: 10Kartik7120)
[13:17:40] <wikibugs>	 (03Abandoned) 10Hashar: Test: added test for UrlBox component [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/900352 (https://phabricator.wikimedia.org/T328928) (owner: 10AFZL210)
[13:17:47] <wikibugs>	 (03Abandoned) 10Hashar: Upgraded the Resultpage UI [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/953251 (owner: 10Punith.nyk)
[13:17:49] <wikibugs>	 (03Abandoned) 10Hashar: Persist VCT on refresh [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/956893 (owner: 10Punith.nyk)
[13:17:52] <wikibugs>	 (03Abandoned) 10Hashar: Create a notification when Internet connection is lost [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/892381 (https://phabricator.wikimedia.org/T330601) (owner: 10Punith.nyk)
[13:17:55] <wikibugs>	 (03Abandoned) 10Hashar: Mpg videos not allowed [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/959795 (owner: 10Vibhoo24)
[13:17:57] <wikibugs>	 (03Abandoned) 10Hashar: Add support to pass category in url params [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/955835 (https://phabricator.wikimedia.org/T344477) (owner: 10Reputation22)
[13:18:00] <wikibugs>	 (03Abandoned) 10Hashar: Add a Back To Home button on Results page [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/897934 (https://phabricator.wikimedia.org/T331954) (owner: 10KunalKatara12)
[13:18:09] <wikibugs>	 (03Abandoned) 10Hashar: Overwrite file text field is now same as the original file [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/902053 (https://phabricator.wikimedia.org/T331235) (owner: 10Najni Fatima)
[13:18:17] <wikibugs>	 (03Abandoned) 10Hashar: Add allow user to paste video from clipboard [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/904603 (https://phabricator.wikimedia.org/T332988) (owner: 10AFZL210)
[13:18:21] <wikibugs>	 (03Abandoned) 10Hashar: Adds dynamic language list to dropdown [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/904758 (https://phabricator.wikimedia.org/T272240) (owner: 10Reputation22)
[13:18:25] <wikibugs>	 (03Abandoned) 10Hashar: Add tests for Header component [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/910843 (https://phabricator.wikimedia.org/T328928) (owner: 10Reputation22)
[13:18:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Ensure unique machine-id across Cloud VPS VMs - https://phabricator.wikimedia.org/T401880#11102190 (10fgiunchedi) 05Open→03Resolved
[13:18:33] <wikibugs>	 (03Abandoned) 10Hashar: Add support to notify users on video upload [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/902893 (https://phabricator.wikimedia.org/T234783) (owner: 10Reputation22)
[13:18:37] <wikibugs>	 (03Abandoned) 10Hashar: Fix allowed extensions from choosing file menu [labs/tools/VideoCutTool] - 10https://gerrit.wikimedia.org/r/925743 (owner: 10Reputation22)
[13:19:32] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:19:34] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:19:48] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:19:50] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:20:06] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:20:08] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:20:39] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:20:42] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:21:00] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:21:02] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:22:30] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:22:33] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[13:22:41] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[13:23:06] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
[13:27:26] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Pontoon, 13Patch-For-Review: Add ability to filter certain sssd users in Pontoon - https://phabricator.wikimedia.org/T402261#11102228 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Done, we can generalize the feature outside of Pontoon if/when needed
[13:34:28] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-harbor-1 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:34:42] <wikibugs>	 (03PS1) 10Brouberol: Add dummy analytics-ml.keytab files to the stat hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1180563
[13:35:24] <wikibugs>	 (03CR) 10Brouberol: [C:03+2] Add dummy analytics-ml.keytab files to the stat hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1180563 (owner: 10Brouberol)
[13:35:26] <wikibugs>	 (03CR) 10Brouberol: [V:03+2 C:03+2] Add dummy analytics-ml.keytab files to the stat hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1180563 (owner: 10Brouberol)
[13:38:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:44:57] <wikibugs>	 (03Restored) 10Hashar: Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery)
[13:45:07] <wikibugs>	 (03Abandoned) 10Hashar: Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery)
[13:45:11] <wikibugs>	 (03Restored) 10Hashar: Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery)
[13:45:20] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery)
[13:45:24] <wikibugs>	 (03CR) 10Hashar: [V:03+2 C:03+2] Archive repo [labs/tools/dawiki] - 10https://gerrit.wikimedia.org/r/1047146 (https://phabricator.wikimedia.org/T270105) (owner: 10Pppery)
[14:09:54] <wikibugs>	 06cloud-services-team, 10Toolforge: Check alerting for Loki - https://phabricator.wikimedia.org/T400981#11102392 (10fnegri) p:05Triage→03Medium
[14:10:54] <wikibugs>	 06cloud-services-team, 10Toolforge: Job not restarting despite liveness probe failures - https://phabricator.wikimedia.org/T400957#11102395 (10fnegri) @Sakretsu sorry for the late reply. Are you still experiencing this issue?
[14:11:41] <wikibugs>	 06cloud-services-team, 10PAWS: Grant membership in cloud-vps project 'PAWS' to vivian rook for volunteer work - https://phabricator.wikimedia.org/T400733#11102396 (10Andrew) p:05Triage→03Medium
[14:12:11] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[42-47] in service - https://phabricator.wikimedia.org/T401693#11102397 (10Andrew) p:05Triage→03Medium
[14:12:27] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: [upstream] [openstack] Fix capi-helm magnum driver to support more template options - https://phabricator.wikimedia.org/T402232#11102398 (10Andrew) p:05Triage→03Medium
[14:17:06] <wikibugs>	 06cloud-services-team, 10Openstack-Magnum: ssh to cloud-vps 'utility' nodes (magnum, trove, octavia) - https://phabricator.wikimedia.org/T402317#11102402 (10Andrew) p:05Triage→03Medium
[14:20:15] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-api] report status pending until the first container is running - https://phabricator.wikimedia.org/T401069#11102404 (10fnegri) 05Open→03Stalled p:05Triage→03Medium {T401172} relates to this and is currently being worked on, we can revisit this task after that...
[14:21:51] <wikibugs>	 06cloud-services-team, 10Toolforge: Allow limiting exposed port access - https://phabricator.wikimedia.org/T400940#11102409 (10fnegri) p:05Triage→03Medium
[14:22:53] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [builds-builder] review and potentially add cmake buildpack - https://phabricator.wikimedia.org/T401169#11102412 (10fnegri) p:05Triage→03Medium
[14:23:59] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [builds-builder] review and potentially add cmake buildpack - https://phabricator.wikimedia.org/T401169#11102430 (10fnegri) Draft version to review: https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/cmake-buildpack
[14:27:15] <wikibugs>	 06cloud-services-team, 10Striker: 500 Internal Server Error when trying to access ssh keys on toolsadmin - https://phabricator.wikimedia.org/T401318#11102452 (10fnegri) p:05Triage→03Low
[14:28:17] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolforge-deploy] detect deployed cli versions in lima-kilo - https://phabricator.wikimedia.org/T401649#11102459 (10fnegri) @Raymond_Ndibe is this issue still happening on your lima-kilo?
[14:28:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#11102462 (10fnegri) p:05Triage→03Medium
[14:29:38] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] Add a "description" field to the deployment - https://phabricator.wikimedia.org/T401993#11102467 (10dcaro) p:05Triage→03Medium
[14:30:25] <wikibugs>	 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11102469 (10fnegri) p:05Triage→03Medium
[14:34:36] <wikibugs>	 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11102493 (10fnegri) @taavi do you think that the expected usage is fine? Do we have quotas/limits in Loki for how many GB of logs can be stored by a single tool?
[14:35:43] <wikibugs>	 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11102500 (10fnegri) p:05Triage→03High
[14:36:21] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate metricsinfra project off of Bullseye - https://phabricator.wikimedia.org/T401813#11102503 (10fnegri) p:05Triage→03Medium
[14:47:29] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11102660 (10fnegri) p:05Triage→03Medium Changing to "Feature Request", as @bd808 explained we don't automatically update from upstream, so the "latest version" will always be the "late...
[14:49:23] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge jobs logs api returns 404 on no log entries - https://phabricator.wikimedia.org/T401420#11102665 (10fnegri) p:05Triage→03Low
[14:49:58] <wikibugs>	 (03merge) 10fnegri: CI: use trusted runner for build_ci_deb [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/3 (https://phabricator.wikimedia.org/T395266)
[14:50:04] <wikibugs>	 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11102669 (10dcaro) p:05Triage→03Medium
[14:50:13] <wikibugs>	 06cloud-services-team, 10Toolforge: [build service] failure due to transient issue - https://phabricator.wikimedia.org/T401917#11102671 (10dcaro) p:05Triage→03Medium
[14:50:15] <wikibugs>	 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401077#11102672 (10fnegri) p:05Triage→03Medium
[14:50:17] <wikibugs>	 (03open) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[14:50:27] <wikibugs>	 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11102685 (10fnegri) p:05Triage→03Medium
[14:50:45] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[14:50:46] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[14:50:52] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[14:50:53] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[14:50:54] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11102686 (10dcaro) p:05Triage→03High
[14:51:00] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11102687 (10dcaro) p:05Triage→03High
[14:51:02] <wikibugs>	 (03update) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[14:51:02] <wikibugs>	 (03update) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[14:51:09] <wikibugs>	 (03update) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[14:52:12] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki bewwiktionary - https://phabricator.wikimedia.org/T402137#11102693 (10fnegri) 05Open→03Stalled This is blocked on {T402133}
[14:52:31] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki bewwiktionary - https://phabricator.wikimedia.org/T402137#11102698 (10fnegri)
[14:52:43] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki bewwiktionary - https://phabricator.wikimedia.org/T402137#11102700 (10fnegri) p:05Triage→03Low
[14:53:13] <wikibugs>	 06cloud-services-team, 10Data-Services, 10BetaFeatures, 06Data-Engineering, 06DBA: Create view for betafeatures_user_counts table in wiki replicas - https://phabricator.wikimedia.org/T402145#11102701 (10fnegri)
[14:54:32] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge components - support providing ref in webhook - https://phabricator.wikimedia.org/T401388#11102702 (10fnegri) p:05Triage→03Low
[14:56:28] <wikibugs>	 06cloud-services-team, 10Toolforge: https://api.svc.toolforge.org endpoint given in OpenAPI spec returns 403 forbidden errors - https://phabricator.wikimedia.org/T402032#11102714 (10fnegri) p:05Triage→03Low
[14:57:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102715 (10fnegri) 05Open→03In progress p:05Triage→03Medium
[14:58:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102719 (10fnegri) a:03fgiunchedi
[14:59:42] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Utility script to get nova server uuid from VM fqdn - https://phabricator.wikimedia.org/T402407 (10fgiunchedi) 03NEW
[15:00:20] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11102738 (10DamianZaremba) Until there is a plan around upgrading the "latest version in tools", what is the suitable mechanism for dealing with vulnerabilities in the available images?  T...
[15:06:23] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Utility script to get nova server uuid from VM fqdn - https://phabricator.wikimedia.org/T402407#11102756 (10fgiunchedi) p:05Triage→03Medium
[15:11:36] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[15:12:49] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
[15:13:10] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[15:14:53] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
[15:15:39] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102767 (10fgiunchedi)
[15:15:50] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[15:21:38] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[15:22:42] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
[15:22:47] <logmsgbot_cloud>	 !log filippo@cloudcumin1001 o11y START - Cookbook wmcs.openstack.cloudvirt.vm_console
[15:24:16] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Audit and potentially fix VMs not reachable by cloudcumin root key - https://phabricator.wikimedia.org/T402185#11102794 (10fgiunchedi)
[15:27:48] <wikibugs>	 (03update) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[15:29:45] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11102808 (10Raymond_Ndibe) >>! In T358496#11101999, @dcaro wrote: >> What am I missing? why is this a bad approach? the upside is that al...
[15:40:31] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#11102839 (10dcaro) > I wasn't advocating we implement the protocol, that'd be unwise. I was just questioning the underlying assumption. D...
[15:48:33] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[15:50:08] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[15:51:46] <wikibugs>	 (03update) 10raymond-ndibe: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204 (owner: 10dcaro)
[15:51:47] <wikibugs>	 (03approved) 10raymond-ndibe: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204 (owner: 10dcaro)
[15:52:16] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11102896 (10fnegri) @DamianZaremba if you believe there is a serious vulnerability in one of the available images, one that can cause security risks to Toolforge or to a tool running on To...
[15:52:55] <wikibugs>	 (03merge) 10dcaro: deploy: adapt readme and file for local deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/204
[15:53:57] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136)
[15:56:34] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.405-20250820155306-2adf51c5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/933
[16:05:39] <wikibugs>	 (03update) 10raymond-ndibe: runtime.k8s.diff_with_running_job: use internal get_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/197 (owner: 10dcaro)
[16:05:52] <wikibugs>	 (03update) 10raymond-ndibe: runtime.k8s.diff_with_running_job: use internal get_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/197 (owner: 10dcaro)
[16:06:31] <wikibugs>	 (03update) 10raymond-ndibe: runtime.k8s.diff_with_running_job: use internal get_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/197 (owner: 10dcaro)
[16:06:35] <wikibugs>	 (03approved) 10raymond-ndibe: runtime.k8s.diff_with_running_job: use internal get_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/197 (owner: 10dcaro)
[16:07:29] <wikibugs>	 (03update) 10raymond-ndibe: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201 (owner: 10dcaro)
[16:08:02] <wikibugs>	 (03update) 10raymond-ndibe: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201 (owner: 10dcaro)
[16:08:10] <wikibugs>	 (03update) 10raymond-ndibe: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201 (owner: 10dcaro)
[16:09:42] <wikibugs>	 (03update) 10raymond-ndibe: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201 (owner: 10dcaro)
[16:09:43] <wikibugs>	 (03approved) 10raymond-ndibe: test_runtime: refactor a bit [repos/cloud/toolforge/jobs-api] (use_get_job_for_diff) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/201 (owner: 10dcaro)
[16:11:30] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11102981 (10fnegri) > (this appears to be the case with Python, so is perhaps in part to how the pack works)  I double checked if there was a different behaviour for Python builds, but it...
[16:15:13] <wikibugs>	 (03update) 10raymond-ndibe: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203 (owner: 10dcaro)
[16:19:30] <wikibugs>	 (03open) 10vriaa: fix: Image components not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/20
[16:20:37] <wikibugs>	 (03update) 10vriaa: fix: Image components not responding to parent modelValue changes [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/20
[17:21:08] <wikibugs>	 (03update) 10dcaro: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203
[17:24:25] <wikibugs>	 (03update) 10raymond-ndibe: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203 (owner: 10dcaro)
[17:28:16] <wikibugs>	 (03approved) 10raymond-ndibe: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203 (owner: 10dcaro)
[17:28:18] <wikibugs>	 (03update) 10raymond-ndibe: create_job: don't use asyncio directly [repos/cloud/toolforge/jobs-api] (refactor_runtime_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/203 (owner: 10dcaro)
[17:33:50] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11103285 (10dcaro) It will depend on the buildpack itself, some buildpacks allow to install a wider range of versions for the language of choice, some would only support a small range, som...
[17:35:48] <wikibugs>	 (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (avoid_using_asyncio_directly) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe)
[17:38:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:43:56] <wikibugs>	 (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (avoid_using_asyncio_directly) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe)
[17:44:58] <wikibugs>	 (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (avoid_using_asyncio_directly) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe)
[18:26:18] <wikibugs>	 (03update) 10raymond-ndibe: [helm_publish, image_publish]: skip harbor helm and image publish if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595)
[18:42:19] <wikibugs>	 (03PS1) 10Dzahn: add dummy private key for zuul intermediate CA [labs/private] - 10https://gerrit.wikimedia.org/r/1180621 (https://phabricator.wikimedia.org/T401614)
[18:43:19] <wikibugs>	 (03CR) 10Dzahn: [V:03+2 C:03+2] "to go with  I89486e5bafee3d4" [labs/private] - 10https://gerrit.wikimedia.org/r/1180621 (https://phabricator.wikimedia.org/T401614) (owner: 10Dzahn)
[18:47:08] <wikibugs>	 (03CR) 10CDanis: [C:03+1] add dummy private key for zuul intermediate CA [labs/private] - 10https://gerrit.wikimedia.org/r/1180621 (https://phabricator.wikimedia.org/T401614) (owner: 10Dzahn)
[19:05:35] <wikibugs>	 06cloud-services-team, 10Toolforge: [Build service] latest builder has old PHP - https://phabricator.wikimedia.org/T401875#11103567 (10DamianZaremba) If you put `3.13.7` in `.python-version` then the build will report `Using Python version 3.13.7 specified in .python-version` and you get the latest version.  O...
[20:06:22] <wikibugs>	 10Tool-paulina: Add credits to README.md - https://phabricator.wikimedia.org/T402052#11104066 (10Pepe_piton) 05Open→03Resolved Done: da04d2dd11ef649f222489c6eb3e436740d20c4f
[20:14:56] <wikibugs>	 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454 (10Aklapper) 03NEW
[20:20:20] <wikibugs>	 (03PS1) 10Krinkle: docs: Update references to repo after move to Wikimedia Gerrit [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641
[20:20:57] <wikibugs>	 (03PS2) 10Krinkle: docs: Update references to repo after move to Wikimedia Gerrit [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641
[20:21:23] <wikibugs>	 (03CR) 10Krinkle: "check php" [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641 (owner: 10Krinkle)
[20:26:39] <wikibugs>	 (03PS3) 10Krinkle: docs: Update references to repo after move to Wikimedia Gerrit [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641
[20:27:10] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] docs: Update references to repo after move to Wikimedia Gerrit [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641 (owner: 10Krinkle)
[20:27:40] <wikibugs>	 (03Merged) 10jenkins-bot: docs: Update references to repo after move to Wikimedia Gerrit [labs/tools/toolbase] - 10https://gerrit.wikimedia.org/r/1180641 (owner: 10Krinkle)
[20:39:43] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki rkiwiki - https://phabricator.wikimedia.org/T392502#11104181 (10Zabe) 05Stalled→03Open blocker resolved
[20:45:44] <wikibugs>	 (03open) 10raymond-ndibe: [config] support port protocol [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T401994)
[20:54:34] <jinxer-wm>	 FIRING: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.711% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[21:04:54] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: CloudVPSDesignateLeaks + managed_resource_id - https://phabricator.wikimedia.org/T402457#11104293 (10Andrew) p:05Triage→03Medium  Here, for example, is a record that is about to leak as soon as the VM is deleted:  mysql:root@localhost [designate]> select data,managed_res...
[21:07:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: CloudVPSDesignateLeaks + managed_resource_id - https://phabricator.wikimedia.org/T402457#11104297 (10Andrew) So, this is awkward. Whatever caused those stray records in the past isn't happening anymore, but we have a big reserve of strays that are just waiting to make noise...
[21:09:42] <wikibugs>	 10Tool-paulina: Add autocomplete suggestions to search bar using Wikidata wbsearchentities API - https://phabricator.wikimedia.org/T402458 (10JosefAnthony) 03NEW
[21:10:04] <wikibugs>	 10Tool-paulina: Add autocomplete suggestions to search bar using Wikidata wbsearchentities API - https://phabricator.wikimedia.org/T402458#11104310 (10JosefAnthony) a:03JosefAnthony
[21:14:34] <jinxer-wm>	 RESOLVED: DiskSpace: Disk space cloudcontrol1011:9100:/ 6.632% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol1011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
[21:18:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:22:32] <wikibugs>	 (03merge) 10bd808: build_deb fixes [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/85 (https://phabricator.wikimedia.org/T400616)
[21:23:02] <wikibugs>	 06cloud-services-team, 06DC-Ops, 06SRE, 13Patch-For-Review: cloudcephosd10[48-51] service implementation - https://phabricator.wikimedia.org/T395910#11104345 (10Andrew) 05Stalled→03In progress
[21:23:31] <wikibugs>	 (03update) 10bd808: build_deb fixes [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/85 (https://phabricator.wikimedia.org/T400616)
[21:26:28] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on cloudinfra-internal-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[22:03:27] <wikibugs>	 10Tool-paulina: Minor improvements in the copyright status of the work - https://phabricator.wikimedia.org/T400748#11104509 (10Pepe_piton) Fixed the second part of the task: the bug in which, when the copyright status statement had no qualifier 'applies to jurisdiction', Paulina showed a huge list of copyright s...
[22:06:10] <wikibugs>	 10Tool-paulina: Minor improvements in the copyright status of the work - https://phabricator.wikimedia.org/T400748#11104511 (10Pepe_piton) 05In progress→03Resolved
[22:26:06] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[42-47] in service - https://phabricator.wikimedia.org/T401693#11104623 (10Andrew)
[23:15:43] <wikibugs>	 10Cloud-VPS (Quota-requests), 10XTools: Request increased quota for xtools Cloud VPS project - https://phabricator.wikimedia.org/T400853#11104830 (10MusikAnimal) >>! In T400853#11049891, @dcaro wrote: > Done! enjoy :) >  >  > I'll be interested to know how it goes with anubis when you have it running.  Con...