[00:20:41] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:25:36] (03open) 10bd808: dev: Update dependencies with `cargo update` [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/6 (https://phabricator.wikimedia.org/T380023) [00:27:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [00:37:45] (03merge) 10bd808: dev: Update dependencies with `cargo update` [toolforge-repos/ircservserv] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv/-/merge_requests/6 (https://phabricator.wikimedia.org/T380023) [02:27:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:20:41] FIRING: CloudVPSDesignateLeaks: Detected 40 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:27:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [06:27:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:52:49] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [07:52:58] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [08:20:41] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:27:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [08:34:44] (03open) 10taavi: proxy-codfw1dev: Permit management traffic from cloudweb2002 v6 address [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/131 [08:35:31] (03merge) 10taavi: proxy-codfw1dev: Permit management traffic from cloudweb2002 v6 address [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/131 [08:35:39] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [08:37:07] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:27:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:05:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:06:38] 10Tools, 06Commons: Intermittent failure of FlickreviewR 2 bot - https://phabricator.wikimedia.org/T379459#10328897 (10zhuyifei1999) > I am not sure whether this is related. That one is not related. It was specific with retrieving the list of global bot wikis. [11:07:16] 10Tools, 06Commons: Intermittent failure of FlickreviewR 2 bot - https://phabricator.wikimedia.org/T379459#10328895 (10zhuyifei1999) a:03zhuyifei1999 I got an email from 廣九直通車 about this. I'll take a look later today or tomorrow. [11:15:43] (03PS1) 10Majavah: Restore key management panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1091851 (https://phabricator.wikimedia.org/T373082) [11:16:22] (03PS1) 10Majavah: Restore key management panel [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1091852 (https://phabricator.wikimedia.org/T373082) [11:16:34] (03CR) 10Majavah: [V:03+2 C:03+2] Restore key management panel [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1091852 (https://phabricator.wikimedia.org/T373082) (owner: 10Majavah) [11:16:42] (03CR) 10Majavah: [V:03+2 C:03+2] Restore key management panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1091851 (https://phabricator.wikimedia.org/T373082) (owner: 10Majavah) [11:25:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:53:31] (03PS1) 10Majavah: fix stylesheet location [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1091853 [11:55:27] (03CR) 10Majavah: [C:03+2] fix stylesheet location [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1091853 (owner: 10Majavah) [11:55:51] (03Merged) 10jenkins-bot: fix stylesheet location [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1091853 (owner: 10Majavah) [11:58:06] 06cloud-services-team, 10Cloud-VPS: [ceph] osd daemon died - 2024-11-06 - https://phabricator.wikimedia.org/T379142#10328949 (10taavi) [12:15:50] 10Horizon, 05Cloud-Services-Origin-User: Horizon nova generated ssh key undeletable - https://phabricator.wikimedia.org/T373082#10328953 (10taavi) a:03taavi This will be fixed with the next Horizon deploy in eqiad1 by adding a new "key pairs" panel under the compute tab [12:18:05] 06cloud-services-team, 10Horizon: Allow creating Designate subdomains for domains already controlled by a project - https://phabricator.wikimedia.org/T131301#10328957 (10taavi) [12:20:41] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:25:03] 10Tools, 06Commons: Intermittent failure of FlickreviewR 2 bot - https://phabricator.wikimedia.org/T379459#10328960 (10KTT-Commons) [12:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [12:35:31] (03PS1) 10Majavah: Move panel to 'network' section [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091855 [12:35:32] (03PS1) 10Majavah: Add data-display to delete confirmation popup [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091856 (https://phabricator.wikimedia.org/T238711) [12:37:54] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Preserve formatting and comments etc. in ENC Hiera - https://phabricator.wikimedia.org/T250622#10328967 (10taavi) [12:38:03] 06cloud-services-team, 10Cloud-VPS, 07Puppet: Preserve formatting and comments etc. in ENC Hiera - https://phabricator.wikimedia.org/T250622#10328969 (10taavi) [12:40:28] 06cloud-services-team, 10Horizon: Feature request: persistent comments in Cloud Horizon hiera configuration - https://phabricator.wikimedia.org/T287395#10328965 (10taavi) →14Duplicate dup:03T250622 [12:52:41] (03CR) 10Majavah: [C:03+2] Move panel to 'network' section [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091855 (owner: 10Majavah) [12:53:13] (03Merged) 10jenkins-bot: Move panel to 'network' section [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091855 (owner: 10Majavah) [12:55:33] (03CR) 10Majavah: [C:03+2] Add data-display to delete confirmation popup [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091856 (https://phabricator.wikimedia.org/T238711) (owner: 10Majavah) [12:56:04] (03Merged) 10jenkins-bot: Add data-display to delete confirmation popup [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091856 (https://phabricator.wikimedia.org/T238711) (owner: 10Majavah) [13:10:07] (03PS1) 10Majavah: Just use get_object_display_key [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091858 (https://phabricator.wikimedia.org/T238711) [13:10:30] (03CR) 10Majavah: [C:03+2] Just use get_object_display_key [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091858 (https://phabricator.wikimedia.org/T238711) (owner: 10Majavah) [13:11:00] (03Merged) 10jenkins-bot: Just use get_object_display_key [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091858 (https://phabricator.wikimedia.org/T238711) (owner: 10Majavah) [13:26:32] (03PS1) 10Majavah: Fix method definition [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091859 [13:26:40] (03CR) 10Majavah: [C:03+2] Fix method definition [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091859 (owner: 10Majavah) [13:27:14] (03Merged) 10jenkins-bot: Fix method definition [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1091859 (owner: 10Majavah) [13:50:28] 10Horizon, 05Cloud-Services-Origin-User: Horizon nova generated ssh key undeletable - https://phabricator.wikimedia.org/T373082#10329010 (10taavi) 05Open→03Resolved [13:52:03] 10Horizon: Horizon proxy deletion UI has blank where we should name the proxy we're deleting - https://phabricator.wikimedia.org/T238711#10329008 (10taavi) 05Open→03Resolved a:03taavi [13:54:47] 10Striker, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380114 (10taavi) 03NEW [13:54:48] 10Striker, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380115 (10taavi) 03NEW [13:54:52] 10Striker, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380114#10329034 (10taavi) [13:55:30] 10Striker, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380115#10329032 (10taavi) →14Duplicate dup:03T380114 [14:27:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:25:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:21:19] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:27:36] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [17:04:23] 10Tools, 06Commons: Intermittent failure of FlickreviewR 2 bot - https://phabricator.wikimedia.org/T379459#10329084 (10zhuyifei1999) 05Open→03Resolved Caused by Flickr adding a rate limit. Tracked on-wiki: https://commons.wikimedia.org/w/index.php?title=Commons:Village_pump/Technical&oldid=957845988#Us... [18:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [19:25:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:40:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.989% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:55:28] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate wikilabels-staging-02.wikilabels.eqiad.wmflabs is about to expire in 27d 23h 58m 10s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [20:00:28] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate wikilabels-backups-01.wikilabels.eqiad.wmflabs is about to expire in 27d 23h 54m 26s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [20:20:42] FIRING: CloudVPSDesignateLeaks: Detected 40 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [20:36:59] (03open) 10bd808: Update irc config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/16 [20:40:59] (03merge) 10bd808: Update irc config [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/16 [22:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:25:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [23:40:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.511% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:54:17] 06cloud-services-team, 10Toolforge: [builds-builder] Support adding repositories for Apt buildpack - https://phabricator.wikimedia.org/T363027#10329408 (10bd808) [23:54:54] 06cloud-services-team, 10Toolforge: [builds-builder] Support using custom buildpacks - https://phabricator.wikimedia.org/T363033#10329410 (10bd808)