[01:17:56] FIRING: [3x] CloudVPSDesignateLeaks: Detected 68 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:21:16] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:46:01] FIRING: [5x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:16:45] FIRING: [4x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:18:55] FIRING: [3x] CloudVPSDesignateLeaks: Detected 68 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:43:00] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [06:50:09] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [06:55:06] (03open) 10aborrero: py3.11-bookworm-tox*: point to the right builder versions [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/39 [08:16:45] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:31:54] (03PS1) 10Muehlenhoff: Remove obsolete wmflabs dummy certs [labs/private] - 10https://gerrit.wikimedia.org/r/1032713 [08:46:17] 10Toolforge (Toolforge iteration 09): [maintain-kubeusers] Increment default services quota - https://phabricator.wikimedia.org/T362520#9807885 (10taavi) We need to have some quota in place to prevent a misbehaving tool from taking kube-apiserver down by creating hundreds or thousands of unfillable ReplicaSets (... [08:58:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:16:31] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [09:21:45] FIRING: [3x] CloudVPSDesignateLeaks: Detected 68 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:22:47] 10Tool-itwiki, 06Commons: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#9808046 (10valerio.bozzolan) [09:24:54] 10Tool-itwiki, 06Commons: Auto-calculate the bearing (compass) from Wikimedia Commons' photos from two pairs of coordinates (Location + Object location) - https://phabricator.wikimedia.org/T363052#9808051 (10valerio.bozzolan) Untested, but this seems gold, at least for the Locator Tool: https://gis.stackexcha... [09:53:41] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [10:21:07] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:01:45] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [13:03:49] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:21:45] FIRING: [3x] CloudVPSDesignateLeaks: Detected 68 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:52:08] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:57:09] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [13:57:56] (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110) [14:34:21] 10VPS-project-Wikistats: Add dtpwiki to wikistats - https://phabricator.wikimedia.org/T365234#9808919 (10Dzahn) 05Open→03Stalled waiting for T365220 [15:11:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9809037 (10fnegri) 05Open→03In progress [16:22:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 10Quarry: Create db user for Quarry with readonly access to public ToolsDB databases - https://phabricator.wikimedia.org/T348407#9809374 (10fnegri) I finally started working on this. As a first proof of concept, I tried creating a new user on ToolsD... [17:01:45] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:21:45] FIRING: [3x] CloudVPSDesignateLeaks: Detected 69 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:30:26] 10Wikibugs, 10Phabricator, 10GitLab (Integrations): Stop sending CodeReviewBot comments to IRC - https://phabricator.wikimedia.org/T364575#9810024 (10brennen) No objections here. [19:43:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:46:37] (03open) 10bd808: phorge: Ignore CodeReviewBot comments [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 (https://phabricator.wikimedia.org/T364575) [19:47:23] 10Wikibugs, 10Phabricator, 10GitLab (Integrations), 13Patch-For-Review: Stop sending CodeReviewBot comments to IRC - https://phabricator.wikimedia.org/T364575#9810071 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 phorge: Ignore CodeReviewBot comm... [19:52:32] (03approved) 10jforrester: phorge: Ignore CodeReviewBot comments [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 (https://phabricator.wikimedia.org/T364575) (owner: 10bd808) [20:35:12] (03update) 10bd808: phorge: Ignore CodeReviewBot comments [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 (https://phabricator.wikimedia.org/T364575) [20:37:47] (03update) 10bd808: phorge: Ignore CodeReviewBot comments [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 (https://phabricator.wikimedia.org/T364575) [21:18:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:21:45] FIRING: [3x] CloudVPSDesignateLeaks: Detected 70 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:03:41] (03merge) 10bd808: phorge: Ignore CodeReviewBot comments [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 (https://phabricator.wikimedia.org/T364575) [22:03:54] 10Wikibugs, 10Phabricator, 10GitLab (Integrations), 13Patch-For-Review: Stop sending CodeReviewBot comments to IRC - https://phabricator.wikimedia.org/T364575#9810369 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/44 phorge: Ignore CodeReviewBot comm... [22:07:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [22:28:55] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [22:45:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:47:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:47:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:51:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:52:33] 10Wikibugs, 10Phabricator, 10GitLab (Integrations): Stop sending CodeReviewBot comments to IRC - https://phabricator.wikimedia.org/T364575#9810481 (10bd808) 05Open→03Resolved p:05Triage→03Medium a:03bd808 [22:56:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:56:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:57:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [23:00:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:02:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:06:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:36:58] 10Tool-bridgebot, 10Education-Program-Dashboard: Should ignore wikibugs messages on #wikimedia-ed - https://phabricator.wikimedia.org/T330341#9810542 (10bd808) >>! In T330341#8659253, @Jony wrote: > @bd808 Amy update? Did this happen to magically start working as previously expected when bridgebot was updated... [23:44:06] 10Tool-bridgebot, 10Education-Program-Dashboard: Should ignore wikibugs messages on #wikimedia-ed - https://phabricator.wikimedia.org/T330341#9810549 (10bd808) Apparently the bot has been kicked out of the Telegram group. @jony, are you interested in trying again to see if I can actually configure the software...