[00:15:28] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:02] (03open) 10bd808: gitlab and concurrency fixes [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/37 [00:20:28] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:25] 10Wikibugs, 13Patch-For-Review: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9784922 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/37 gitlab and concurrency fixes [03:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:44:48] (03CR) 10Muehlenhoff: [C:03+1] "Always nice to see 1036 manually managed files go away :-) Or rather 2072, also accounting for the equivalents in the private repo..." [labs/private] - 10https://gerrit.wikimedia.org/r/1029567 (https://phabricator.wikimedia.org/T352647) (owner: 10Elukey) [06:11:30] 10Tool-yearinreview, 10New-Engagement-Experiments, 03Wikimedia-Hackathon-2024: Improve the year in review tool for next year - https://phabricator.wikimedia.org/T362898#9785086 (10Reputation22) Yeah agreed, So apart from those, we planned to have one GET api (something along the lines of action=query & meta=... [08:31:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9785309 (10taavi) [08:33:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9785316 (10taavi) [08:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:46:23] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9785335 (10taavi) [08:47:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 cloudnets to Neutron OVS agent - https://phabricator.wikimedia.org/T364459#9785336 (10taavi) [08:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:55:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [09:57:45] 06cloud-services-team, 10wikitech.wikimedia.org, 10Bitu, 06Infrastructure-Foundations, 07Epic: Move Striker to Bitu username validation API. - https://phabricator.wikimedia.org/T364605 (10SLyngshede-WMF) 03NEW [10:02:36] 10Striker: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#9785475 (10taavi) [10:02:39] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API. - https://phabricator.wikimedia.org/T364605#9785474 (10taavi) [10:03:20] 10Striker: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#9785482 (10SLyngshede-WMF) p:05Triage→03Medium [10:03:32] 10Striker: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#9785483 (10SLyngshede-WMF) a:03SLyngshede-WMF [10:04:32] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API. - https://phabricator.wikimedia.org/T364605#9785477 (10SLyngshede-WMF) p:05Triage→03Medium [10:05:21] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#9785472 (10taavi) [10:05:22] 10Striker: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#9785487 (10SLyngshede-WMF) I'll assume that I need to at least build the Bitu container image. Then we'll figure out who adds it to the Striker developer environment. [10:05:32] (03PS1) 10Majavah: build: Remove .travis.yml [labs/striker] - 10https://gerrit.wikimedia.org/r/1030053 [10:08:44] (03CR) 10Majavah: [C:03+2] build: Remove .travis.yml [labs/striker] - 10https://gerrit.wikimedia.org/r/1030053 (owner: 10Majavah) [10:10:06] (03Merged) 10jenkins-bot: build: Remove .travis.yml [labs/striker] - 10https://gerrit.wikimedia.org/r/1030053 (owner: 10Majavah) [10:12:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:18:22] 10Toolforge: [toolforge] webservice logs crashes with some unicode chars - https://phabricator.wikimedia.org/T364609 (10fnegri) 03NEW [10:19:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09): [toolforge] webservice logs crashes with some unicode chars - https://phabricator.wikimedia.org/T364609#9785555 (10fnegri) [10:20:28] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09): [toolforge] webservice logs crashes with some unicode chars - https://phabricator.wikimedia.org/T364609#9785553 (10fnegri) p:05Triage→03High [10:22:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:23:15] (03open) 10aborrero: k8s_api: drop unneeded code [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/22 [11:25:14] (03open) 10aborrero: Draft: k8s_api: drop unneeded code [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 [11:25:30] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [infra] Replace PodSecurityPolicy in Toolforge Kubernetes - https://phabricator.wikimedia.org/T279110#9785661 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 Draft: k8s_api: dr... [11:32:57] 10Wikibugs: Send updates about open MRs - https://phabricator.wikimedia.org/T364615 (10taavi) 03NEW [11:36:50] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#9785729 (10taavi) [11:38:05] 10Striker: Add Bitu container to Striker development environment - https://phabricator.wikimedia.org/T362318#9785730 (10taavi) Yes, we either need to add Blubber config to build a container in the Bitu repo and publish that to docker-registry.wikimedia.org or add [[ https://gerrit.wikimedia.org/r/plugins/gitiles... [11:54:34] (03CR) 10Elukey: [V:03+2 C:03+2] Add fake TLS keystore password for Cassandra clusters [labs/private] - 10https://gerrit.wikimedia.org/r/1029538 (https://phabricator.wikimedia.org/T352647) (owner: 10Elukey) [11:55:00] (03CR) 10Elukey: [V:03+2 C:03+2] Delete the Cassandra directory in secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1029567 (https://phabricator.wikimedia.org/T352647) (owner: 10Elukey) [12:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:04:48] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9785879 (10fnegri) [13:32:13] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9785921 (10fnegri) 05Stalled→03Resolved We did a second drill this week, the incident doc is [here](). We should consider doing more drills in... [13:35:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9785927 (10fnegri) 05In progress→03Stalled [13:39:19] FIRING: TektonDown: Tekton is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/TektonDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTektonDown [13:39:23] FIRING: ToolforgeKubernetesNodeNotReady: Multiple Kubernetes nodes are not ready #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [13:40:09] 10Tools: 'hoiscript' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T349913#9785931 (10Hoi) Done. [13:47:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Drop gu_salt from globaluser - https://phabricator.wikimedia.org/T364435#9785946 (10fnegri) a:03fnegri Claiming this task temporarily: I will deploy the wikireplicas change on Mon... [13:48:09] 10cloud-services-team (FY2023/2024-Q3-Q4), 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Drop gu_salt from globaluser - https://phabricator.wikimedia.org/T364435#9785950 (10fnegri) 05Open→03In progress [13:49:32] 06cloud-services-team, 10Toolforge: toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9785968 (10taavi) [13:51:32] 06cloud-services-team, 10Toolforge: toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9785970 (10taavi) The current certificates expire on June 1st. [13:52:51] 06cloud-services-team, 10Toolforge: toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9785987 (10taavi) p:05Medium→03High [13:55:46] 10Cloud Services Proposals, 10cloud-services-team (FY2023/2024-Q3-Q4): Decision Request - Incident Response Process - https://phabricator.wikimedia.org/T348887#9785988 (10fnegri) [14:04:19] RESOLVED: TektonDown: Tekton is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/TektonDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTektonDown [14:04:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Multiple Kubernetes nodes are not ready #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [14:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:44:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:47:34] 10Wikibugs: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9786327 (10bd808) [15:52:33] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9786340 (10bd808) test [15:54:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:05:13] 10Tool-yearinreview, 07good first task: Shareable image download issue - https://phabricator.wikimedia.org/T364642 (10Gopavasanth) 03NEW Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to seasoned contributors, s... [17:06:51] 10Wikibugs, 13Patch-For-Review: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9786575 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/37 gitlab and concurrency fixes [17:07:03] 10Tool-yearinreview, 07good first task: Shareable image download issue - https://phabricator.wikimedia.org/T364642#9786576 (10Gopavasanth) [17:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:16:48] 10Wikibugs: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9786614 (10bd808) > Any new commits added / force-pushed to the source branch I think we can do this one with the existing hooks. `lang=shell-session $ grep 'toolforge-repos/wikibugs2/-/merge_requests/37' events.log | jq --comp... [17:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:34:32] 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9786658 (10bd808) Let's see what happens with the health-check band-aid in place for a while. [18:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:11:10] (03open) 10bd808: gitlab: Announce all merge_request actions [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/39 [21:12:05] 10Wikibugs, 13Patch-For-Review: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9787229 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/39 gitlab: Announce all merge_request actions [21:16:49] (03merge) 10bd808: gitlab: Announce all merge_request actions [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/39 [21:16:55] 10Wikibugs, 13Patch-For-Review: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9787237 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/39 gitlab: Announce all merge_request actions [21:19:34] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:20:31] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:25:50] ------ [21:26:06] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:26:14] (03approved) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:26:17] (03unapproved) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:26:23] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 [21:28:37] 10Wikibugs: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9787260 (10bd808) [21:32:15] 10Wikibugs: Send updates about open MRs - https://phabricator.wikimedia.org/T364615#9787265 (10bd808) With {rTWBTb63637c0700709cea62ace48438fb114224f6c4c} landed we will start seeing 3 more actions: `lang=irc [21:20] < wb-test> (update) bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs... [21:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:49:24] 10Toolforge (Toolforge iteration 09): [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9787382 (10bd808) >>! In T363417#9774388, @dcaro wrote: > @bd808 This should have been fixed, can you try now? `lang=shell-session $ echo $PATH /l... [23:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:52:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks