[00:08:51] <wikibugs>	 (03merge) 10bd808: gitlab: handle "last_commit: null" properly [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/43
[00:15:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:20:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:42:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:47:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:52:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[00:57:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:13:16] <jinxer-wm>	 FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[02:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[02:22:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:13:16] <jinxer-wm>	 FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[05:22:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:27:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[06:17:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[07:40:51] <wikibugs>	 (03update) 10dcaro: secrets: use 60 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:40:53] <wikibugs>	 (03update) 10dcaro: secrets: use 60 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:43:52] <wikibugs>	 (03update) 10dcaro: secrets: use 60 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:43:55] <wikibugs>	 (03update) 10dcaro: secrets: use 60 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:45:34] <wikibugs>	 (03update) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:45:53] <wikibugs>	 (03update) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878)
[07:46:48] <wikibugs>	 (03approved) 10aborrero: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24 (https://phabricator.wikimedia.org/T364878) (owner: 10dcaro)
[08:04:30] <jinxer-wm>	 RESOLVED: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[08:14:34] <wikibugs>	 10Toolforge, 10GitLab, 10Phabricator: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9798495 (10hashar) I am removing #gerrit and #continuous-integration-config since this task is not immediate actionable for any of those projects and that requires...
[08:39:11] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:42:09] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9798601 (10aborrero) working on a resource abstraction that would allow to store state of each resource in the configmap  Code...
[08:57:20] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' (T319184)
[08:57:28] <stashbot>	 T319184: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184
[09:02:11] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Some VPS instances still using ns-recursor0 - https://phabricator.wikimedia.org/T346426#9798631 (10aborrero) 05Open→03Resolved Given we deleted the NAT compat rule, I guess there is nothing else to do regarding this ticket.  Please reopen if required.
[09:05:18] <wikibugs>	 10Toolforge: Scheduler on Toolforge is broken - https://phabricator.wikimedia.org/T364963 (10Iluvatar) 03NEW
[09:07:15] <wikibugs>	 10Toolforge: Scheduler on Toolforge is broken - https://phabricator.wikimedia.org/T364963#9798656 (10Iluvatar)
[09:08:49] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T319184)
[09:08:55] <stashbot>	 T319184: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184
[09:12:14] <wikibugs>	 10Toolforge: Scheduler on Toolforge is broken - https://phabricator.wikimedia.org/T364963#9798673 (10taavi) 05Open→03Invalid The scheduler is working correctly. The first item in `"* */6 * * *"` is the minute - you have specified a star which means that it's going to run the job every minute when the res...
[09:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:17:25] <wikibugs>	 (03update) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24
[09:22:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:51:09] <wikibugs>	 (03update) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24
[10:08:12] <wikibugs>	 (03update) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24
[10:08:14] <wikibugs>	 (03approved) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24
[10:08:18] <wikibugs>	 (03merge) 10dcaro: secrets: use 64 as default quota [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/24
[10:11:13] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.130-20240515100829-b436eda6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/282 (https://phabricator.wikimedia.org/T364878)
[10:11:31] <wikibugs>	 10Toolforge, 13Patch-For-Review: Running out of "secrets" quota (envvars) produces unhelpful error message from `toolforge envvars create` - https://phabricator.wikimedia.org/T364878#9798880 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/tool...
[10:15:23] <wikibugs>	 10Cloud-VPS: Zone delegation for o11y.wmcloud.org - https://phabricator.wikimedia.org/T364973 (10fgiunchedi) 03NEW
[10:25:05] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
[10:25:17] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
[10:26:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
[10:26:57] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
[10:31:15] <wikibugs>	 (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.130-20240515100829-b436eda6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/282 (https://phabricator.wikimedia.org/T364878) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:31:15] <wikibugs>	 (03update) 10dcaro: maintain-kubeusers: bump to 0.0.130-20240515100829-b436eda6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/282 (https://phabricator.wikimedia.org/T364878) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:31:19] <wikibugs>	 (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.130-20240515100829-b436eda6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/282 (https://phabricator.wikimedia.org/T364878) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:31:29] <wikibugs>	 10Toolforge, 13Patch-For-Review: Running out of "secrets" quota (envvars) produces unhelpful error message from `toolforge envvars create` - https://phabricator.wikimedia.org/T364878#9799026 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/282...
[10:32:38] <wikibugs>	 10Toolforge (Toolforge iteration 09): [maintain-kubeusers] Increment default Secrets (envvars) quota - https://phabricator.wikimedia.org/T364883#9799029 (10dcaro) 05Open→03Resolved
[10:32:42] <wikibugs>	 10Toolforge (Toolforge iteration 09): [maintain-kubeusers] Increment default Secrets (envvars) quota - https://phabricator.wikimedia.org/T364883#9799031 (10dcaro) Deployed, the changes should be there already :)
[10:35:10] <wikibugs>	 10Toolforge (Toolforge iteration 09): [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9799034 (10dcaro) 05In progress→03Resolved
[10:35:33] <wikibugs>	 10Toolforge (Toolforge iteration 09): [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9799042 (10dcaro) This is deployed already: ` dcaro@tools-bastion-13:~$ curl -v --insecure https://api.svc.tools.eqiad1.wikimedia.cloud:30003/openapi.json -o...
[10:35:56] <wikibugs>	 10Toolforge (Toolforge iteration 09): [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9799043 (10dcaro)
[10:46:42] <wikibugs>	 10Cloud-VPS: Zone delegation for o11y.wmcloud.org - https://phabricator.wikimedia.org/T364973#9799073 (10taavi) a:03taavi
[10:52:57] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799115 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with OS bookworm
[10:54:39] <wikibugs>	 10Cloud-VPS: Zone delegation for o11y.wmcloud.org - https://phabricator.wikimedia.org/T364973#9799120 (10taavi) 05Open→03Resolved You should be able to create that proxy via Horizon now, please re-open if that doesn't work for some reason.
[11:05:53] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799155 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with...
[11:11:11] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799171 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet...
[11:41:24] <wikibugs>	 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9799232 (10Slst2020) I think this is solved by {T362299} which was t...
[11:41:46] <wikibugs>	 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9799235 (10Slst2020) 05Open→03Resolved
[11:45:54] <wikibugs>	 06cloud-services-team, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984 (10aborrero) 03NEW
[11:49:11] <wikibugs>	 06cloud-services-team, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9799282 (10aborrero)
[11:49:42] <wikibugs>	 10Toolforge (Toolforge iteration 09): [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#9799284 (10Slst2020) Returning to the original intent of this task, there seem to only be two good alternatives, OpenAPI Generator...
[11:52:49] <wikibugs>	 06cloud-services-team, 06Infrastructure-Foundations, 10netops, 06SRE, 13Patch-For-Review: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184#9799293 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1041.eqiad.wmnet with...
[11:58:07] <wikibugs>	 06cloud-services-team, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9799298 (10aborrero) p:05Triage→03Medium hey @Jclark-ctr or @Jhancock.wm could you please advice / help with this server? thanks in advance.
[12:03:09] <wikibugs>	 06cloud-services-team, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9799319 (10aborrero) additional information: when reimaging the server, the debian installer failed, complaining about the volume group name being in use already.  To try to workarou...
[12:12:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:22:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:03:31] <wikibugs>	 (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110)
[13:42:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:45:07] <wikibugs>	 (03update) 10dcaro: k8s_api: drop unneeded code [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/22 (owner: 10aborrero)
[13:47:00] <wikibugs>	 (03update) 10dcaro: k9s: add config files [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/128 (owner: 10aborrero)
[13:47:28] <wikibugs>	 (03open) 10raymond-ndibe: DO_NOT_MERGE: testing _display_messages move to toolforge-weld [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/90
[13:52:41] <jinxer-wm>	 RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:09:46] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 iiab START - Cookbook wmcs.openstack.quota_increase (T361946)
[14:09:49] <stashbot>	 T361946: Request temporary storage quota increase for project iiab for migration to bookworm image - https://phabricator.wikimedia.org/T361946
[14:09:54] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 iiab END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T361946)
[14:10:56] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 (T364822)
[14:10:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:10:59] <stashbot>	 T364822: [infra] NFS hangs in some workers until the worker is rebooted (2024-05-14) - https://phabricator.wikimedia.org/T364822
[14:11:50] <wm-bot2>	 !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 (T364822)
[14:11:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:16:51] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 (T364822)
[14:16:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:16:57] <stashbot>	 T364822: [infra] NFS hangs in some workers until the worker is rebooted (2024-05-14) - https://phabricator.wikimedia.org/T364822
[14:17:41] <wm-bot2>	 !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 (T364822)
[14:17:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:20:08] <wikibugs>	 (03PS1) 10Majavah: neutron: Migrate to openstack network agent list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031934 (https://phabricator.wikimedia.org/T365000)
[14:20:10] <wikibugs>	 (03PS1) 10Majavah: openstack: Support OVS agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031935 (https://phabricator.wikimedia.org/T326373)
[14:20:13] <wikibugs>	 (03PS1) 10Majavah: neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000)
[14:23:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[14:27:32] <wikibugs>	 (03PS2) 10Majavah: neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000)
[14:27:32] <wikibugs>	 (03PS1) 10Majavah: neutron: Filter agent hosts server side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031939
[14:27:51] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46
[14:32:39] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[14:33:03] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Filter agent hosts server side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031939 (owner: 10Majavah)
[14:36:46] <wikibugs>	 (03PS3) 10Majavah: neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000)
[14:36:46] <wikibugs>	 (03PS2) 10Majavah: neutron: Filter agent hosts server side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031939
[14:47:58] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46
[14:48:02] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [toolforge-weld] move _display_message into toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/46
[14:49:10] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [builds-cli] remove _display_messages [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/69
[14:57:50] <jinxer-wm>	 FIRING: NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown
[14:58:21] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [builds-cli] remove _display_messages [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/69
[15:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:17:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:22:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:27:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:55:53] <wikibugs>	 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9800850 (10fnegri) As defined in the [application process](https://wikitech.wikimedia.org/wiki/Help:Access_policies#Application_Process) I have sent a message to the cloud-admi...
[15:59:38] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-api] add messages to all responses [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T356974)
[16:06:24] <wikibugs>	 (03update) 10aborrero: Draft: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T279110)
[16:18:39] <wikibugs>	 (03PS1) 10Majavah: neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000)
[16:21:58] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[16:27:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[16:51:15] <wikibugs>	 (03PS2) 10Majavah: neutron: Support OVS agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031935 (https://phabricator.wikimedia.org/T326373)
[16:51:15] <wikibugs>	 (03PS4) 10Majavah: neutron: Migrate to openstack router list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031936 (https://phabricator.wikimedia.org/T365000)
[16:51:15] <wikibugs>	 (03PS3) 10Majavah: neutron: Filter agent hosts server side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031939
[16:51:15] <wikibugs>	 (03PS2) 10Majavah: neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000)
[16:51:16] <wikibugs>	 (03PS1) 10Majavah: openstack: cloudnet: Don't run with proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031968
[16:51:18] <wikibugs>	 (03PS1) 10Majavah: neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969
[16:51:22] <wikibugs>	 (03PS1) 10Majavah: openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970
[16:51:26] <wikibugs>	 (03PS1) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[16:51:30] <wikibugs>	 (03PS1) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[16:51:34] <wikibugs>	 (03PS1) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[16:52:28] <wmcs-alerts>	 RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[16:54:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack: cloudnet: Don't run with proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031968 (owner: 10Majavah)
[16:55:04] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970 (owner: 10Majavah)
[16:55:16] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969 (owner: 10Majavah)
[16:55:17] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[16:55:35] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[16:55:38] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[16:56:11] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973 (owner: 10Majavah)
[16:57:51] <wikibugs>	 (03PS3) 10Majavah: neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000)
[16:57:51] <wikibugs>	 (03PS2) 10Majavah: openstack: cloudnet: Don't run with proxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031968
[16:57:51] <wikibugs>	 (03PS2) 10Majavah: neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969
[16:57:51] <wikibugs>	 (03PS2) 10Majavah: openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970
[16:57:52] <wikibugs>	 (03PS2) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[16:57:53] <wikibugs>	 (03PS2) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[16:57:57] <wikibugs>	 (03PS2) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[16:58:44] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+1] neutron: Migrate to openstack network agent list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031934 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[16:59:23] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+1] neutron: Support OVS agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031935 (https://phabricator.wikimedia.org/T326373) (owner: 10Majavah)
[17:01:34] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970 (owner: 10Majavah)
[17:01:36] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969 (owner: 10Majavah)
[17:01:41] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973 (owner: 10Majavah)
[17:01:49] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[17:02:17] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:06:09] <wikibugs>	 (03PS3) 10Majavah: neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969
[17:06:09] <wikibugs>	 (03PS3) 10Majavah: openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970
[17:06:09] <wikibugs>	 (03PS3) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[17:06:10] <wikibugs>	 (03PS3) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[17:06:10] <wikibugs>	 (03PS3) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[17:06:13] <wikibugs>	 (03CR) 10Majavah: [C:03+2] neutron: Migrate to openstack network agent list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031934 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:06:17] <wikibugs>	 (03CR) 10Majavah: [C:03+2] neutron: Support OVS agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031935 (https://phabricator.wikimedia.org/T326373) (owner: 10Majavah)
[17:07:05] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+1] neutron: Use openstack commands to get agent HA state [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031964 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:08:47] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+1] neutron: Filter for agent types server-side [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031969 (owner: 10Majavah)
[17:09:31] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[17:09:35] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:09:36] <wikibugs>	 (03Merged) 10jenkins-bot: neutron: Migrate to openstack network agent list [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031934 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:09:37] <wikibugs>	 (03Merged) 10jenkins-bot: neutron: Support OVS agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031935 (https://phabricator.wikimedia.org/T326373) (owner: 10Majavah)
[17:09:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973 (owner: 10Majavah)
[17:09:53] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970 (owner: 10Majavah)
[17:12:20] <wikibugs>	 (03PS4) 10Majavah: openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970
[17:12:21] <wikibugs>	 (03PS4) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[17:12:21] <wikibugs>	 (03PS4) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[17:12:21] <wikibugs>	 (03PS4) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[17:15:30] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[17:15:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973 (owner: 10Majavah)
[17:15:55] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:16:41] <wikibugs>	 (03PS5) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[17:16:42] <wikibugs>	 (03PS5) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[17:17:16] <wikibugs>	 (03PS5) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[17:17:16] <wikibugs>	 (03PS6) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[17:17:17] <wikibugs>	 (03PS6) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[17:19:57] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[17:20:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973 (owner: 10Majavah)
[17:20:47] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000) (owner: 10Majavah)
[17:20:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972 (owner: 10Majavah)
[17:21:23] <wikibugs>	 (03PS6) 10Majavah: neutron: Use openstack agent set to enable/disable agents [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031971 (https://phabricator.wikimedia.org/T365000)
[17:21:23] <wikibugs>	 (03PS7) 10Majavah: neutron: Remove unused error classes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031972
[17:21:23] <wikibugs>	 (03PS7) 10Majavah: neutron: Remove now-unused command running functionality [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031973
[17:21:46] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+1] openstack: cloudnet: Remove check for router count [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1031970 (owner: 10Majavah)
[17:43:41] <wikibugs>	 10Toolforge: Deleting an envvar breaks ReplicaSet driven automatic restarts of a Pod (CreateContainerConfigError) - https://phabricator.wikimedia.org/T365048 (10bd808) 03NEW
[17:51:04] <wikibugs>	 06cloud-services-team, 10Toolforge: Request for access for user dr0ptp4kt for 'admin' tool - https://phabricator.wikimedia.org/T364761#9801439 (10bd808) >>! In https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/NKIDWUGLPZOQMQ4EI3WXYU47SKWDG5WF/, @fnegri wrote: > This is a subset...
[18:13:23] <wikibugs>	 (03open) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974)
[18:13:35] <wikibugs>	 10Toolforge (Toolforge iteration 09), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9801477 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforg...
[18:30:27] <wikibugs>	 (03open) 10raymond-ndibe: [toolforge-weld] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130
[18:30:31] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130
[18:30:37] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130
[18:31:34] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-weld] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130
[18:31:48] <wikibugs>	 (03update) 10raymond-ndibe: [lima-kilo] enable toolforge-weld installation [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/130
[18:52:54] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-api] add messages to all responses [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T356974)
[18:58:04] <jinxer-wm>	 FIRING: NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown
[19:10:46] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-api] add messages to all responses [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T356974)
[19:16:41] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974)
[19:21:18] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974)
[19:27:30] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974)
[19:28:14] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-cli] add messages to all responses [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/38 (https://phabricator.wikimedia.org/T356974)
[19:28:42] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-api] add messages to all responses [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T356974)
[19:28:47] <wikibugs>	 (03update) 10raymond-ndibe: [envvars-api] add messages to all responses [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/25 (https://phabricator.wikimedia.org/T356974)
[19:34:05] <wikibugs>	 (03update) 10raymond-ndibe: Draft: [builds-cli] remove _display_messages [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/69
[19:42:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:47:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:52:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:57:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[19:58:31] <wikibugs>	 10Wikibugs: Wikibugs' gitlab connector stops working without a strong sign of why - https://phabricator.wikimedia.org/T364490#9801775 (10bd808) No connection loss triggering restarts or mystery hangs triggering the health-check since cutting the Kubernetes ingress layer out of the connection between wikibugs and...
[20:23:30] <wikibugs>	 10Tool-wikiloves: Wiki Loves Earth in Democratic Republic of the Congo - https://phabricator.wikimedia.org/T365064 (10JeanFred) 03NEW
[20:23:55] <wikibugs>	 10Tool-wikiloves: Wiki Loves Earth in Democratic Republic of the Congo - https://phabricator.wikimedia.org/T365064#9801878 (10JeanFred)
[20:25:33] <wikibugs>	 10Tool-wikiloves: Wiki Loves Earth in Democratic Republic of the Congo - https://phabricator.wikimedia.org/T365064#9801880 (10JeanFred) Reason is that the tool expects the category names to be « Category:Images from Wiki Loves XYZ 20XX in the Democratic Republic of the Congo » (note the `the`), but the WLE categ...
[20:32:43] <wikibugs>	 10Toolforge, 10GitLab, 10Phabricator: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9801906 (10bd808) >>! In T364516#9798495, @hashar wrote: > I am removing #gerrit and #continuous-integration-config since this task is not immediate actionable for...
[20:35:40] <wikibugs>	 10Tool-wikiloves: Wiki Loves Earth in Democratic Republic of the Congo - https://phabricator.wikimedia.org/T365064#9801912 (10JeanFred) For background − the tool looks up categories with the format `Images from <contest> <year> in <country>`.  when we added WLA support back in 2017, stats for Congo (as well as C...
[20:43:22] <wikibugs>	 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9801927 (10Soda) @Cunegonde1 It should work, which page is this? (I might be able to debug your issues better if I know the exact...
[20:56:00] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[21:12:41] <jinxer-wm>	 FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:16:00] <jinxer-wm>	 FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[21:17:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:22:41] <jinxer-wm>	 FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[21:27:41] <jinxer-wm>	 RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:42:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:52:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[22:58:05] <jinxer-wm>	 FIRING: NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown
[23:51:00] <jinxer-wm>	 FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[23:59:40] <wikibugs>	 10Toolforge (Toolforge iteration 09): [maintain-kubeusers] Increment default services quota - https://phabricator.wikimedia.org/T362520#9802568 (10Raymond_Ndibe) >>! In T362520#9796110, @bd808 wrote: >> to some TBD higher value. >  > How about 16 to match the number of pods in the default quota? We can certainly...