[00:14:37] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] (refactor_validate_kube_quant) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120)
[00:16:28] <wmcs-alerts>	 FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:20:10] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 80.39% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:21:28] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:25:10] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.39% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:25:28] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-harbor-2 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[00:44:10] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 81.09% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:49:10] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 81.09% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:50:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:00:56] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[01:05:28] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-harbor-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[02:41:54] <wikibugs>	 (03merge) 10samwilson: Flush jobs before loading [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/2
[03:58:33] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[05:10:05] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
[05:11:16] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
[05:11:30] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
[05:12:38] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
[05:13:48] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
[05:15:01] <logmsgbot_cloud>	 !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
[07:58:33] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[08:08:45] <wikibugs>	 (03update) 10dcaro: show task status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370321)
[08:09:00] <wikibugs>	 (03update) 10dcaro: checks: add database check [toolforge-repos/sample-complex-app-frontend] (show_deployment_steps) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/4 (https://phabricator.wikimedia.org/T370317)
[08:11:28] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [sct.backend] Create trove database - https://phabricator.wikimedia.org/T370317#10111748 (10dcaro) 05In progress→03Resolved
[08:12:24] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [sct.backend] Create worker and connect to redis - https://phabricator.wikimedia.org/T370321#10111751 (10dcaro) 05In progress→03Resolved
[08:16:16] <wikibugs>	 (03update) 10dcaro: builds-api: bump to 0.0.170-20240902125720-e7b15322 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[08:19:41] <wikibugs>	 (03CR) 10FNegri: [C:03+1] with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 (owner: 10David Caro)
[08:24:30] <wmcs-alerts>	 RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance cloudinfra-idp-1 in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[08:26:02] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Migrate Cloud VPS instances to VXLAN based networks - https://phabricator.wikimedia.org/T364725#10111792 (10aborrero) reading: https://docs.openstack.org/neutron/2024.1/admin/deploy-ovs-ha-vrrp.html
[08:28:15] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudinfra] Upgrade cloudinfra-idp-* to bookworm - https://phabricator.wikimedia.org/T373840 (10dcaro) 03NEW
[08:28:20] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudinfra] Upgrade cloudinfra-idp-* to bookworm - https://phabricator.wikimedia.org/T373840#10111809 (10dcaro) p:05Triage→03High
[08:38:53] <wikibugs>	 (03open) 10dcaro: calico: use only 1 replica for typha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/510
[08:40:09] <wikibugs>	 (03CR) 10David Caro: [C:03+2] with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 (owner: 10David Caro)
[08:43:12] <wikibugs>	 (03approved) 10sstefanova: calico: use only 1 replica for typha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/510 (owner: 10dcaro)
[08:44:38] <wikibugs>	 (03Merged) 10jenkins-bot: with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 (owner: 10David Caro)
[08:52:43] <wikibugs>	 (03approved) 10dcaro: calico: use only 1 replica for typha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/510
[08:52:46] <wikibugs>	 (03merge) 10dcaro: calico: use only 1 replica for typha [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/510
[09:10:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Cloud VPS: investigate conntrack table usage on cloudvirt1050 - https://phabricator.wikimedia.org/T373816#10111944 (10aborrero) `lang=shell-session aborrero@cloudvirt1050:~ $ sudo conntrack -L | grep -o "src=[[:digit:]]*.[[:digit:]]*.[[:digit:]]*.[[:digit:]]*" | sort -n | un...
[09:19:33] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:20:33] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS: investigate conntrack table usage on cloudvirt1050 - https://phabricator.wikimedia.org/T373816#10111961 (10aborrero) 05Open→03In progress p:05Triage→03Medium a:03aborrero
[09:20:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:35:56] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[09:39:48] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
[09:45:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
[09:51:28] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
[09:57:34] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
[10:04:09] <wikibugs>	 (03approved) 10dcaro: builds-builder: bump to 0.0.117-20240902122920-3b0529c7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:04:12] <wikibugs>	 (03update) 10dcaro: builds-builder: bump to 0.0.117-20240902122920-3b0529c7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:04:26] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[10:05:10] <wikibugs>	 (03merge) 10dcaro: builds-builder: bump to 0.0.117-20240902122920-3b0529c7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:10:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[10:15:56] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[10:21:37] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[10:22:02] <wikibugs>	 (03approved) 10dcaro: builds-api: bump to 0.0.170-20240902125720-e7b15322 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:22:03] <wikibugs>	 (03update) 10dcaro: builds-api: bump to 0.0.170-20240902125720-e7b15322 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:22:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[10:22:25] <wikibugs>	 (03merge) 10dcaro: builds-api: bump to 0.0.170-20240902125720-e7b15322 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[10:27:16] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[10:34:25] <wikibugs>	 (03open) 10samtar: Update submodules [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/3
[10:34:48] <wikibugs>	 (03merge) 10samtar: Update submodules [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/3
[10:46:33] <wikibugs>	 (03open) 10samtar: Add hourly job for update-wish-talks [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/4
[10:46:57] <wikibugs>	 (03merge) 10samtar: Add hourly job for update-wish-talks [toolforge-repos/wishlist] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist/-/merge_requests/4
[11:28:06] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:30:39] <wikibugs>	 (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:30:41] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/17 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:31:06] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:31:53] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:32:05] <wikibugs>	 (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:32:23] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/123 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:32:37] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/123 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:33:32] <wikibugs>	 (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/122 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:33:33] <wikibugs>	 (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/122 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:34:24] <wikibugs>	 (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/12 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:37:38] <wikibugs>	 (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203)
[11:37:43] <wikibugs>	 (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203)
[11:40:49] <wikibugs>	 (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/122 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:40:55] <wikibugs>	 (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/123 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:43:03] <wikibugs>	 (03update) 10sstefanova: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[11:45:31] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:45:31] <wikibugs>	 (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:50:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:51:56] <wikibugs>	 (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/10 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3)
[11:52:11] <wikibugs>	 (03update) 10sstefanova: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[11:53:47] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[11:59:16] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[12:05:56] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:07:27] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#10112657 (10Slst2020) a:03Slst2020
[12:08:46] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#10112659 (10Slst2020)
[12:16:46] <wikibugs>	 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866 (10Slst2020) 03NEW
[12:28:20] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112698 (10dcaro)
[12:29:30] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112699 (10dcaro)
[12:31:39] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112704 (10dcaro)
[12:32:03] <notefromgithub>	 dhinus closed https://github.com/toolforge/quarry/pull/66
[12:32:33] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112705 (10dcaro)
[12:33:40] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112717 (10dcaro)
[12:33:59] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Cloud VPS: design target vxlan setup - https://phabricator.wikimedia.org/T373869 (10aborrero) 03NEW
[12:36:12] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Cloud VPS: design target vxlan setup - https://phabricator.wikimedia.org/T373869#10112732 (10aborrero) 05Open→03In progress p:05Triage→03Medium
[12:41:04] <wikibugs>	 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10112745 (10dcaro)
[12:46:44] <wikibugs>	 (03open) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[12:47:55] <wikibugs>	 (03approved) 10dcaro: envvars-api: bump to 0.0.60-20240902124757-342c4931 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[12:47:59] <wikibugs>	 (03update) 10dcaro: envvars-api: bump to 0.0.60-20240902124757-342c4931 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[12:48:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[12:48:27] <wikibugs>	 (03merge) 10dcaro: envvars-api: bump to 0.0.60-20240902124757-342c4931 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[12:50:51] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
[12:53:03] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[12:54:09] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Cloud VPS: design target vxlan setup - https://phabricator.wikimedia.org/T373869#10112826 (10aborrero) This is a preliminary plan / idea.  The target setup is as follows: * we have no vlan-based network in the system. * we have a single, flat vxlan-based...
[12:55:14] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[12:58:01] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[12:59:48] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[13:04:39] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[13:04:40] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[13:05:01] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[13:05:27] <wikibugs>	 (03approved) 10dcaro: jobs-api: bump to 0.0.332-20240902122934-464b67dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:05:31] <wikibugs>	 (03update) 10dcaro: jobs-api: bump to 0.0.332-20240902122934-464b67dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:05:54] <wikibugs>	 (03merge) 10dcaro: jobs-api: bump to 0.0.332-20240902122934-464b67dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:05:57] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[13:10:36] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[13:17:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
[13:21:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:23:21] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
[13:28:18] <wikibugs>	 (03approved) 10dcaro: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:28:23] <wikibugs>	 (03update) 10dcaro: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:28:40] <wikibugs>	 (03update) 10dcaro: registry-admission: bump to 0.0.50-20240902122909-507a2f0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[13:28:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[13:31:56] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:33:56] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[13:39:43] <wikibugs>	 10Toolforge, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10113021 (10dcaro)
[13:41:20] <wikibugs>	 10Toolforge, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10113024 (10dcaro)
[13:49:08] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[13:50:43] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[13:53:00] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (T359641)
[13:53:00] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 (T359641)
[13:53:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:53:03] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[13:53:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:53:28] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 (T359641)
[13:53:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:53:54] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 (T359641)
[13:53:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:54:20] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 (T359641)
[13:54:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:54:47] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 (T359641)
[13:54:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:55:09] <wikibugs>	 06cloud-services-team: openstack: fix missing prometheus metrics - https://phabricator.wikimedia.org/T373878 (10aborrero) 03NEW
[13:55:12] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 (T359641)
[13:55:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:55:58] <wm-bot2>	 !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) (T359641)
[13:56:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:56:08] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[13:58:13] <wikibugs>	 (03PS1) 10David Caro: kyverno.copy_images_to_registry: allow passing the versions on cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070258
[14:04:08] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (T359641)
[14:04:08] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 (T359641)
[14:04:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:04:12] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[14:04:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:04:23] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 (T359641)
[14:04:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:04:56] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 (T359641)
[14:04:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:05:09] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 (T359641)
[14:05:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:05:22] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 (T359641)
[14:05:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:05:35] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 (T359641)
[14:05:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:05:47] <wm-bot2>	 !log dcaro@urcuchillay tools Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 (T359641)
[14:05:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:05:59] <wm-bot2>	 !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) (T359641)
[14:06:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:06:46] <wikibugs>	 (03PS1) 10David Caro: kyverno.copy_images_to_registry: add missing kyverno-cli image [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070262 (https://phabricator.wikimedia.org/T359641)
[14:07:23] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[14:11:14] <wikibugs>	 (03approved) 10dcaro: registry-admission: bump to 0.0.50-20240902122909-507a2f0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:11:27] <wikibugs>	 (03merge) 10dcaro: registry-admission: bump to 0.0.50-20240902122909-507a2f0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:11:40] <wikibugs>	 (03update) 10dcaro: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[14:11:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[14:12:39] <wikibugs>	 (03CR) 10Raymond Ndibe: [C:03+2] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070258 (owner: 10David Caro)
[14:16:35] <wikibugs>	 (03Merged) 10jenkins-bot: kyverno.copy_images_to_registry: allow passing the versions on cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070258 (owner: 10David Caro)
[14:17:36] <wikibugs>	 10Toolforge, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10113222 (10dcaro)
[14:19:29] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[14:19:35] <wikibugs>	 (03update) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[14:23:00] <wikibugs>	 (03open) 10dcaro: kyverno_pod_policy: remove kyverno versions from annotations [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/61 (https://phabricator.wikimedia.org/T359641)
[14:24:51] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[14:24:59] <wikibugs>	 (03approved) 10aborrero: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641) (owner: 10dcaro)
[14:30:31] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[14:32:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[14:32:37] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
[14:32:57] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[14:36:14] <wmcs-alerts>	 FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[14:39:29] <wmcs-alerts>	 FIRING: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-control-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[14:41:02] <wm-bot2>	 !log dcaro@urcuchillay tools START - Cookbook wmcs.openstack.cloudvirt.vm_console
[14:41:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:41:07] <wm-bot2>	 !log dcaro@urcuchillay tools END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
[14:41:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:44:16] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console
[14:44:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:44:21] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
[14:44:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:44:45] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console
[14:44:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:44:51] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
[14:44:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:44:58] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.openstack.cloudvirt.vm_console
[14:44:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:45:24] <wmcs-alerts>	 FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-control-7 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[14:50:08] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
[14:50:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[14:54:42] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
[14:55:06] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[14:58:05] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno
[15:05:38] <wikibugs>	 10wikitech.wikimedia.org, 10MW-on-K8s, 06serviceops, 13Patch-For-Review: MVP: Privately serve wikitech via mwdebug1001 - https://phabricator.wikimedia.org/T371537#10113442 (10jijiki)
[15:06:29] <wmcs-alerts>	 RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-control-7 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[15:08:47] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[15:13:44] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown
[15:14:55] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10113481 (10fnegri) 05Open→03In progress
[15:15:29] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
[15:22:27] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[15:27:30] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
[15:27:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:27:56] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
[15:27:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[15:29:26] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
[15:35:32] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#10113626 (10wiki_willy) Thanks @dcaro, sounds good. I'll bug them again about the drive number, if we don't hear bac...
[15:40:48] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component kyverno
[15:46:11] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
[15:57:16] <wikibugs>	 10PAWS: Can gitlab build docker images? - https://phabricator.wikimedia.org/T373896 (10rook) 03NEW
[16:12:00] <wikibugs>	 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10113782 (10LucasWerkmeister) FWIW, according to the commit [deprecating](https://github.com/kubernetes/kubernetes/commit/09804a198c) this and other flags, `--wait` had...
[16:13:11] <wikibugs>	 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10113812 (10LucasWerkmeister) (Kubernetes 1.31 apparently [adds a --wait option again](https://github.com/kubernetes/website/commit/59df28c34043443883c1e20503ffc1f70d61b...
[16:20:49] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS: Cloud VPS: design target vxlan setup - https://phabricator.wikimedia.org/T373869#10113877 (10aborrero) How do we allow users to create VMs on a different subnet? The horizon 'new vm' panel doesn't show the option. Having a default one is also fine, maybe....
[16:22:19] <wikibugs>	 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10113894 (10Slst2020) want to submit a patch, @LucasWerkmeister ? 😁
[16:23:24] <wikibugs>	 (03approved) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[16:25:40] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
[16:25:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:26:02] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
[16:26:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:26:12] <wikibugs>	 10Toolforge: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10113901 (10LucasWerkmeister) I guess so :P since I already started looking into this after noticing the output (I’m guessing it’s due to the Kubernetes upgrade yesterday?)
[16:29:08] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
[16:29:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:34:03] <wikibugs>	 (03open) 10lucaswerkmeister: shell: drop --wait [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/54 (https://phabricator.wikimedia.org/T373866)
[16:37:35] <wikibugs>	 (03approved) 10dcaro: shell: drop --wait [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/54 (https://phabricator.wikimedia.org/T373866) (owner: 10lucaswerkmeister)
[16:38:34] <wikibugs>	 06cloud-services-team, 10Toolforge: Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#10113951 (10fnegri) a:03bd808
[16:39:33] <wikibugs>	 (03merge) 10dcaro: upgrade kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/511 (https://phabricator.wikimedia.org/T359641)
[16:40:00] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10113978 (10dcaro)
[16:40:35] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
[16:40:35] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
[16:40:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:40:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:42:31] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
[16:42:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:44:03] <wikibugs>	 10Cloud-VPS: object store for tf-infra-test - https://phabricator.wikimedia.org/T365820#10113992 (10rook) Created tofuinfratest project. Added to metrics infra (https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS):    `INSERT INTO alerts VALUES (NULL, 240, 'ApplyFailed'...
[16:47:33] <wikibugs>	 10Quarry: remove buster systems - https://phabricator.wikimedia.org/T364753#10114012 (10rook) 05Open→03Resolved
[16:50:28] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-control-11 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[16:53:05] <wikibugs>	 10Openstack-Magnum: Create default magnum template - https://phabricator.wikimedia.org/T328713#10114055 (10rook) As magnum is further understood a default template seems somewhat ill advisable. Some parts of a template would be the same, namely the bits out of the magnum documentation describing how to setup...
[16:53:07] <wikibugs>	 10Openstack-Magnum: Create default magnum template - https://phabricator.wikimedia.org/T328713#10114056 (10rook) 05Open→03Declined
[16:53:40] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
[16:53:40] <wm-bot2>	 !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
[16:53:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:53:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:54:17] <wikibugs>	 10Openstack-Magnum: Create a community offering of OpenStack Magnum - https://phabricator.wikimedia.org/T328712#10114061 (10rook)
[16:55:28] <wmcs-alerts>	 RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-control-11 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[17:02:27] <wikibugs>	 10Quarry: Upgrade quarry os - https://phabricator.wikimedia.org/T340762#10114095 (10rook) 05Open→03Resolved a:03rook
[17:17:33] <wikibugs>	 (03update) 10dcaro: kyverno_pod_policy: remove kyverno versions from annotations [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/61 (https://phabricator.wikimedia.org/T359641)
[17:23:02] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10114197 (10dcaro) Merged and deployed :), got a bit in between the cracks
[17:23:38] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 14): [lima-kilo, k8s] Upgrade Kubernetes in lima-kilo to version 1.26 - https://phabricator.wikimedia.org/T370244#10114210 (10dcaro) 05In progress→03Resolved
[17:23:42] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10114206 (10dcaro) 05Open→03Resolved a:03dcaro
[17:23:53] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Please re-install "joe" - https://phabricator.wikimedia.org/T371556#10114218 (10dcaro) >>! In T371556#10034913, @Plenz wrote: > OK, so please install "joe" on the permant operating system and then tell me how to login.  You can use login.toolforge.o...
[17:29:17] <wikibugs>	 10Toolforge (Toolforge iteration 14): Install msbuild - https://phabricator.wikimedia.org/T311454#10114230 (10dcaro) 05Open→03Resolved a:03dcaro I think this is fixed already on the parent task using the build service (https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Buildpack_.NET_tool), cl...
[17:35:38] <wikibugs>	 10Cloud-VPS: move project to tofuinfratest - https://phabricator.wikimedia.org/T365830#10114318 (10rook)
[17:35:54] <wikibugs>	 10Cloud-VPS: move project to tofuinfratest - https://phabricator.wikimedia.org/T365830#10114321 (10rook) Created tofuinfratest project. Added to metrics infra (https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring#Monitoring_for_Cloud_VPS): ` INSERT INTO alerts VALUES (NULL, 240, 'ApplyFailed', '...
[17:36:33] <wikibugs>	 10Cloud-VPS: move project to tofuinfratest - https://phabricator.wikimedia.org/T365830#10114322 (10rook) 05Open→03Resolved
[17:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[17:52:04] <wikibugs>	 (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:52:40] <wikibugs>	 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10114450 (10bd808) I have just emailed the Wikimedia Foundation's [[https://meta.wikimedia.org/wiki/Wikimedia_Foundation/Legal/Community_Resilience_and_Sustainability/...
[17:56:23] <wikibugs>	 (03open) 10marrivs: Added search while selecting options [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/5 (https://phabricator.wikimedia.org/T369400)
[18:11:40] <wikibugs>	 10Tool-yearinreview: Center Align Username Placeholder for Consistency - https://phabricator.wikimedia.org/T373919 (10marrivs) 03NEW
[18:13:21] <wikibugs>	 10Tool-yearinreview, 07good first task: Center Align Username Placeholder for Consistency - https://phabricator.wikimedia.org/T373919#10114552 (10marrivs) Thank you for tagging this task with #good_first_task for Wikimedia newcomers!  Newcomers often may not be aware of things that may seem obvious to seasoned...
[18:38:34] <wikibugs>	 (03update) 10raymond-ndibe: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[18:40:12] <wikibugs>	 10Toolforge, 13Patch-For-Review: [infra, k8s, webservice] remove deprecated kubectl --wait flag before k8s 1.29 upgrade - https://phabricator.wikimedia.org/T373866#10114629 (10Slst2020) 05Open→03In progress thanks @LucasWerkmeister!  you know... if you want to be part of the k8s upgrades workgroup... just...
[18:43:59] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[18:44:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[18:50:08] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
[18:50:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:01:05] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[19:01:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:07:16] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission
[19:07:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:12:54] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
[19:12:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:14:21] <wikibugs>	 10Cloud-VPS: object store for tf-infra-test - https://phabricator.wikimedia.org/T365820#10114760 (10rook) 05Open→03Resolved
[19:17:57] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
[19:18:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:20:57] <wikibugs>	 (03approved) 10raymond-ndibe: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:21:04] <wikibugs>	 (03merge) 10raymond-ndibe: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:21:37] <wikibugs>	 (03update) 10raymond-ndibe: volume-admission: bump to 0.0.54-20240902122354-18500992 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:23:20] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[19:23:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:28:50] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[19:28:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:29:43] <wm-bot2>	 !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[19:29:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:36:02] <wm-bot2>	 !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[19:36:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:36:27] <wikibugs>	 (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.54-20240902122354-18500992 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:36:49] <wikibugs>	 (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.54-20240902122354-18500992 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:37:11] <wikibugs>	 (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:40:29] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[19:40:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:47:06] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
[19:47:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[19:48:25] <wm-bot2>	 !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
[19:48:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:53:41] <wm-bot2>	 !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
[19:53:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:54:23] <wikibugs>	 (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:54:29] <wikibugs>	 (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:54:56] <wikibugs>	 (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:56:10] <wm-bot2>	 !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
[19:56:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[20:01:31] <wm-bot2>	 !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[20:01:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[20:19:54] <wm-bot2>	 !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[20:19:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:20:45] <wikibugs>	 (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:20:51] <wikibugs>	 (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:21:06] <wikibugs>	 (03update) 10raymond-ndibe: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:24:59] <wikibugs>	 (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:25:04] <wikibugs>	 (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:45:40] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#10115189 (10Raymond_Ndibe)
[20:47:36] <wikibugs>	 (03approved) 10raymond-ndibe: jobs,cronjobs: add clarifying note on why the limits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/60 (https://phabricator.wikimedia.org/T372720) (owner: 10dcaro)
[20:48:54] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Possible error in jobs and cronjobs quotas in maintain-kubeusers - https://phabricator.wikimedia.org/T372720#10115205 (10Raymond_Ndibe) a:05Raymond_Ndibe→03dcaro
[20:49:24] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#10115194 (10Raymond_Ndibe) 05In progress→03Resolved
[20:49:25] <wikibugs>	 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: Possible error in jobs and cronjobs quotas in maintain-kubeusers - https://phabricator.wikimedia.org/T372720#10115207 (10Raymond_Ndibe) 05Open→03Resolved
[21:22:07] <wikibugs>	 10Horizon, 13Patch-For-Review: Use IDP for authentication in Horizon - https://phabricator.wikimedia.org/T359590#10115293 (10Andrew)
[21:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:24:01] <wikibugs>	 10Tool-paulina: Set up of local versions - https://phabricator.wikimedia.org/T370226#10115482 (10Pepe_piton) 05Open→03In progress I forked the main repository and started [[ https://gitlab.wikimedia.org/pepepiton/paulina-local | "paulina local" repository ]] to develop the localizable version of Paulina. Lat...