[00:20:42] FIRING: CloudVPSDesignateLeaks: Detected 41 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [02:27:49] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:20:42] FIRING: CloudVPSDesignateLeaks: Detected 40 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [04:52:37] 06cloud-services-team, 10Toolforge: Your template file (/data/project/spamcheck/service.template) contains unknown keys: - extra_args - https://phabricator.wikimedia.org/T380141 (10Count_Count) 03NEW [05:07:49] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [07:51:22] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [08:00:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:13:57] (03update) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [08:14:54] (03merge) 10raymond-ndibe: [lima-kilo] cache disk for caching container images [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/201 (https://phabricator.wikimedia.org/T378180) [08:14:55] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [08:15:10] (03approved) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [08:15:14] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [08:15:49] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:16:02] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:16:13] (03merge) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [08:16:14] (03approved) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:16:15] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:20:01] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:21:09] (03merge) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [08:23:02] (03open) 10raymond-ndibe: [lima-kilo] support maintain-harbor functional tests [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/214 (https://phabricator.wikimedia.org/T358225) [08:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [08:29:22] (03merge) 10raymond-ndibe: [maintain-harbor] do not clean up images currently running in production [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/35 (https://phabricator.wikimedia.org/T377854) [08:30:19] (03update) 10raymond-ndibe: [toolforge-deploy] deploy maintain-harbor [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/563 (https://phabricator.wikimedia.org/T358225) [08:47:39] (03approved) 10raymond-ndibe: [envvars-api] k8s.io deps to 0.28.14 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48 (https://phabricator.wikimedia.org/T362867) [08:47:43] (03merge) 10raymond-ndibe: [envvars-api] k8s.io deps to 0.28.14 [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/48 (https://phabricator.wikimedia.org/T362867) [08:47:48] (03approved) 10raymond-ndibe: [registry-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17 (https://phabricator.wikimedia.org/T362867) [08:47:52] (03merge) 10raymond-ndibe: [registry-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/17 (https://phabricator.wikimedia.org/T362867) [08:47:54] (03approved) 10raymond-ndibe: [ingress-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T362867) [08:48:00] (03merge) 10raymond-ndibe: [ingress-admission] k8s.io deps to 0.28.14 [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/14 (https://phabricator.wikimedia.org/T362867) [09:07:46] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10330180 (10dcaro) >>! In T379924#10325730, @aborrero wrote: > this is what I was referring to: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-ema... [09:09:06] (03update) 10dcaro: emailer: run webserver in a different thread [repos/cloud/toolforge/jobs-emailer] (adopt_common_practices) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/9 (https://phabricator.wikimedia.org/T379924) (owner: 10aborrero) [09:12:22] (03update) 10raymond-ndibe: [maintain-harbor] specify config through environment variables [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/36 (https://phabricator.wikimedia.org/T358225) [09:13:36] (03close) 10dcaro: webserver: add a minimal metrics endpoint [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7 (https://phabricator.wikimedia.org/T320284) [09:13:47] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] (allow_getting_config_from_env) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [09:14:57] (03update) 10raymond-ndibe: [toolforge-deploy] kube-state-metrics v5.16.4 --> v5.18.0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/586 (https://phabricator.wikimedia.org/T362867) [09:39:37] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 07Epic: Function infrastructure for Cloud/Toolforge ("Wikimedia Cloud Functions") - https://phabricator.wikimedia.org/T379704#10330245 (10fnegri) Somewhat relevant to this discussion, I just found this commented document explaining how and why... [09:43:09] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge, 07Epic: Function infrastructure for Cloud/Toolforge ("Wikimedia Cloud Functions") - https://phabricator.wikimedia.org/T379704#10330251 (10aborrero) [10:27:45] (03merge) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [10:27:49] (03update) 10dcaro: emailer: run webserver in a different thread [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/9 (https://phabricator.wikimedia.org/T379924) (owner: 10aborrero) [10:29:40] (03open) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 [10:38:59] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-emailer: bump to 0.0.44-20241118103718-c9ddd271 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/601 [10:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:53:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [10:57:56] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [11:02:00] (03open) 10aborrero: nova_fullstack_test.py: add exec permissions [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/6 [11:09:39] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [11:12:35] (03open) 10lucaswerkmeister: Don’t warn about extra_args in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/64 (https://phabricator.wikimedia.org/T380141) [11:12:53] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Your template file (/data/project/spamcheck/service.template) contains unknown keys: - extra_args - https://phabricator.wikimedia.org/T380141#10330603 (10LucasWerkmeister) a:03LucasWerkmeister [11:14:59] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [11:25:49] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [11:27:59] (03approved) 10dcaro: Don’t warn about extra_args in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/64 (https://phabricator.wikimedia.org/T380141) (owner: 10lucaswerkmeister) [11:28:23] (03approved) 10dcaro: jobs-emailer: bump to 0.0.44-20241118103718-c9ddd271 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/601 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:28:27] (03merge) 10dcaro: jobs-emailer: bump to 0.0.44-20241118103718-c9ddd271 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/601 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:37:58] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Your template file (/data/project/spamcheck/service.template) contains unknown keys: - extra_args - https://phabricator.wikimedia.org/T380141#10330642 (10LucasWerkmeister) a:05LucasWerkmeister→03None (Placing the task back up for grabs because the r... [12:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:21:09] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/23 [12:27:01] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [12:43:36] (03merge) 10dcaro: Don’t warn about extra_args in service.template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/64 (https://phabricator.wikimedia.org/T380141) (owner: 10lucaswerkmeister) [12:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:51:42] (03open) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [12:53:37] (03update) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [12:54:38] (03update) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [12:54:40] (03update) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [12:59:37] (03approved) 10aborrero: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 (owner: 10dcaro) [13:16:07] (03update) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [13:17:12] (03merge) 10dcaro: bump_version: add bump branch creation [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/65 [13:20:40] (03merge) 10aborrero: nova_fullstack_test.py: add exec permissions [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/6 [13:23:30] (03open) 10aborrero: verify_dns_reverse_cleanup: fix call [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/7 (https://phabricator.wikimedia.org/T379356) [13:24:13] (03merge) 10aborrero: verify_dns_reverse_cleanup: fix call [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/7 (https://phabricator.wikimedia.org/T379356) [13:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:38:53] (03open) 10dcaro: bump tools webservice [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/66 (https://phabricator.wikimedia.org/T380141) [13:39:05] (03update) 10dcaro: bump tools webservice [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/66 (https://phabricator.wikimedia.org/T380141) [13:40:33] (03close) 10dcaro: bump tools webservice [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/66 (https://phabricator.wikimedia.org/T380141) [13:40:53] (03open) 10dcaro: update bump version [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/67 [13:46:52] (03update) 10dcaro: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 [13:48:00] (03approved) 10aborrero: update bump version [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/67 (owner: 10dcaro) [13:54:45] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [14:03:49] 06cloud-services-team, 10Cloud-VPS, 07IPv6: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174 (10aborrero) 03NEW [14:04:38] 06cloud-services-team, 10Cloud-VPS, 07IPv6: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10331345 (10aborrero) p:05Triage→03Medium [14:05:20] (03merge) 10dcaro: update bump version [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/67 [14:06:15] (03open) 10dcaro: d/changelog: bump to 0.103.15 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/68 (https://phabricator.wikimedia.org/T380141) [14:10:06] (03open) 10aborrero: eqiad1: add IPv6 support [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132 (https://phabricator.wikimedia.org/T380174) [14:18:01] 10Tool-wikiqanda, 06Future-Audiences: Misc bugs from v01 testing - https://phabricator.wikimedia.org/T378850#10331420 (10Aklapper) [14:26:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [14:32:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [14:33:48] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [14:35:57] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [14:39:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [14:39:47] 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10331500 (10Andrew) a:03Andrew [14:41:58] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10331517 (10Andrew) a:03Andrew [14:45:27] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [14:47:10] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: neutron: clarify why DNS extension is not enabled - https://phabricator.wikimedia.org/T377740#10331533 (10Andrew) I'm glad you looked at this. In theory the neutron/dns extension is superior to using designate sink, since it makes updates synchronou... [14:50:23] (03approved) 10dcaro: d/changelog: bump to 0.103.15 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/68 (https://phabricator.wikimedia.org/T380141) [14:50:27] (03merge) 10dcaro: d/changelog: bump to 0.103.15 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/68 (https://phabricator.wikimedia.org/T380141) [15:08:18] RESOLVED: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol1006:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [15:17:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 272714 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [15:32:08] 10tool-wscontest, 07good first task: Add UTC in the WSContest contest page - https://phabricator.wikimedia.org/T331225#10331693 (10AS1100K) a:03AS1100K [15:35:48] 10cloud-services-team (Hardware), 10Cloud-VPS: wmcs codfw hardware changes proposal - https://phabricator.wikimedia.org/T377568#10331724 (10aborrero) [15:40:42] 06cloud-services-team, 10Toolforge: Your template file (/data/project/spamcheck/service.template) contains unknown keys: - extra_args - https://phabricator.wikimedia.org/T380141#10331734 (10dcaro) 05Open→03Resolved a:03dcaro The fix has been deployed, thanks @LucasWerkmeister ! [16:16:44] 10Tools: bullseye's Spur API key has expired - https://phabricator.wikimedia.org/T380193 (10TheresNoTime) 03NEW [16:20:45] 10Tools: bullseye's Spur API key has expired - https://phabricator.wikimedia.org/T380193#10332019 (10TheresNoTime) Thinking out loud, given that #ipoid-service exists, perhaps an API call to //that// can be used in place of Spur directly? [16:35:02] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 [16:46:40] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: CloudVPS: IPv6 in eqiad1 - https://phabricator.wikimedia.org/T380174#10332212 (10aborrero) allocated 2a02:ec80:a000:1::/64 -- cloud-flat-1-eqiad1-dualstack-ipv6 https://netbox.wikimedia.org/ipam/prefixes/1102/ [17:00:29] RESOLVED: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate wikilabels-backups-01.wikilabels.eqiad.wmflabs is about to expire in 26d 2h 55m 26s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [17:07:17] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects: Quota leak for Trove dbs? - https://phabricator.wikimedia.org/T373348#10332391 (10fnegri) A few days after, the quotas were like this: ` fnegri@cloudcontrol1005:~$ sudo OS_PROJECT_ID=tofuinfratest wmcs-openstack database quota show tofuinfratest +---------... [17:18:40] 10Quarry: Allow SQL queries on result-set tables generated from previous queries - https://phabricator.wikimedia.org/T380206 (10Zar2gar1) 03NEW [17:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:22:48] 06cloud-services-team, 10Cloud-VPS, 07IPv6: openstack: nova-fullstack failing in codfw1dev - https://phabricator.wikimedia.org/T380208 (10Andrew) 03NEW [17:29:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:40:42] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Make watchlist table available as curated foo_p.watchlist_count on labsdb - https://phabricator.wikimedia.org/T59617#10332764 (10Zar2gar1) Just swinging by to check the status on this ticket since there hasn't been a comment in a while. I'm... [17:53:24] 10Cloud Services Proposals: Cloud Services: introduce feedback in webpages for some of our services - https://phabricator.wikimedia.org/T332906#10332859 (10Aklapper) Indeed, see T290018 and T248892 [17:59:24] 10Tool-wikiqanda, 06Future-Audiences, 07Epic: Create documentation & qual feedback form for internal bot release - https://phabricator.wikimedia.org/T379791#10332898 (10Maryana) Working on this here: https://docs.google.com/document/d/16697BngJhTMCglb22TEyY9VgLLqOQEXs7s774k9LaJE/edit?usp=sharing [18:19:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:23:01] 10Tool-wikiqanda, 06Future-Audiences: Decide on UX for internal release for bot responses - https://phabricator.wikimedia.org/T380098#10332989 (10Maryana) [18:26:10] 10Tool-wikiqanda, 06Future-Audiences: Slack version of bad response logging - https://phabricator.wikimedia.org/T380216 (10Maryana) 03NEW [18:38:52] 06cloud-services-team, 10Cloud-VPS: Audit WMCS compute capacity - https://phabricator.wikimedia.org/T380099#10333092 (10rook) When running queries in https://thanos.wikimedia.org ` avg by (instance)(sum by (cpu,instance)(rate(node_cpu_seconds_total{instance=~"cloudvirt1.+",mode!="idle"}[2m]))*100) ` works fine... [18:39:17] (03CR) 10Brennen Bearnes: [C:03+1] Index releng/dev-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1091896 (https://phabricator.wikimedia.org/T380133) (owner: 10Daimona Eaytoy) [18:40:22] 10VPS-project-Codesearch, 10dev-images, 13Patch-For-Review: Add releng/dev-images to codesearch - https://phabricator.wikimedia.org/T380133#10333094 (10brennen) [19:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:19:56] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [19:23:22] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [19:23:54] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [19:23:58] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [19:26:16] (03update) 10dcaro: add prometheus stats [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/10 (https://phabricator.wikimedia.org/T320284 https://phabricator.wikimedia.org/T379924) [19:30:31] (03update) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/23 (owner: 10l10n-bot) [19:31:16] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/23 (owner: 10l10n-bot) [19:31:19] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/23 (owner: 10l10n-bot) [19:34:00] 06cloud-services-team, 10Toolforge: Your template file (/data/project/spamcheck/service.template) contains unknown keys: - extra_args - https://phabricator.wikimedia.org/T380141#10333354 (10LucasWerkmeister) Thanks for deploying! [19:47:50] 10Tool-lexeme-forms, 06translatewiki.net: translatewiki export for Wikidata Lexeme Forms tries to remove sh-latn translations - https://phabricator.wikimedia.org/T379188#10333460 (10LucasWerkmeister) The export kept deleting the file through several re-exports; I’ve now manually restored it to unblock all the... [20:59:44] 10Tool-wikiqanda, 06Future-Audiences: Slack version of bad response logging - https://phabricator.wikimedia.org/T380216#10333758 (10Maryana) [20:59:46] 10Tool-wikiqanda, 06Future-Audiences: Flag incorrect answer (internal testing version) - https://phabricator.wikimedia.org/T378821#10333759 (10Maryana) [21:00:22] (03PS1) 10Sbisson: Index research/recommendation-api under services [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1092326 (https://phabricator.wikimedia.org/T379761) [21:04:33] 10VPS-project-Codesearch, 10LPL Hypothesis, 13Patch-For-Review: Index recommendation-api in the code search tool - https://phabricator.wikimedia.org/T379761#10333763 (10SBisson) p:05Triage→03Medium a:03SBisson [23:39:01] (03open) 10bd808: Big pile of Toolforge changes [toolforge-repos/jenkins-build-stats] - 10https://gitlab.wikimedia.org/toolforge-repos/jenkins-build-stats/-/merge_requests/1 (https://phabricator.wikimedia.org/T376830) [23:42:10] (03merge) 10bd808: Big pile of Toolforge changes [toolforge-repos/jenkins-build-stats] - 10https://gitlab.wikimedia.org/toolforge-repos/jenkins-build-stats/-/merge_requests/1 (https://phabricator.wikimedia.org/T376830) [23:44:01] RESOLVED: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 4740 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh