[00:44:35] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:44:37] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:44:42] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/41 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:44:52] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/77 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:44:53] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/77 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:44:56] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/77 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:05] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:06] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:09] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/93 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:19] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/102 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:20] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/102 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:23] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/102 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:50] (03update) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:51] (03approved) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:46:54] (03merge) 10raymond-ndibe: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:49:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-54 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [00:49:11] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.95-20250404004707-346b75a8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/736 [01:06:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-33 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:11:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-33 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:13:58] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:18:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:49:34] FIRING: DiskSpace: Disk space cloudcontrol2004-dev:9100:/ 5.686% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2004-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [01:54:34] RESOLVED: DiskSpace: Disk space cloudcontrol2004-dev:9100:/ 5.504% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2004-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:12:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-13 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:17:16] (03update) 10raymond-ndibe: [jobs-api] move core logic to seperate core module [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T359808) [02:17:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-13 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:18:33] (03update) 10raymond-ndibe: [jobs-api] move core logic to seperate core module [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804 https://phabricator.wikimedia.org/T359808) [02:44:14] FIRING: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [02:46:23] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes [02:49:14] RESOLVED: [8x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [03:27:34] FIRING: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 5.75% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:57:34] RESOLVED: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 4.006% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [04:56:34] FIRING: DiskSpace: Disk space cloudcontrol2004-dev:9100:/ 5.706% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2004-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:02:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [05:04:32] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services [05:08:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [05:10:34] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97) on deployment codfw1dev for all services [05:10:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [05:11:34] RESOLVED: DiskSpace: Disk space cloudcontrol2004-dev:9100:/ 3.979% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2004-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:12:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [06:49:36] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [07:02:44] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [07:02:53] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [07:03:03] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [07:03:12] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [07:14:06] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [07:23:58] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [07:24:25] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [07:36:16] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [07:37:38] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [07:51:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [07:53:20] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [08:03:54] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [08:04:22] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [08:04:48] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [08:17:26] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [08:48:57] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:01:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:03:42] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:06:37] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [09:16:31] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:16:59] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [09:17:16] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [09:18:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [09:21:14] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [09:28:38] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [09:30:38] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [09:31:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [09:40:09] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [09:41:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [09:43:46] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [09:52:57] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [09:54:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [10:06:34] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [10:48:22] (03approved) 10aborrero: [jobs-api] stream logs from all containers in all pods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/148 (https://phabricator.wikimedia.org/T388274) (owner: 10raymond-ndibe) [10:48:41] (03update) 10aborrero: [jobs-api] stream logs from all containers in all pods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/148 (https://phabricator.wikimedia.org/T388274) (owner: 10raymond-ndibe) [11:06:38] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10712330 (10Ladsgroup) >>! In T390954#10712161, @taavi wrote: > Yeah. I'm thinking of `termstore.wikidatawiki.{web.analytics}.db.svc.wikimedia.cloud` probably being t... [11:36:11] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/14 (owner: 10l10n-bot) [11:36:14] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/14 (owner: 10l10n-bot) [11:43:16] (03CR) 10Eugene233: "Please check the following comments." [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1132174 (owner: 10Ketulucas) [11:43:28] (03CR) 10Eugene233: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1132174 (owner: 10Ketulucas) [11:47:53] (03update) 10arthurtaylor: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) [11:56:08] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: consider removing labs-ip-aliaser - https://phabricator.wikimedia.org/T374129#10712546 (10aborrero) the particular problem you are describing in your last comment is that both origin and destination virtual machines have floating IPs. cloud... [11:59:25] (03update) 10arthurtaylor: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) [12:00:10] (03update) 10arthurtaylor: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) [12:03:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:04:43] (03update) 10arthurtaylor: Merge cache data from independent test runs [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/6 (https://phabricator.wikimedia.org/T388767) [12:13:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:27:55] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [12:41:14] (03update) 10fnegri: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [12:44:21] PROBLEM - Check DNS auth via TCP of tools-puppetserver-01.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is CRITICAL: DNS CRITICAL - 7.083 seconds response time (tools-puppetserver-01.tools.eqiad1.wikimedia.cloud. 60 IN A 172.16.3.13) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:44:21] PROBLEM - Check DNS auth via UDP of tools-puppetserver-01.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:44:22] PROBLEM - Check DNS auth via TCP of k8s.svc.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:45:11] RECOVERY - Check DNS auth via UDP of tools-puppetserver-01.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is OK: DNS OK - 0.066 seconds response time (tools-puppetserver-01.tools.eqiad1.wikimedia.cloud. 60 IN A 172.16.3.13) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:45:11] RECOVERY - Check DNS auth via TCP of k8s.svc.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is OK: DNS OK - 0.063 seconds response time (k8s.svc.tools.eqiad1.wikimedia.cloud. 300 IN A 172.16.6.113) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:45:15] RECOVERY - Check DNS auth via TCP of tools-puppetserver-01.tools.eqiad1.wikimedia.cloud on server ns1.openstack.eqiad1.wikimediacloud.org on cloudservices1006 is OK: DNS OK - 0.084 seconds response time (tools-puppetserver-01.tools.eqiad1.wikimedia.cloud. 60 IN A 172.16.3.13) https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [12:52:52] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133577 (owner: 10Bovimacoco) [12:55:24] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133587 (owner: 10Bovimacoco) [12:57:34] (03approved) 10fnegri: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) (owner: 10aborrero) [12:58:02] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 13Patch-For-Review, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10712758 (10Andrew) Hello @Volans ! I think I've addressed almost all of your bullet points above -- do you mind retesting? I needed to rebuild the... [13:04:14] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [13:05:31] (03update) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [13:05:52] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132119 (https://phabricator.wikimedia.org/T388195) (owner: 10Juniorbesong) [13:05:56] (03merge) 10aborrero: tofu-infra: refactor flavors into newer shape [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/163 (https://phabricator.wikimedia.org/T375283) [13:08:59] (03update) 10aborrero: flavors: move eqiad1 flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/166 (https://phabricator.wikimedia.org/T375283) [13:10:08] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133586 (owner: 10Bovimacoco) [13:10:34] (03merge) 10aborrero: flavors: move eqiad1 flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/166 (https://phabricator.wikimedia.org/T375283) [13:11:54] (03open) 10aborrero: tofu-infra: cleanup after flavors refactor [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/168 [13:12:45] (03merge) 10aborrero: tofu-infra: cleanup after flavors refactor [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/168 [13:13:02] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:13:29] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:29:28] (03CR) 10Eugene233: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132567 (owner: 10Creative Gurus) [14:20:58] (03CR) 10GergesShamon: "recheck" [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133587 (owner: 10Bovimacoco) [14:22:22] (03CR) 10GergesShamon: [C:03+1] T386329 = Remove app pycache files from git bug: T386329 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133587 (owner: 10Bovimacoco) [15:11:29] (03open) 10aborrero: codfw1dev: add vxlan-ipv4-only.cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud FQDN [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/169 (https://phabricator.wikimedia.org/T380728) [15:13:36] (03merge) 10aborrero: codfw1dev: add vxlan-ipv4-only.cloudinstances2b-gw.svc.codfw1dev.wikimedia.cloud FQDN [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/169 (https://phabricator.wikimedia.org/T380728) [15:13:38] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:14:11] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:33:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:43:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:19:59] (03open) 10piracalamina: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 [16:20:14] (03update) 10piracalamina: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 [16:20:30] (03update) 10piracalamina: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 [16:20:41] (03update) 10piracalamina: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 [16:29:24] (03update) 10piracalamina: T388575 Filter: is author in public domain [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/2 [16:33:37] 10Tool-paulina: Filter: is author in public domain - https://phabricator.wikimedia.org/T388575#10713663 (10Piracalamina) [16:58:41] 10Tool-documentation, 10Adiutor, 06Wiki-Mentor-Africa, 03WMA-Hackathon-2025: [Documentation] Update and improve docs for Adiutor tool - https://phabricator.wikimedia.org/T390392#10713768 (10Eugene233) [17:03:27] 10Tool-documentation, 06Wiki-Mentor-Africa, 05Goal: Review at least 100 Toolforge tools each month. - https://phabricator.wikimedia.org/T363664#10713784 (10Eugene233) [17:05:03] 10Tool-documentation, 07Documentation: Request for Documentation: Python Flask OAuth2 Integration - https://phabricator.wikimedia.org/T382634#10713797 (10Eugene233) >>! In T382634#10420012, @Chinmayee_Mishra wrote: >>>! In T382634#10419977, @Gopavasanth wrote: >> @Chinmayee_Mishra please also include informati... [18:24:10] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10714061 (10debt) a:03Sarai-WMF