[03:53:32] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team: Request to increase object storage capacity - https://phabricator.wikimedia.org/T392870 (10Scardenasmolinar) 03NEW [04:14:12] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [04:15:24] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [04:45:07] 06cloud-services-team, 10Toolforge: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10775171 (10Raymond_Ndibe) wondering while we just don't fetch the images from k8s config every time we need it. Doesn't feel like a heavy operation and the validation we do to chec... [04:45:27] 06cloud-services-team, 10Toolforge: [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10775174 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:45:48] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api] Periodically refresh image-config data - https://phabricator.wikimedia.org/T357112#10775175 (10Raymond_Ndibe) [04:54:24] 06cloud-services-team, 10Toolforge: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#10775178 (10Raymond_Ndibe) this might work here ` >>> import subprocess >>> subprocess.run( ... ["sh", "-n", "-c", "sh -c 'env;... [05:05:51] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [05:45:59] 06cloud-services-team, 10Toolforge: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#10775194 (10Raymond_Ndibe) a:03Raymond_Ndibe [05:47:54] (03open) 10raymond-ndibe: [jobs-api] validate command string [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T356267) [05:54:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#10775201 (10Raymond_Ndibe) [05:55:12] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#10775203 (10Raymond_Ndibe) 05Open→03In progress [05:55:13] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: Check for diff in services when running diff_with_running_job - https://phabricator.wikimedia.org/T392717#10775205 (10Raymond_Ndibe) 05Open→03In progress [07:15:04] (03PS4) 10Majavah: prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570) [07:17:23] (03CR) 10Majavah: [C:03+2] prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah) [07:17:59] (03Merged) 10jenkins-bot: prometheus: Use DNS names to refer to instances [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139514 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah) [07:21:03] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 07IPv6, 13Patch-For-Review: metricsinfra: Support scraping v6-enabled instances - https://phabricator.wikimedia.org/T392570#10775347 (10taavi) 05Open→03Resolved [07:33:55] (03update) 10taavi: eqiad1: maps-proxy: Import static-proxy rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 (https://phabricator.wikimedia.org/T392826) [07:40:47] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 [07:41:06] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 [07:41:27] (03merge) 10taavi: eqiad1: maps-proxy: Import static-proxy rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/221 (https://phabricator.wikimedia.org/T392826) [07:41:33] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [07:42:07] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [07:43:05] (03open) 10taavi: eqiad1: project-proxy: Fix ethertype [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/222 [07:43:14] (03merge) 10taavi: eqiad1: project-proxy: Fix ethertype [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/222 [07:43:17] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [07:43:55] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [07:45:34] (03open) 10taavi: eqiad1: project-proxy: Add maps-proxy ingress rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 (https://phabricator.wikimedia.org/T392826) [07:45:36] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 [07:46:12] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 [08:08:13] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10775440 (10FCeratto-WMF) The PII data has been sanitized and the nupwiki_p table has been created on an-redacteddb1001.eqiad.wmnet and has the grant applied.... [08:15:07] (03approved) 10aborrero: eqiad1: project-proxy: Add maps-proxy ingress rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 (https://phabricator.wikimedia.org/T392826) (owner: 10taavi) [08:15:20] (03merge) 10taavi: eqiad1: project-proxy: Add maps-proxy ingress rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/223 (https://phabricator.wikimedia.org/T392826) [08:15:22] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [08:16:03] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [08:31:19] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.55 to server 'maps-proxy-5' [08:31:31] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.55 to server 'maps-proxy-5' [08:51:33] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team: Request to increase object storage capacity - Wikilink project - https://phabricator.wikimedia.org/T392870#10775538 (10Peachey88) [08:52:18] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10Wikilink-Tool: Request to increase object storage capacity - Wikilink project - https://phabricator.wikimedia.org/T392870#10775539 (10Peachey88) [09:11:14] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.55 to server 'maps-proxy-6' [09:11:26] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.55 to server 'maps-proxy-6' [09:25:48] 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10775663 (10Tkarcher) >>! In T392781#10773574, @LucasWerkmeister wrote: > Should be done, I think – please check :) Thanks! Looks good - I can log in and access all files. Unfortunate... [09:40:36] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Enable IPv6 for the maps proxy - https://phabricator.wikimedia.org/T392826#10775706 (10taavi) 05Open→03Resolved [09:41:15] 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10775708 (10Tkarcher) 05Open→03Stalled Thanks for following up on this, and I agree with creating a new bot for this as well. But as I just adopted another tool yesterday (T392781) which I'm analy... [10:05:38] 06cloud-services-team, 10Cloud-VPS: replication broken on cloudinfra-db04 - https://phabricator.wikimedia.org/T392889 (10taavi) 03NEW [10:51:18] (03open) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [10:51:32] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [10:52:08] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [10:54:05] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [10:54:33] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175) [10:54:36] T379175: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175 [10:55:08] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [10:55:24] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:00:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [11:00:22] !log taavi@cloudcumin1001 project-proxy END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' [11:02:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance proxy-5 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:05:18] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:05:22] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:05:34] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:07:46] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:09:13] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:09:29] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:10:20] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:10:24] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:10:36] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:11:39] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:11:43] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:11:55] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:13:41] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:13:45] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:13:57] !log taavi@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:14:37] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:14:37] (03update) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:15:13] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 [11:23:36] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.refresh_puppet_certs on proxy-5.project-proxy.eqiad1.wikimedia.cloud [11:25:08] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on proxy-5.project-proxy.eqiad1.wikimedia.cloud [11:26:08] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175) [11:26:10] T379175: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175 [11:26:27] !log taavi@cloudcumin1001 project-proxy END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' (T379175) [11:26:59] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.remove_instance for instance maps-proxy-03 [11:27:21] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance maps-proxy-03 [11:27:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance proxy-5 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [11:27:32] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.remove_instance for instance maps-proxy-04 [11:28:23] !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance maps-proxy-04 [11:28:36] !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'proxy' (T379175) [11:35:08] !log taavi@cloudcumin1001 project-proxy END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'proxy' [11:35:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-6 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [11:39:28] RESOLVED: InstanceDown: Project project-proxy instance maps-proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:40:10] RESOLVED: ProjectProxyMainProxyInstanceDown: Proxy on proxy-6 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [11:45:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-6 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [11:46:24] (03approved) 10aborrero: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) (owner: 10taavi) [11:47:26] (03merge) 10taavi: eqiad1: project-proxy: Reserve main proxy VIP port [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/224 (https://phabricator.wikimedia.org/T379175) [11:47:29] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:48:45] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:57:42] 10Tool-wosretbot, 06Toolforge-standards-committee: Adoption request for wosretbot - https://phabricator.wikimedia.org/T392781#10776131 (10LucasWerkmeister) 05Open→03Resolved a:03LucasWerkmeister Alright, best of luck! [11:58:04] 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10776135 (10LucasWerkmeister) Sure, that’s fair :) [12:44:12] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [12:45:46] 10Tool-refill: Also fill the language= parameter with the reFill tool - https://phabricator.wikimedia.org/T392901 (10Prototyperspective) 03NEW [12:47:28] (03open) 10taavi: eqiad1: project-proxy: Add IPv6 security group rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 (https://phabricator.wikimedia.org/T379175) [12:48:23] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 [12:48:57] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10776314 (10taavi) [12:49:00] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 [12:51:16] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [12:53:46] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [12:54:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:58:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:00:24] (03approved) 10aborrero: eqiad1: project-proxy: Add IPv6 security group rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 (https://phabricator.wikimedia.org/T379175) (owner: 10taavi) [13:00:42] (03merge) 10taavi: eqiad1: project-proxy: Add IPv6 security group rules [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/225 (https://phabricator.wikimedia.org/T379175) [13:01:00] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [13:01:40] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [13:03:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:10:41] 10Tool-wosretbot: [WosretBot] Deletion of empty daily headers on WP:FZW - https://phabricator.wikimedia.org/T392902 (10Tkarcher) 03NEW [13:17:36] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903 (10Galr600) 03NEW [13:23:41] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:23:56] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:25:11] RESOLVED: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [13:37:34] (03PS1) 10Majavah: prometheus: Run blackbox probes on IPv4 [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139858 (https://phabricator.wikimedia.org/T392570) [14:02:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [14:07:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [14:11:28] 10Tool-inteGraality, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10776648 (10jon_amar-WMDE) 05Open→03Declined [14:12:19] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [14:13:05] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10776662 (10bd808) a:05Galr600→03None [14:17:44] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [14:18:06] 10Cloud-VPS (Project-requests): Request creation of MissSearch VPS project - https://phabricator.wikimedia.org/T392903#10776686 (10dcaro) Hi @Galr600! Can you elaborate on the storage needs of your tool? If you can give some numbers and estimations of usage it will help us understand the project needs. If storag... [14:39:11] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [14:45:46] (03open) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [14:45:47] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [components-api] Rename the CRDs groups to be `components-api.toolforge.org` - https://phabricator.wikimedia.org/T386829#10776800 (10dcaro) a:05Raymond_Ndibe→03dcaro [14:46:35] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [components-api] Rename the CRDs groups to be `components-api.toolforge.org` - https://phabricator.wikimedia.org/T386829#10776808 (10dcaro) Pushing this as it's the last think blocking us from deploying in tools [15:01:57] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [15:04:24] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [15:08:17] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [15:08:47] (03CR) 10Majavah: [C:03+2] prometheus: Run blackbox probes on IPv4 [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139858 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah) [15:09:34] (03Merged) 10jenkins-bot: prometheus: Run blackbox probes on IPv4 [cloud/metricsinfra/prometheus-configurator] - 10https://gerrit.wikimedia.org/r/1139858 (https://phabricator.wikimedia.org/T392570) (owner: 10Majavah) [15:27:19] (03update) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [15:32:15] 06cloud-services-team, 10Cloud-VPS: Allow encrypting connection from the shared Cloud VPS web proxy to the backend - https://phabricator.wikimedia.org/T274386#10777003 (10fgiunchedi) Just a quick note/reminder that having the shared Cloud VPS web proxy speak https to its backends would be very nice and welcome [15:37:38] (03update) 10dcaro: crds: use the api name in the crd name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/68 (https://phabricator.wikimedia.org/T386829) [15:39:26] (03update) 10aborrero: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 (owner: 10chuckonwumelu) [15:39:30] (03approved) 10aborrero: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 (owner: 10chuckonwumelu) [15:40:08] (03merge) 10chuckonwumelu: Initial test [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/11 [16:03:19] (03open) 10bd808: wikimedia-zuul: bootstrap new channel [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/24 [16:07:36] (03approved) 10bd808: wikimedia-zuul: bootstrap new channel [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/24 [16:07:40] (03merge) 10bd808: wikimedia-zuul: bootstrap new channel [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/24 [16:09:45] (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) [16:54:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:02:01] 06Toolforge-standards-committee: Adoption request for "request" tool - https://phabricator.wikimedia.org/T389540#10777585 (10LucasWerkmeister) >>! In T389540#10774116, @LucasWerkmeister wrote: > 2. The password of FNBot on https://test-cac.wmflabs.org/. Whatever that was, it seems to be dead now, so we can proba... [17:27:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-6 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:28:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-haproxy-5 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:36:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:39:20] (03open) 10chuckonwumelu: Moved dns provider block for toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/12 [17:42:03] (03update) 10chuckonwumelu: Moved dns provider block for toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/12 [17:43:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-haproxy-5 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:47:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-5 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:49:06] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 06Data-Platform-SRE, and 2 others: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10777833 (10Ahoelzl) [17:56:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:57:28] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:02:28] FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:11:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:12:28] RESOLVED: [5x] PuppetAgentNoResources: No Puppet resources found on instance tools-elastic-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:13:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-haproxy-5 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:21:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance proxy-03 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:28:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-haproxy-5 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [18:39:13] (03open) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [18:43:20] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [18:51:38] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [18:52:08] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [19:39:52] 10Tool-inteGraality, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10778238 (10JeanFred) 05Declined→03Open What is the meaning behind declining this @jon_amar-WMDE? Why shouldn’t I aim to support m... [19:58:31] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [20:10:30] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [20:44:51] (03update) 10chuckonwumelu: Importing tools dns resources [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/13 [20:54:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:32:26] 10Tool-bridgebot: wm-bb eats newlines and doesn't replace with spaces... - https://phabricator.wikimedia.org/T392964 (10Reedy) 03NEW [21:43:54] 10Tool-bridgebot, 07Upstream: wm-bb eats newlines and doesn't replace with spaces... - https://phabricator.wikimedia.org/T392964#10778535 (10bd808) [[https://github.com/42wim/matterbridge/issues/1262|Newlines get lost when Telegram is bridged to other platforms. #1262]] open upstream since 2020-10-17. Upstrea... [21:45:26] 10Tool-bridgebot, 07Upstream: wm-bb Telegram bridge eats newlines and doesn't replace with spaces - https://phabricator.wikimedia.org/T392964#10778542 (10bd808)