[07:23:28] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: [wmcs-cookbooks] changes to openstack cli / auth things broke several cookbooks - https://phabricator.wikimedia.org/T346427#10754509 (10dcaro) 05Open→03Resolved I think we can close it, if there was any cookbook missing changing we did not use it xd
[08:23:38] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-api,builds-service,builds-cli] toolforge build --envvar does not accept values containing equals character - https://phabricator.wikimedia.org/T389694#10754532 (10Don-vip) Yay, thank you! It works now:  ` tools.spacemedia@tools-bastion-12...
[09:19:28] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-meta-monitor-1 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:38:31] <wikibugs>	 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10754623 (10taavi) 05Open→03Resolved
[09:39:05] <wikibugs>	 10Quarry: worker nodes issue with garbage collection - https://phabricator.wikimedia.org/T375997#10754627 (10taavi)
[09:39:10] <wikibugs>	 06cloud-services-team, 10Quarry: quarry.wmcloud.org: "This web service cannot be reached" due to redis pod running out of disk space - https://phabricator.wikimedia.org/T392107#10754628 (10taavi)
[09:40:38] <wikibugs>	 10Quarry: Scrape prometheus metrics from Quarry - https://phabricator.wikimedia.org/T360220#10754634 (10taavi) 05Open→03Resolved a:05brouberol→03taavi
[09:40:39] <wikibugs>	 06cloud-services-team, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#10754632 (10taavi) Untagging #Quarry which has an active subtask.
[09:42:11] <wikibugs>	 10Quarry, 10observability: Develop the monitoring of Quarry - https://phabricator.wikimedia.org/T205150#10754639 (10taavi) →14Duplicate dup:03T392138
[09:42:13] <wikibugs>	 06cloud-services-team, 10Quarry: No alerting for quarry - https://phabricator.wikimedia.org/T392138#10754641 (10taavi)
[09:59:54] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of wmgmc VPS project - https://phabricator.wikimedia.org/T391742#10754651 (10XtexChooser) Hi @dcaro, `wmgmc_monitoring` is okay. Thanks!
[10:00:16] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of wmgmc_monitoring VPS project - https://phabricator.wikimedia.org/T391742#10754652 (10XtexChooser)
[10:07:45] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of futureaudiences VPS project - https://phabricator.wikimedia.org/T389158#10754670 (10fnegri) @derenrich are you still interested in a Cloud VPS project with a more restricted scope?
[10:10:34] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of antivandalismai VPS project - https://phabricator.wikimedia.org/T388151#10754673 (10fnegri) 05Stalled→03Invalid @ItsNyoty I'm closing this task as "Invalid", feel free to reopen if you are still interested in a Cloud VPS project.
[10:13:23] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of wmgmc_monitoring VPS project - https://phabricator.wikimedia.org/T391742#10754675 (10fnegri) 05Open→03In progress a:03fnegri
[10:17:26] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 wmgmc_monitoring START - Cookbook wmcs.vps.create_project for project wmgmc_monitoring in eqiad1
[10:17:27] <stashbot>	 fnegri@cloudcumin1001: Unknown project "wmgmc_monitoring"
[10:20:26] <logmsgbot_cloud>	 fnegri@cloudcumin1001 create_project (PID 3913634) is awaiting input
[10:20:35] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 wmgmc_monitoring END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project wmgmc_monitoring in eqiad1
[10:20:36] <stashbot>	 fnegri@cloudcumin1001: Unknown project "wmgmc_monitoring"
[10:21:43] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 wmgmc-monitoring START - Cookbook wmcs.vps.create_project for project wmgmc-monitoring in eqiad1 (T391742)
[10:21:43] <stashbot>	 fnegri@cloudcumin1001: Unknown project "wmgmc-monitoring"
[10:21:44] <stashbot>	 T391742: Request creation of wmgmc_monitoring VPS project - https://phabricator.wikimedia.org/T391742
[10:22:21] <wikibugs>	 (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project wmgmc-monitoring [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/201 (https://phabricator.wikimedia.org/T391742)
[10:24:19] <wikibugs>	 (03merge) 10fnegri: projects: added project wmgmc-monitoring [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/201 (https://phabricator.wikimedia.org/T391742) (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49)
[10:27:59] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 wmgmc-monitoring END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project wmgmc-monitoring in eqiad1 (T391742)
[10:28:03] <stashbot>	 T391742: Request creation of wmgmc_monitoring VPS project - https://phabricator.wikimedia.org/T391742
[10:28:18] <wikibugs>	 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of wmgmc-monitoring VPS project - https://phabricator.wikimedia.org/T391742#10754689 (10fnegri)
[10:29:04] <wikibugs>	 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of wmgmc-monitoring VPS project - https://phabricator.wikimedia.org/T391742#10754690 (10fnegri) 05In progress→03Resolved Project created! I had to change the name again from `wmgmc_monitoring` to `wmgmc-monitoring`, because underscore...
[13:20:35] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10754764 (10Jclark-ctr) @fnegri. I installed several blanking panels on the hot aisle to help prevent hot air bleeding into cold aisle  be...
[13:22:02] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10754765 (10Jclark-ctr) Also updated idrac firmware from   5.00.20.00  to  7.00.00.181
[13:30:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10754774 (10fgiunchedi) I tested provisioning `filippo-centrallog-01.o11y.eqiad1.wikimedia.cloud` with the following metadata at instance creation time (passed to nova python api): `meta=...
[13:44:32] <wikibugs>	 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294 (10hashar) 03NEW
[13:56:48] <wikibugs>	 (03open) 10grapedog: in the "all languages" table, show actually all languages, not just the ones that changed [toolforge-repos/wsstats] - 10https://gitlab.wikimedia.org/toolforge-repos/wsstats/-/merge_requests/1
[14:22:23] <wikibugs>	 (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/18
[14:24:01] <wikibugs>	 (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/18 (owner: 10l10n-bot)
[14:24:03] <wikibugs>	 (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/18 (owner: 10l10n-bot)
[15:03:24] <wikibugs>	 14cloud-services-team (Kanban), 10GitLab (Project Migration): GitLab: separate trees for WMCS infrastructure and tenants - https://phabricator.wikimedia.org/T305301#10755003 (10bd808)
[15:25:33] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755081 (10bd808) Normally I would +1 this, but since I'm a member of the project somebody else probably should.
[15:34:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755126 (10fnegri) Am I reading this correctly and 16 CPU and 40GB are the desired final values, not the desired amount of...
[16:15:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755194 (10dcaro) +1
[16:18:36] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 zuul3 START - Cookbook wmcs.openstack.quota_increase (T392294)
[16:18:40] <stashbot>	 T392294: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294
[16:18:44] <logmsgbot_cloud>	 !log fnegri@cloudcumin1001 zuul3 END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T392294)
[16:19:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755198 (10fnegri) 05Open→03Resolved a:03fnegri Increased quotas by 8 cores, 24576 ram.
[17:09:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755290 (10bd808) @fnegri Can we get access to the `g4.cores8.ram24.disk20.ephemeral90.4xiops` flavor as well? Horizon...
[17:17:09] <wikibugs>	 (03open) 10tburmeister: Redo form implementation and improve overall design and UI [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/1 (https://phabricator.wikimedia.org/T390390)
[17:18:14] <wikibugs>	 (03update) 10tburmeister: Redo form implementation and improve overall design and UI [toolforge-repos/tech-doc-metrics] - 10https://gitlab.wikimedia.org/toolforge-repos/tech-doc-metrics/-/merge_requests/1 (https://phabricator.wikimedia.org/T390390)
[17:54:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755340 (10fnegri) @bd808 I'm confused about the state of g3 vs g4 flavors.  The flavors tracked by tofu-infra are onl...
[18:11:38] <wikibugs>	 (03open) 10andrew: proxies: handle 400 return code from proxy API [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/1
[18:19:57] <wikibugs>	 (03close) 10andrew: proxies: handle 400 return code from proxy API [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/1
[18:23:45] <wikibugs>	 (03open) 10andrew: web proxy: don't error out if a proxy doesn't exist. [repos/cloud/cloud-vps/terraform-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/terraform-cloudvps/-/merge_requests/3 (https://phabricator.wikimedia.org/T391718)
[18:26:05] <wikibugs>	 (03open) 10andrew: proxies: handle 400 return code from proxy API [repos/cloud/cloud-vps/go-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps/-/merge_requests/2
[20:01:06] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755510 (10bd808) >>! In T392294#10755339, @fnegri wrote: > @bd808 I'm confused about the state of g3 vs g4 flavors. >...
[20:10:39] <wikibugs>	 (03open) 10bd808: flavors: Expose g4.cores8.ram24.disk20.ephemeral90.4xiops to zuul3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/202 (https://phabricator.wikimedia.org/T392294)
[20:11:29] <wikibugs>	 (03update) 10bd808: flavors: Expose g4.cores8.ram24.disk20.ephemeral90.4xiops to zuul3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/202 (https://phabricator.wikimedia.org/T392294)
[21:14:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10755729 (10Andrew) Just so we know where the setting is getting ignored, can you try the same from the UI and add the setting in the user data panel? Just in case I was fooling myself an...
[21:16:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Quota increase for zuul3 project - https://phabricator.wikimedia.org/T392294#10755734 (10Andrew) We can do a purge and remove unused flavors if you like. Until convinced othe...