[00:35:25] 06cloud-services-team, 10Toolforge: mbh can't login to Toolforge - https://phabricator.wikimedia.org/T389704#10672098 (10bd808) It could have been a temporary LDAP lookup failure. In Cloud VPS and by extension Toolforge the ssh public key information used to decide if your login was successful is stored in a O... [07:31:22] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917 (10Novem_Linguae) 03NEW [07:34:11] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#10672548 (10Novem_Linguae) [07:37:42] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#10672550 (10Novem_Linguae) I propose deleting the 4GB error.log file as a mitigation to keep from using too much disk space. Any objections? Root cause will still need solving, can be solved l... [07:40:14] 10Tool-refill, 07Regression: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T389918 (10Novem_Linguae) 03NEW [07:40:24] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#10672565 (10Curb_Safe_Charmer) Agreed, please go ahead [07:43:30] 10Tool-refill, 07Regression: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T389918#10672566 (10Curb_Safe_Charmer) a:03Novem_Linguae [07:50:31] 10Tool-refill, 07Regression: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T389918#10672571 (10Novem_Linguae) 05Open→03Resolved a:05Novem_Linguae→03SD0001 Unassigning myself. Usually on Phab we only assign tickets when a person has written a patch or is working on th... [07:52:00] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#10672574 (10Novem_Linguae) error.log deleted. Will leave this open since this is likely to come back and root cause will need investigating. [08:27:53] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx [08:39:09] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx [08:39:36] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx [08:39:41] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx [08:42:15] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx [08:52:45] 10wikitech.wikimedia.org: wikitech: Current deployments sidebar link uses a new line on new Vector - https://phabricator.wikimedia.org/T389923 (10taavi) 03NEW [08:55:17] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx [08:58:38] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade ingress-nginx to v1.12.1+ - https://phabricator.wikimedia.org/T383516#10672733 (10taavi) [09:01:06] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade ingress-nginx to v1.12.1+ - https://phabricator.wikimedia.org/T383516#10672741 (10taavi) 1.12.0 disables metrics by default, we will need to re-enable that. [09:01:25] supertassu opened https://github.com/toolforge/paws/pull/484 [09:14:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:17:56] 06cloud-services-team, 10Toolforge: mbh can't login to Toolforge - https://phabricator.wikimedia.org/T389704#10672788 (10dcaro) There were some (extra) issues with ldap during the weekend yep, I think it's very likely related, I'd say we can close this and reopen if you see any new issues, or leave it open for... [09:21:13] 06cloud-services-team, 10Toolforge: mbh can't login to Toolforge - https://phabricator.wikimedia.org/T389704#10672795 (10MBH) 05Open→03Resolved a:03MBH We could close this, no problem. Could you also see T389733? [09:23:34] supertassu closed https://github.com/toolforge/paws/pull/484 [09:27:33] (03PS18) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [09:32:45] (03PS19) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [09:43:29] 10Toolforge (Quota-requests): Increase RAM quota for mbh tool - https://phabricator.wikimedia.org/T389733#10672875 (10dcaro) Currently toolforge workers have 16G of RAM each, this means that 16G is a hard upper limit. Said that, the scheduling will have a really hard time finding a 16G free slot in the fleet (as... [09:44:51] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [components-api] restrict running deployments to 1 - https://phabricator.wikimedia.org/T388643#10672901 (10Aklapper) [09:44:56] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:44:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:47:00] !log aborrero@cloudcumin1001 traffic START - Cookbook wmcs.openstack.quota_increase (T389196) [09:47:02] T389196: Increase quota for Traffic cloud project - https://phabricator.wikimedia.org/T389196 [09:47:08] !log aborrero@cloudcumin1001 traffic END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T389196) [09:47:45] 10Cloud-VPS (Quota-requests), 06Traffic: Increase quota for Traffic cloud project - https://phabricator.wikimedia.org/T389196#10672930 (10aborrero) 05Open→03Resolved Sorry this took so long, the person in clinic duty overlooked it. [09:57:24] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [09:57:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:16:32] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:16:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:25:23] (03CR) 10David Caro: "What happens if while this is running the puppet one runs also? (until now it would fail)" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:27:41] (03CR) 10Arturo Borrero Gonzalez: "with this change, the only contact point is the `cp` operation from the main `/srv/tofu-infra` directory into a temporal directory." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:28:59] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [10:29:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:31:31] (03CR) 10David Caro: "What I mean is, you run this cookbook, it starts running the apply of the checked out MR." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:32:11] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [10:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:32:34] (03CR) 10Arturo Borrero Gonzalez: "You can only run `plan` for MRs. To run `apply`, the cookbook only allows that for the main branch." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:32:41] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [10:32:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:42:12] (03CR) 10Arturo Borrero Gonzalez: "Also, puppet does not run `apply` either, that's always a manual operation (so far)." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:43:31] (03CR) 10David Caro: [C:03+1] "I see, `_with_merge_request` is quite misleading for a name that does not always have to do anything with merge request xd" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:45:53] (03PS20) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) [10:46:15] (03CR) 10Arturo Borrero Gonzalez: wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:58:05] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs.openstack.tofu: when dealing with MR checkouts, operate on a different dir [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1129305 (https://phabricator.wikimedia.org/T374022) (owner: 10Arturo Borrero Gonzalez) [10:58:34] 06cloud-services-team, 10Cloud-VPS, 07Epic, 13Patch-For-Review: tofu-infra: the cookbook should use a different git tree copy than the main one - https://phabricator.wikimedia.org/T374022#10673483 (10aborrero) 05Open→03Resolved [10:59:18] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Gather usage stats - https://phabricator.wikimedia.org/T381587#10673501 (10aborrero) I created this https://grafana-rw.wikimedia.org/d/a688b60e-d3a8-478c-8c75-9b06c1ea32a9/wiki-replicas?orgId=1&from=now-90d&to=now [11:20:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [11:32:56] 10Toolforge (Toolforge iteration 19): [components-api] use the component name for the image instead of the default tool - https://phabricator.wikimedia.org/T388830#10673621 (10dcaro) a:03dcaro [11:33:01] 10Toolforge (Toolforge iteration 19): [components-api] use the component name for the image instead of the default tool - https://phabricator.wikimedia.org/T388830#10673623 (10dcaro) 05Open→03In progress [11:33:06] 10Toolforge (Toolforge iteration 19): [components-api] restrict running deployments to 1 - https://phabricator.wikimedia.org/T388643#10673626 (10dcaro) 05In progress→03Resolved [11:41:43] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: network problems when introducing new networks - https://phabricator.wikimedia.org/T380728#10673664 (10cmooney) I'd forgot about this task, apologies. The reason the problems occurred last time with this is the cloud switches had n... [11:49:26] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Gather usage stats - https://phabricator.wikimedia.org/T381587#10673669 (10fnegri) The current prometheus metrics only give us the total number of queries (including the ones from the replication threads). Enabling https://mariadb.com/... [12:14:01] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:14:54] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:21:51] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:22:23] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:45:49] 06cloud-services-team, 10Cloud-VPS: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942 (10aborrero) 03NEW [12:46:10] 06cloud-services-team, 10Cloud-VPS: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10673856 (10aborrero) 05Open→03In progress p:05Triage→03Medium [12:54:46] 06cloud-services-team, 10Cloud-VPS: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10673882 (10aborrero) see also: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/157 [13:05:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:53:48] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10674145 (10aborrero) [13:56:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:58:07] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 [14:02:28] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 [14:04:07] 10Cloud-VPS (Project-requests): Request creation of antivandalismai VPS project - https://phabricator.wikimedia.org/T388151#10674188 (10ItsNyoty) Hi thank you for reaching out, I was going to train the models on the cloud services but since WikiMedia doesn't have GPU's VMs available, I will try to look at the d... [14:04:21] 10Cloud-VPS (Project-requests): Request creation of antivandalismai VPS project - https://phabricator.wikimedia.org/T388151#10674190 (10ItsNyoty) 05Open→03Stalled [14:12:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:16:44] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:29:18] 10Toolforge (Toolforge iteration 19): [builds-api] don't cleanup pending builds - https://phabricator.wikimedia.org/T389954 (10dcaro) 03NEW [14:29:24] 10Toolforge (Toolforge iteration 19): [builds-api] don't cleanup pending builds - https://phabricator.wikimedia.org/T389954#10674295 (10dcaro) p:05Triage→03High [15:02:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:03:00] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:03:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:12:48] 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10674530 (10Chuckonwumelu) [15:14:56] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:18:31] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [15:18:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:23:56] 10Toolforge (Toolforge iteration 19): [builds-api] don't cleanup pending builds - https://phabricator.wikimedia.org/T389954#10674628 (10dcaro) 05Open→03In progress [15:32:54] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [15:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:04:43] 06cloud-services-team, 10Cloud-VPS: tofu-infra: implement some state backup mechanism - https://phabricator.wikimedia.org/T389964 (10aborrero) 03NEW [16:04:50] 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User: Horizon: network topology panel ignores user policy, suggests deleting networks and instances - https://phabricator.wikimedia.org/T389965 (10Andrew) 03NEW [16:09:31] 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User: Horizon: network topology panel ignores user policy, suggests deleting networks and instances - https://phabricator.wikimedia.org/T389965#10674835 (10Andrew) I created upstream bug https://bugs.launchpad.net/horizon/+bug/2104173 [16:12:34] 10Tool-toolwatch: Investigate and Resolve QS-Dev Tools Unavailability Issue - https://phabricator.wikimedia.org/T389967 (10Gopavasanth) 03NEW [16:13:18] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:17:01] 06cloud-services-team, 10Cloud-VPS: tofu-infra: implement some state backup mechanism - https://phabricator.wikimedia.org/T389964#10674888 (10aborrero) p:05Triage→03Medium [16:28:43] (03PS1) 10Arturo Borrero Gonzalez: wmcs inventory: refresh codfw1dev flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131042 (https://phabricator.wikimedia.org/T389942) [16:29:29] 06cloud-services-team, 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Debug-Logger, 06MediaWiki-Platform-Team: Throttler IP logging uses internal IPs - https://phabricator.wikimedia.org/T389887#10674966 (10Scott_French) Thanks for the follow-up @kostajh, and for sharing an example logstash query.... [16:29:35] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs inventory: refresh codfw1dev flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131042 (https://phabricator.wikimedia.org/T389942) (owner: 10Arturo Borrero Gonzalez) [16:30:47] (03PS1) 10Arturo Borrero Gonzalez: cookbooks: update references to flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131043 (https://phabricator.wikimedia.org/T389942) [16:33:18] (03Merged) 10jenkins-bot: wmcs inventory: refresh codfw1dev flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131042 (https://phabricator.wikimedia.org/T389942) (owner: 10Arturo Borrero Gonzalez) [16:35:14] 10Tool-toolwatch: Investigate and Resolve QS-Dev Tools Unavailability Issue - https://phabricator.wikimedia.org/T389967#10674999 (10Hrideshmg) This is because the site blocks HTTP [[ https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/HEAD | HEAD ]] requests. See: `sh (.venv) /tmp/br -> curl --... [16:38:35] 06cloud-services-team, 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Debug-Logger, 06MediaWiki-Platform-Team: Throttler IP logging uses internal IPs - https://phabricator.wikimedia.org/T389887#10675008 (10Andrew) Within the wmcs network those IPs (172.16.1.2 and 172.16.1.149) resolve to specific... [16:50:55] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10675065 (10fnegri) a:05Andrew→03fnegri I'm back from holidays and I'm re-assigning this task to myself. @aborrero is quite confident that upda... [16:54:56] 06cloud-services-team, 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Debug-Logger, 06MediaWiki-Platform-Team: Throttler IP logging uses internal IPs - https://phabricator.wikimedia.org/T389887#10675088 (10Scott_French) Thank you very much for confirming, @Andrew. So, to summarize, these are ind... [16:55:34] 06cloud-services-team, 10Toolforge, 10MediaWiki-Core-AuthManager, 10MediaWiki-Debug-Logger, 06MediaWiki-Platform-Team: Throttler IP logging uses internal IPs - https://phabricator.wikimedia.org/T389887#10675096 (10Scott_French) a:03Tgr [17:02:21] 10Tool-toolwatch: Investigate and Resolve QS-Dev Tools Unavailability Issue - https://phabricator.wikimedia.org/T389967#10675123 (10Arcstur) @Hrideshmg thanks for the explanation. I'll add that to our backlog. I would suggest to document the use of HEAD somewhere in the tool's page. Best regards! [17:37:03] (03PS1) 10Andrew Bogott: Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 [17:37:37] (03CR) 10CI reject: [V:04-1] Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 (owner: 10Andrew Bogott) [17:37:54] 10wikitech.wikimedia.org: wikitech: Current deployments sidebar link uses a new line on new Vector - https://phabricator.wikimedia.org/T389923#10675269 (10Quiddity) Note: I believe that string is added via this code: https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-site-deploycal.js#L-154 It still works as... [17:39:02] (03PS2) 10Andrew Bogott: Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 [17:39:24] (03CR) 10CI reject: [V:04-1] Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 (owner: 10Andrew Bogott) [17:45:59] (03PS3) 10Andrew Bogott: Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 [17:49:29] (03CR) 10Jgreen: [C:03+1] community_civicrm: add stub for dovecot_passwd [labs/private] - 10https://gerrit.wikimedia.org/r/1124204 (https://phabricator.wikimedia.org/T383715) (owner: 10Dwisehaupt) [19:13:22] 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10675811 (10Tacsipacsi) Thanks for fixing! [20:08:23] 10wikitech.wikimedia.org: wikitech: Current deployments sidebar link uses a new line on new Vector - https://phabricator.wikimedia.org/T389923#10676019 (10bd808) a:03Krinkle [20:14:20] (03PS1) 10Andrew Bogott: network_topology: check policy before offering to delete instances or networks [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1131085 (https://phabricator.wikimedia.org/T389965) [20:14:22] (03PS1) 10Andrew Bogott: update .gitreview [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1131086 [20:15:27] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] network_topology: check policy before offering to delete instances or networks [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1131085 (https://phabricator.wikimedia.org/T389965) (owner: 10Andrew Bogott) [20:15:34] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] update .gitreview [openstack/horizon/horizon] (2024.1) - 10https://gerrit.wikimedia.org/r/1131086 (owner: 10Andrew Bogott) [20:54:22] (03PS1) 10Andrew Bogott: Re-enable the network panel for instance creation [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131090 (https://phabricator.wikimedia.org/T380081) [20:54:23] (03PS1) 10Andrew Bogott: launch-instance-model: support default network ID in the network panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131091 (https://phabricator.wikimedia.org/T380081) [20:54:25] (03PS1) 10Andrew Bogott: Revert "Remove some networking panels" [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131092 (https://phabricator.wikimedia.org/T380531) [20:54:26] (03PS1) 10Andrew Bogott: Replace keypair step in instance creation [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131093 [20:54:26] (03PS1) 10Andrew Bogott: network_topology: disable VM creation from this panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131094 [20:54:26] (03PS1) 10Andrew Bogott: network_topology: check policy before offering to delete instances or networks [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131095 (https://phabricator.wikimedia.org/T389965) [20:55:29] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Re-enable the network panel for instance creation [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131090 (https://phabricator.wikimedia.org/T380081) (owner: 10Andrew Bogott) [20:55:36] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] launch-instance-model: support default network ID in the network panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131091 (https://phabricator.wikimedia.org/T380081) (owner: 10Andrew Bogott) [20:55:41] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Revert "Remove some networking panels" [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131092 (https://phabricator.wikimedia.org/T380531) (owner: 10Andrew Bogott) [20:55:46] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Replace keypair step in instance creation [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131093 (owner: 10Andrew Bogott) [20:55:50] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] network_topology: disable VM creation from this panel [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131094 (owner: 10Andrew Bogott) [20:55:57] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] network_topology: check policy before offering to delete instances or networks [openstack/horizon/horizon] - 10https://gerrit.wikimedia.org/r/1131095 (https://phabricator.wikimedia.org/T389965) (owner: 10Andrew Bogott) [20:58:05] 10Tool-refill, 07Regression: https://refill.toolforge.org/ returns 403 - https://phabricator.wikimedia.org/T389918#10676167 (10Novem_Linguae) Taavi gave some advice on Discord that we should perhaps change the service.template file from `type: php5.6` to `type: php8.2`. I went ahead and did that just now.... [20:58:23] (03CR) 10Andrew Bogott: [C:03+2] Update policy checks to use new Nova policy name [openstack/horizon/wmf-puppet-dashboard] - 10https://gerrit.wikimedia.org/r/1131061 (owner: 10Andrew Bogott) [21:01:43] 10Toolforge-standards-committee (Maintainer needed), 10Tools, 06Commons: Steinsplitter's tools need a new maintainer - https://phabricator.wikimedia.org/T350953#10676186 (10Ahecht) I've been running a mirror of Steinsplitter's [[ https://globalusagecount.toolforge.org | GlobalUsageCount ]] tool at https://gl... [21:17:58] 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User, 13Patch-For-Review: Horizon: network topology panel ignores user policy, suggests deleting networks and instances - https://phabricator.wikimedia.org/T389965#10676245 (10Andrew) I think this is fixed in codfw1dev. @aborrero do you agree? [21:18:18] 10Toolforge-standards-committee (Maintainer needed), 10Tools, 06Commons: Steinsplitter's tools need a new maintainer - https://phabricator.wikimedia.org/T350953#10676266 (10AntiCompositeNumber) @Ahecht https://toolsadmin.wikimedia.org/tools/id/globalusagecount lists Steinsplitter as the only maintainer, so i... [22:35:24] 06cloud-services-team, 10Toolforge: kmlexport tool returning 403 Forbidden error - https://phabricator.wikimedia.org/T390009 (10Paul2520) 03NEW [22:43:02] 10Tools: kmlexport tool returning 403 Forbidden error - https://phabricator.wikimedia.org/T390009#10676515 (10bd808) [22:48:06] 10Tools: kmlexport tool returning 403 Forbidden error - https://phabricator.wikimedia.org/T390009#10676520 (10DB111) Works for me, do you have special browser settings / plugins suppressing the referer?