[00:10:21] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: Updating vite to 6.2.3 [labs/striker] - 10https://gerrit.wikimedia.org/r/1131111 (owner: 10Libraryupgrader)
[00:42:57] <wikibugs>	 10Tools: kmlexport tool returning 403 Forbidden error - https://phabricator.wikimedia.org/T390009#10676691 (10bd808) Adding a body to the 403 response that says anything about why an authorization block is being applied might be helpful. There is obviously a thin line to walk here as you don't want to be too exp...
[03:31:14] <wikibugs>	 10Tools: kmlexport tool returning 403 Forbidden error - https://phabricator.wikimedia.org/T390009#10676849 (10Paul2520) @DB111 @bd808 it didn't work on Firefox or Chrome on Mac, nor Firefox on Android. But it did work on Chrome on Android!   I don't think I have any special settings/plugins... but I'm glad I was...
[05:56:36] <wikibugs>	 (03CR) 10Majavah: [C:03+2] "..." [labs/striker] - 10https://gerrit.wikimedia.org/r/1131111 (owner: 10Libraryupgrader)
[08:02:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-37 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[09:10:26] <wikibugs>	 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User: Horizon: network topology panel ignores user policy, suggests deleting networks and instances - https://phabricator.wikimedia.org/T389965#10677365 (10aborrero) >>! In T389965#10676244, @Andrew wrote: > I think this is fixed in codfw1dev. @aborre...
[09:11:12] <wikibugs>	 06cloud-services-team, 10Horizon, 05Cloud-Services-Origin-User, 07Upstream: Horizon: network topology panel ignores user policy, suggests deleting networks and instances - https://phabricator.wikimedia.org/T389965#10677366 (10taavi)
[09:27:18] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: network problems when introducing new networks - https://phabricator.wikimedia.org/T380728#10677422 (10taavi)
[09:40:53] <wikibugs>	 (03PS1) 10Slyngshede: apereo_cas::service: remove unused service entry [labs/private] - 10https://gerrit.wikimedia.org/r/1131280
[10:18:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10677644 (10aborrero)
[10:18:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10677645 (10aborrero) announcement: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/ZEYISHVFSSMGRBB4JQAQEQVD4CAH...
[10:23:36] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10677668 (10github-toolforge-bot) aborrero opened https://github.com/toolforge/paws/pull/485
[10:23:41] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10677669 (10aborrero) created: https://github.com/toolforge/paws/pull/485
[10:23:51] <notefromgithub>	 aborrero opened https://github.com/toolforge/paws/pull/485
[10:31:04] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10677699 (10aborrero) refreshed a couple of wikitech pages: * https://wikitech.wikimedia.org/wiki/Help:Using_OpenTofu_on_Cloud_VPS * https://wikitech....
[10:36:40] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+1] "It used to exist, but was recently decommissioned, so let's link https://phabricator.wikimedia.org/T389172 to the commit message?" [labs/private] - 10https://gerrit.wikimedia.org/r/1131280 (owner: 10Slyngshede)
[10:37:41] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[10:38:14] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[10:45:15] <wikibugs>	 (03PS2) 10Slyngshede: apereo_cas::service: remove unused service entry [labs/private] - 10https://gerrit.wikimedia.org/r/1131280
[10:47:07] <wikibugs>	 (03CR) 10Muehlenhoff: apereo_cas::service: remove unused service entry [labs/private] - 10https://gerrit.wikimedia.org/r/1131280 (owner: 10Slyngshede)
[10:48:29] <wikibugs>	 (03CR) 10David Caro: [C:03+1] cookbooks: update references to flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131043 (https://phabricator.wikimedia.org/T389942) (owner: 10Arturo Borrero Gonzalez)
[10:48:42] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C:03+2] cookbooks: update references to flat network name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1131043 (https://phabricator.wikimedia.org/T389942) (owner: 10Arturo Borrero Gonzalez)
[11:26:22] <wikibugs>	 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10677846 (10fnegri) a:05fnegri→03taavi Thanks @taavi. For some reason I never received an email about this task even if it was assigned to me, but I received the comments in...
[11:39:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove database flavors are out of sync with reality - https://phabricator.wikimedia.org/T390042 (10taavi) 03NEW
[12:00:12] <notefromgithub>	 vivian-rook closed https://github.com/vivian-rook/paws/pull/3
[12:01:23] <notefromgithub>	 vivian-rook closed https://github.com/toolforge/paws/pull/284
[12:02:03] <notefromgithub>	 vivian-rook closed https://github.com/vivian-rook/paws/pull/4
[12:02:25] <notefromgithub>	 vivian-rook closed https://github.com/toolforge/paws/pull/479
[12:02:51] <wikibugs>	 (03CR) 10Slyngshede: [V:03+2 C:03+2] apereo_cas::service: remove unused service entry [labs/private] - 10https://gerrit.wikimedia.org/r/1131280 (owner: 10Slyngshede)
[12:18:30] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10678053 (10aborrero) 05In progress→03Resolved
[12:52:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-37 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[13:40:22] <wikibugs>	 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10678401 (10aborrero) So:  35 DNS toolsbeta 61 DNS tools 15 volumes tools 7 volumes toolsbeta 5 proxies tools 5 proxies toolsbeta 10 floating IPs tools 3 floatings IPs tool...
[13:44:29] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: introduce additional IaC automation - https://phabricator.wikimedia.org/T390056 (10aborrero) 03NEW
[13:44:42] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: introduce additional IaC automation - https://phabricator.wikimedia.org/T390056#10678430 (10aborrero)
[13:44:43] <wikibugs>	 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10678431 (10aborrero)
[13:44:45] <wikibugs>	 06cloud-services-team, 10Toolforge: Analyze Toolforge and Toolsbeta for Virtual Resource Usage - https://phabricator.wikimedia.org/T389081#10678432 (10Chuckonwumelu) 05Open→03Resolved
[13:45:12] <wikibugs>	 06cloud-services-team, 10Toolforge, 07Epic: toolforge: introduce additional IaC automation - https://phabricator.wikimedia.org/T390056#10678440 (10aborrero) p:05Triage→03Medium
[13:47:37] <wikibugs>	 06cloud-services-team, 10Toolforge: bootstrap Toolforge IaC automation - https://phabricator.wikimedia.org/T390057 (10aborrero) 03NEW
[13:48:07] <wikibugs>	 06cloud-services-team, 10Toolforge: bootstrap Toolforge IaC automation - https://phabricator.wikimedia.org/T390057#10678459 (10aborrero) 05Open→03In progress p:05Triage→03Medium
[13:49:29] <wikibugs>	 06cloud-services-team, 10Toolforge: bootstrap Toolforge IaC automation - https://phabricator.wikimedia.org/T390057#10678469 (10aborrero) some implementation examples: * https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/ * https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning...
[13:53:39] <wikibugs>	 06cloud-services-team, 10Data-Services: Update views on global_block_whitelist tables - https://phabricator.wikimedia.org/T387663#10678512 (10Tacsipacsi) >>! In T387663#10677846, @fnegri wrote: > For some reason I never received an email about this task even if it was assigned to me, but I received the com...
[14:05:58] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10678670 (10fnegri) > @dcaro is concerned about active NFS mounts from pods, those might require a restart of the NFS server (unless Puppet is doin...
[14:10:26] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove database flavors are out of sync with reality - https://phabricator.wikimedia.org/T390042#10678706 (10Andrew) Is this the case for lots of VMs or just that one?
[14:19:35] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10678758 (10dcaro) >>! In T383723#10678670, @fnegri wrote: >> @dcaro is concerned about active NFS mounts from pods, those might require a restart...
[14:27:11] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10678812 (10fnegri) > It's not that the service not need restarting, but if there's any processes (ex. pods) that have a file open from before the...
[14:56:01] <wikibugs>	 10Toolforge (Quota-requests): Increase RAM quota for mbh tool - https://phabricator.wikimedia.org/T389733#10678994 (10MBH) @dcaro Let's start from 8 or, better, 12 GB, if you could set 12. I don't want to cause you any unnecessary inconvenience and will try to stay within these limits.  After quote increase, sho...
[15:04:10] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove database flavors are out of sync with reality - https://phabricator.wikimedia.org/T390042#10679024 (10Andrew) 05Open→03Resolved a:03Andrew I resized each of these VMs in the horizon UI and now they display with g4 flavors. Things seem consistent between trove...
[15:23:29] <wikibugs>	 10Tool-documentation, 10gadget-Cat-a-lot, 07Documentation: Write Cat-a-lot developing process documentation - https://phabricator.wikimedia.org/T387057#10679141 (10TBurmeister)
[15:29:31] <wikibugs>	 10Tool-documentation, 07Documentation: Request for Documentation: Python Flask OAuth2 Integration - https://phabricator.wikimedia.org/T382634#10679180 (10TBurmeister)
[15:30:40] <wikibugs>	 10Tool-bub2, 10Tool-documentation, 07Documentation: Documentation for BUB2 Tool - https://phabricator.wikimedia.org/T364082#10679189 (10TBurmeister)
[15:31:41] <wikibugs>	 10Tool-documentation, 10ISA, 06Wiki-Mentor-Africa, 07Documentation: Review, improve the code comments on the tool for programmers - https://phabricator.wikimedia.org/T355473#10679195 (10TBurmeister)
[15:33:54] <wikibugs>	 10Tool-documentation, 05Goal: Review at least 100 Toolforge tools each month. - https://phabricator.wikimedia.org/T363664#10679208 (10TBurmeister)
[16:48:41] <wikibugs>	 10Cloud-VPS (Project-requests), 06Release-Engineering-Team: Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081 (10bd808) 03NEW
[16:49:55] <wikibugs>	 10Cloud-VPS (Project-requests), 06Release-Engineering-Team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081#10679700 (10bd808)
[16:50:51] <wikibugs>	 10Cloud-VPS (Project-requests), 06Release-Engineering-Team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081#10679708 (10taavi) Is this meant to eventually go away or replace the current `integration` project?
[17:06:19] <wikibugs>	 10Cloud-VPS (Project-requests), 06Release-Engineering-Team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081#10679840 (10dcaro) +1 from me (with the extra info requested)  zuul3 neat :)
[17:08:04] <wikibugs>	 10Cloud-VPS (Project-requests), 06Release-Engineering-Team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081#10679856 (10bd808) >>! In T390081#10679708, @taavi wrote: > Is this meant to eventually go away or replace the...
[17:54:24] <wikibugs>	 06cloud-services-team, 10Toolforge: [lima-kilo] ansible random timeouts downloading - https://phabricator.wikimedia.org/T390095 (10fnegri) 03NEW
[17:54:34] <wikibugs>	 06cloud-services-team, 10Toolforge: [lima-kilo] ansible random timeouts downloading - https://phabricator.wikimedia.org/T390095#10680120 (10fnegri)
[17:59:59] <wikibugs>	 06cloud-services-team, 10Toolforge: [lima-kilo] ansible random timeouts downloading - https://phabricator.wikimedia.org/T390095#10680134 (10fnegri) 05Open→03Resolved a:03fnegri Fully recreating the VM with `./start-devenv.sh` and answering `y` was not enough to fix it: it downloaded `kind` successful...
[18:01:16] <wikibugs>	 06cloud-services-team, 10Toolforge: [lima-kilo] ansible random timeouts when downloading files - https://phabricator.wikimedia.org/T390095#10680142 (10fnegri)
[20:22:42] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Project-requests), 06Release-Engineering-Team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul3 VPS project - https://phabricator.wikimedia.org/T390081#10680577 (10bd808)
[20:41:11] <wikibugs>	 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#10680630 (10LibUp-bot) A new upstream version of OpenRefine is now available: 3.9.2. * https://github.com/OpenRefine/OpenRefine/releases/tag/3.9.2
[23:45:20] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T390124 (10Keith_D) 03NEW
[23:53:19] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T390124#10681263 (10Novem_Linguae) Oooh. My time to shine. With my fancy new refill-api access and CurbSafeCharmer's new [[ https://en.wikipedia.org/wiki/Wikipedia:Refill/restart | restart work instruction ]]....
[23:57:24] <wikibugs>	 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T390124#10681277 (10Novem_Linguae) @Keith_D, unable to reproduce when visiting https://refill.toolforge.org/ng/ and performing ReFill on the page "Test" on en.wikipedia. I have not restarted yet. Are you still...