[00:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:05:49] (TfInfraTestDestroyFailed) resolved: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:08:34] 10Grid-Engine-to-K8s-Migration: Migrate shrinitools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320037 (10Tshrinivasan) Shrinitools is not required. no need to migrate it. Please delete the relevant resources. Thanks. [03:06:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [04:06:56] (SystemdUnitDown) firing: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:15:50] 10Grid-Engine-to-K8s-Migration: Migrate flickrdash from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319746 (10Samwilson) 05Open→03Resolved [05:04:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:09:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:01:57] (SystemdUnitDown) firing: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:02:02] 10cloud-services-team: SystemdUnitDown Unit purge_vm_rbd_images.service on node cloudcontrol1005 has been down for long. - https://phabricator.wikimedia.org/T356473 (10phaultfinder) [06:11:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [07:04:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance project-proxy-acme-chief-02 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:09:28] (PuppetAgentNoResources) firing: (2) No Puppet resources found on instance maps-proxy-04 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:19:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance maps-proxy-04 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:03:28] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10tstarling) >>! In T356377#9506478, @dcaro wrote: > I'd say that the quicker solution right now and more stable is just to make... [09:07:47] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10dcaro) >>! In T356377#9508759, @tstarling wrote: >>>! In T356377#9506478, @dcaro wrote: >> I'd say that the quicker solution ri... [09:09:28] (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance maps-proxy-04 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:10:02] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10dcaro) :facepalm: it's a typo, it's `$TOOL_DATA_DIR`, not `$TOOLS_DATA_DIR`, let me give you a tested curl example [09:11:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:12:13] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10tstarling) It's OK, the container just hadn't been restarted since that task was fixed. I see the right environment now. [09:20:08] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10dcaro) >>! In T356377#9508767, @tstarling wrote: > It's OK, the container just hadn't been restarted since that task was fixed.... [09:24:28] (PuppetAgentNoResources) resolved: (2) No Puppet resources found on instance maps-proxy-04 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:40:55] 10Toolforge (Toolforge iteration 04): [toolforge] several tools get periods of connection refused (104) when connecting to wikis - https://phabricator.wikimedia.org/T356164 (10dcaro) I'm running a script to try to trigger the issue and capture a pcap: ` # cat check_wiki_connections.sh #!/bin/bash main() {... [10:42:32] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] several tools get periods of connection refused (104) when connecting to wikis - https://phabricator.wikimedia.org/T356164 (10aborrero) [11:32:58] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) 000_apt.sh lacks PYTHONPATH which should be something like ` PYTHONPATH=/layers/fagiani_apt/apt/usr/lib/python3/dist-packages ` This sort of thing would... [11:43:00] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10dcaro) >>! In T319953#9509109, @tstarling wrote: > 000_apt.sh lacks PYTHONPATH which should be something like > > ` > PYTHONPATH=/layers/fagiani_apt/apt/usr/lib/pyt... [11:45:42] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10dcaro) >>! In T319953#9509136, @dcaro wrote: >>>! In T319953#9509109, @tstarling wrote: >> 000_apt.sh lacks PYTHONPATH which should be something like >> >> ` >> PYT... [11:46:35] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Set PYTHONPATH whenever python packages are installed - https://phabricator.wikimedia.org/T356499 (10dcaro) [11:48:52] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Installed python scripts with a hardcoded shebang to the python binary will not work when installing new pythons - https://phabricator.wikimedia.org/T356500 (10dcaro) [11:57:53] 10Cloud-VPS (Project-requests): Request creation of topic-curator VPS project - https://phabricator.wikimedia.org/T356195 (10So9q) 05Open→03Resolved a:03So9q Big thanks for taking a deeper look into this and for the suggestion to adjust the encoding. I'll close this as resolved for now, while I try diving... [11:58:16] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10dcaro) [11:58:33] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Set PYTHONPATH whenever python packages are installed - https://phabricator.wikimedia.org/T356499 (10dcaro) 05Open→03In progress [12:07:49] 10Cloud-VPS (Project-requests): Request creation of topic-curator VPS project - https://phabricator.wikimedia.org/T356195 (10So9q) >>! In T356195#9500686, @bd808 wrote: > With the changes from https://github.com/dpriskorn/WikidataTopicCurator/commit/d532547c74c4ca156d712299f580bc72e50f645a now in place on your T... [12:11:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:23:27] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Set PYTHONPATH whenever python packages are installed - https://phabricator.wikimedia.org/T356499 (10dcaro) This was not all that's needed, as numy for example will fail to load the compiled extensions :/ ` heroku@8ffd1f604c73:/workspace$ python Python 3.10.... [12:24:06] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] allow calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377 (10tstarling) A number of libraries are installed into subdirectories of /usr/lib/x86_64-linux-gnu , and usually they would be fou... [12:24:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [12:24:58] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) A number of libraries are installed into subdirectories of /usr/lib/x86_64-linux-gnu , and usually they would be found by the ldconfig cache. But this fag... [12:29:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [12:30:31] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Set PYTHONPATH whenever python packages are installed - https://phabricator.wikimedia.org/T356499 (10dcaro) Adding an overextended LD_LIBRARY_PATH allows loading numy: ` heroku@473c113710f3:/workspace$ python Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC... [12:37:44] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10dcaro) [12:37:58] 10Toolforge (Toolforge iteration 04): [apt-buildpack] Set PYTHONPATH whenever python packages are installed - https://phabricator.wikimedia.org/T356499 (10dcaro) 05In progress→03Resolved Rolled out the fix [12:38:38] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10dcaro) @tstarling noticed also :), should be fixed now, can you try? [12:50:06] 10Cloud-VPS (Project-requests): Request creation of topic-curator VPS project - https://phabricator.wikimedia.org/T356195 (10So9q) To see the error in action try this url: https://topic-curator.toolforge.org/results?lang=en&subgraph=&qid=Q7889&limit=&prefix=&affix=&terms=videogame&terms=computer+games&terms=vide... [12:52:38] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) Can't rebuild ` [step-export] 2024-02-02T12:47:50.127313093Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.... [12:55:47] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) Did `toolforge build clean` and trying again. [13:14:32] 10Cloud-Services: toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503 (10tstarling) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to... [13:17:27] 10Toolforge Jobs framework: toolforge jobs logs read timeout error - https://phabricator.wikimedia.org/T356503 (10tstarling) [13:21:53] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10tstarling) 05Open→03Resolved Looks good, thanks @dcaro. I reverted my temporary hacks in the tool. There are some remaining followup tasks, but migration of thi... [14:24:51] 10Toolforge Build Service: Build service: Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016 (10dcaro) [14:29:24] 10Grid-Engine-to-K8s-Migration: Migrate apersonbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319561 (10dcaro) Hi! I was wondering if you are having any issues or blockers migrating the tool that I could help with. [14:55:28] 10Grid-Engine-to-K8s-Migration: Migrate articles-by-lat-lon-without-images from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319573 (10dcaro) Hi! I just sent a sample PR on github that will allow you to run this tool under the toolforge build service. Feel free to ignore it... [14:59:48] 10Grid-Engine-to-K8s-Migration: Migrate as-info-dev from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319574 (10dcaro) Hi @ST47, did the suggestion help? I see that there's a 'range-update' job that still runs on grid that needs migrating, let us know if you have more questions. [15:03:14] 10Grid-Engine-to-K8s-Migration: Migrate botleo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319602 (10dcaro) HI @GPSLeo, did you try again to migrate the tool? Are you finding any issues? [15:05:43] 10Grid-Engine-to-K8s-Migration: Migrate botwikiawk from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319607 (10dcaro) HI @Green_Cardamom, where you able to try this again? Do you need any assistance? [15:06:51] 10Grid-Engine-to-K8s-Migration: Migrate cewbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319622 (10dcaro) @Kanashimi Hi! Were you able to request the quota you needed? Do you need anything else? [15:12:08] 10Grid-Engine-to-K8s-Migration: Migrate chie-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319625 (10dcaro) Hi @Leloiandudu, just a gentle reminder that there's still a webservice running in the grid, let me know if you are having more issues or need some help. [15:16:01] 10wikitech.wikimedia.org, 10Gerrit, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558 (10sbassett) >>! In T307558#9508299, @bd808 wrote: > This task can be made public agai... [15:16:38] 10wikitech.wikimedia.org, 10Gerrit, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558 (10sbassett) [15:16:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:18:50] 10Grid-Engine-to-K8s-Migration: Migrate commons-android-app from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319638 (10dcaro) Hi! I see that most of the tool is migrated, but there's still a stray cron (cron-29, coming from crontab) that needs migrating, no rush, just fyi.... [15:20:59] 10Grid-Engine-to-K8s-Migration: Migrate dow from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319700 (10dcaro) Hi @UkrFace, were you able to migrate your tool? Is there any issues/questions we can help with? [15:27:41] 10Cloud-VPS, 10cloud-services-team, 10Goal: Update designate-sink handlers to catch up with upstream refactors - https://phabricator.wikimedia.org/T356515 (10Andrew) [15:33:34] 10Cloud-VPS, 10cloud-services-team: Recurring designate record leaks - https://phabricator.wikimedia.org/T356516 (10Andrew) [15:39:36] 10Grid-Engine-to-K8s-Migration: Migrate erex-yomi from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319727 (10dcaro) Hi @Asaf, this is still running or the grid, as your code is public and simple, I was able to bundle a PR with a working example on how to migrate it to the ne... [15:56:33] 10Grid-Engine-to-K8s-Migration: Migrate family from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319739 (10dcaro) Hi @Multichill, I have just sent a sample PR of adding the bits necessary to run this tool under the toolforge build service: https://github.com/metaodi/ancestors... [16:05:23] 10Cloud-VPS (Project-requests): Request creation of topic-curator VPS project - https://phabricator.wikimedia.org/T356195 (10JJMC89) 05Resolved→03Invalid [16:08:48] 10Grid-Engine-to-K8s-Migration: Migrate geograph from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319765 (10dcaro) Hi @Multichill, let us know if you are having issues migrating or need some advice, cheers. [16:09:56] 10Toolforge-standards-committee: Adoption request for fpcstats - https://phabricator.wikimedia.org/T338557 (10fnegri) +1, before adding the user we should also follow this part of the [Adoption policy](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy#Adoption): > Prior to granting access... [16:10:27] 10Grid-Engine-to-K8s-Migration: Migrate gerakibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319767 (10dcaro) Hi @geraki, please let us know if you are finding any issues or need some advice/help with the migration. I don't see any activity on the grid anymore though,... [16:17:30] 10Grid-Engine-to-K8s-Migration: Migrate gerakitools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319768 (10dcaro) Hi @geraki, please let us know if you are finding any issues or need some advice/help with the migration. I don't see any activity on the grid for this tool... [16:25:20] 10Cloud-VPS, 10cloud-services-team: Recurring designate record leaks - https://phabricator.wikimedia.org/T356516 (10Andrew) [16:28:44] (03PS1) 10Umherirrender: Upgrade phpunit to 9.6.16 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995262 (https://phabricator.wikimedia.org/T342110) [16:34:15] 10Grid-Engine-to-K8s-Migration: Migrate hazard-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319785 (10dcaro) @Hazard-SJ How is the migration going? I still see many crons running on the grid, is there any bug/issue with the platform or advice we can help you with? [16:36:03] (03CR) 10Daimona Eaytoy: [C: 03+1] Upgrade phpunit to 9.6.16 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995262 (https://phabricator.wikimedia.org/T342110) (owner: 10Umherirrender) [16:36:37] (03CR) 10Daimona Eaytoy: [C: 03+1] "I'm happy with this -- just not sure if we're ready to go ahead given the recent LibUP issues." [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995262 (https://phabricator.wikimedia.org/T342110) (owner: 10Umherirrender) [16:51:28] (03PS3) 10Majavah: phabricator: Offer to set issue tracker URL in toolinfo [labs/striker] - 10https://gerrit.wikimedia.org/r/992146 [16:51:30] (03PS3) 10Majavah: dev(docker): Add repository url custom field [labs/striker] - 10https://gerrit.wikimedia.org/r/992156 [16:51:32] (03PS5) 10Majavah: phabricator: Allow setting source repository project field [labs/striker] - 10https://gerrit.wikimedia.org/r/992157 [16:52:04] (03CR) 10Majavah: phabricator: Offer to set issue tracker URL in toolinfo (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/992146 (owner: 10Majavah) [16:57:53] 10Striker: Striker dev env fails to start with `manage.py runserver: error: unrecognized arguments: --nostatic` - https://phabricator.wikimedia.org/T355522 (10taavi) I have not found any way to not have that flag error out. Are you sure you don't have some outdated dependency stuck somewhere like the Docker cach... [17:00:14] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge API] Investigate ways to present our openapi definitions to users - https://phabricator.wikimedia.org/T354745 (10aborrero) [17:03:55] 10Toolforge (Toolforge iteration 04), 10User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523 (10aborrero) [17:07:35] 10Toolforge (Toolforge iteration 04), 10User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523 (10aborrero) a:03aborrero [17:08:12] 10Toolforge (Toolforge iteration 04), 10User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523 (10aborrero) p:05Triage→03Medium [17:11:02] 10Toolforge (Toolforge iteration 04), 10Toolforge Jobs framework, 10User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523 (10taavi) [17:26:16] 10Toolforge: Very long response headers cause a 502 response from the Toolforge front proxy - https://phabricator.wikimedia.org/T356525 (10bd808) [17:27:12] 10Toolforge: Very long response headers cause a 502 response from the Toolforge front proxy - https://phabricator.wikimedia.org/T356525 (10bd808) [18:13:03] 10Cloud-VPS, 10cloud-services-team: Update designate-sink handlers to catch up with upstream refactors - https://phabricator.wikimedia.org/T356527 (10Andrew) [18:16:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:18:29] (03CR) 10Jforrester: [C: 03+2] "Can't hurt!" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995262 (https://phabricator.wikimedia.org/T342110) (owner: 10Umherirrender) [18:19:06] (03Merged) 10jenkins-bot: Upgrade phpunit to 9.6.16 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995262 (https://phabricator.wikimedia.org/T342110) (owner: 10Umherirrender) [18:30:36] (03PS1) 10Jforrester: Bump grunt-eslint to 24.3.0 (low-priority weight) [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995278 [18:30:38] (03PS1) 10Jforrester: Bump qunit to 2.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995279 [18:30:40] (03PS1) 10Jforrester: Bump stylelint-config-wikimedia to 0.16.1 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995280 [18:30:42] (03PS1) 10Jforrester: Bump svgo to 3.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995281 [18:30:44] (03PS1) 10Jforrester: Bump @wikimedia/mw-node-qunit to 7.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995282 [19:09:32] 10Cloud-VPS, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 10Goal, 10User-Marostegui: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291 (10Andrew) > All existing guest VMs need a config upgrade to find the right container images This is harder than I expected. Many c... [19:11:59] (03CR) 10Jforrester: [C: 03+2] Bump @wikimedia/mw-node-qunit to 7.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995282 (owner: 10Jforrester) [19:12:02] (03CR) 10Jforrester: [C: 03+2] Bump svgo to 3.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995281 (owner: 10Jforrester) [19:12:04] (03CR) 10Jforrester: [C: 03+2] Bump stylelint-config-wikimedia to 0.16.1 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995280 (owner: 10Jforrester) [19:12:06] (03CR) 10Jforrester: [C: 03+2] Bump qunit to 2.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995279 (owner: 10Jforrester) [19:12:10] (03CR) 10Jforrester: [C: 03+2] Bump grunt-eslint to 24.3.0 (low-priority weight) [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995278 (owner: 10Jforrester) [19:12:45] (03Merged) 10jenkins-bot: Bump grunt-eslint to 24.3.0 (low-priority weight) [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995278 (owner: 10Jforrester) [19:12:47] (03Merged) 10jenkins-bot: Bump qunit to 2.20.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995279 (owner: 10Jforrester) [19:12:49] (03Merged) 10jenkins-bot: Bump stylelint-config-wikimedia to 0.16.1 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995280 (owner: 10Jforrester) [19:12:51] (03Merged) 10jenkins-bot: Bump svgo to 3.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995281 (owner: 10Jforrester) [19:12:57] (03Merged) 10jenkins-bot: Bump @wikimedia/mw-node-qunit to 7.2.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/995282 (owner: 10Jforrester) [20:19:52] 10Grid-Engine-to-K8s-Migration: Migrate robokobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320014 (10Thibaut120094) Thank you, I set up the OAuth 1.0a environment variables and tried to run this command as a test: ` toolforge jobs run --image tool-pywikibot/pywikibot... [20:57:31] 10Striker: Striker dev env fails to start with `manage.py runserver: error: unrecognized arguments: --nostatic` - https://phabricator.wikimedia.org/T355522 (10bd808) >>! In T355522#9509980, @taavi wrote: > I have not found any way to not have that flag error out. Are you sure you don't have some outdated depende... [21:04:29] 10Toolforge: Very long response headers cause a 502 response from the Toolforge front proxy - https://phabricator.wikimedia.org/T356525 (10bd808) [21:16:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [22:15:12] 10Toolforge-standards-committee: Adoption request for fpcstats - https://phabricator.wikimedia.org/T338557 (10bd808) >>! In T338557#9509786, @fnegri wrote: > +1, before adding the user we should also follow this part of the [Adoption policy](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_polic... [22:15:32] 10Toolforge-standards-committee: Adoption request for fpcstats - https://phabricator.wikimedia.org/T338557 (10bd808) [22:20:20] 10Toolforge-standards-committee, 10User-bd808: Adoption request for fpcstats - https://phabricator.wikimedia.org/T338557 (10bd808) 05Open→03Resolved a:03bd808 @mdaniels5757 I have added you as a maintainer following the purge of potentially sensitive information documented in T338557#9510659. Thanks for... [22:30:12] 10Cloud-VPS, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 10Goal, 10User-Marostegui: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291 (10Andrew) [23:04:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [23:08:26] 10Cloud-VPS, 10cloud-services-team, 10User-aborrero: [openstack] cloudservices + Designate are using different source addresses for local vs. remote updates - https://phabricator.wikimedia.org/T350995 (10Andrew) [23:09:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [23:12:59] 10Cloud-VPS, 10cloud-services-team, 10User-aborrero: [openstack] cloudservices + Designate are using different source addresses for local vs. remote updates - https://phabricator.wikimedia.org/T350995 (10Andrew) This is silly, but I think the solution to this is moving Designate services onto cloudcontrol no... [23:28:16] 10Openstack-Magnum, 10cloud-services-team: Hide fedora images from human Horizon users - https://phabricator.wikimedia.org/T356547 (10Andrew) [23:48:31] 10Openstack-Magnum, 10cloud-services-team: Hide fedora images from human Horizon users - https://phabricator.wikimedia.org/T356547 (10bd808) I'm not sure if I can see some extras because of my user's admin rights, but this is the list I see currently when using the create instance wizard from the reading-web-s...