[00:01:22] (03update) 10sopzy: T407551-proof-of-concept-add-author-form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138 [00:05:21] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11296694 (10Sopzy) Hi @Pepe_piton @Nat_WDU Kindly review my attempt at this task: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138 Thanks! {F66776715} [00:10:19] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296696 (10DamianZaremba) The file is readable when mounted under a different path ` root@jobs-api-5cbf4675c8-fxl56:/app# ls -lra /etc/nsswitch.conf -rw-r--r-- 1 root root 530 Oct 7 07:55 /etc/nssw... [00:18:47] Hi [00:19:05] Im hot [00:19:37] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296706 (10DamianZaremba) [00:53:49] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296733 (10DamianZaremba) Created a new vm... jobs-api was running after `[k8s : Deploy toolforge-deploy components into local k8s]` ` damian@lima-kilo:/Users/damian/Projects/wikimedia/lima-kilo$ ku... [00:55:53] (03reopen) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [00:56:33] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296734 (10DamianZaremba) ` root@lima-kilo:~# journalctl | grep nsswitch.conf Oct 22 02:21:16 lima-kilo ansible-ansible.legacy.stat[8324]: Invoked with path=/etc/nsswitch.conf follow=False get_chec... [01:00:30] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11296745 (10Andrew) With preseed-test I get different but also bad behavior. Grub works, but the kernel won't boot: ` Loading Linux 6.12.43+deb13-amd64 ... Loading initial ramdisk ...... [01:00:46] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [01:05:45] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11296749 (10Andrew) I really need a second config B (or at least 4-drive sw raid) prod server to test this on. [01:15:11] (03update) 10sopzy: T407551-proof-of-concept-add-author-form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138 [01:15:41] (03open) 10nurahwakili: T407558- Proof of concept add work form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/139 [01:18:05] (03update) 10sopzy: T407551-proof-of-concept-add-author-form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138 [01:18:08] 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11296756 (10Nurah_Wakili) Hello @Pepe_piton, @Nat_WDU I implemented this task. Kindly review the MR: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/139. Thanks. [01:31:32] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [01:34:26] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30041 bytes in 4.462 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [01:36:24] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296769 (10DamianZaremba) Best guess this is being done by ` damian@lima-lima-kilo:/Users/damian/Projects/wikimedia/lima-kilo$ systemctl status sssd-autofs ○ sssd-autofs.service - SSSD AutoFS Servic... [01:37:43] (03open) 10raymond-ndibe: [helm image publish]: compare harbor_auth to null, not "" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/76 (https://phabricator.wikimedia.org/T394595) [01:40:59] (03update) 10raymond-ndibe: [helm image publish]: compare harbor_auth to null, not "" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/76 (https://phabricator.wikimedia.org/T394595) [01:41:00] (03approved) 10raymond-ndibe: [helm image publish]: compare harbor_auth to null, not "" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/76 (https://phabricator.wikimedia.org/T394595) [01:41:09] (03merge) 10raymond-ndibe: [helm image publish]: compare harbor_auth to null, not "" [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/76 (https://phabricator.wikimedia.org/T394595) [01:42:50] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [01:43:35] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [01:44:29] (03open) 10damian: random change [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/144 [01:47:41] (03close) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [01:48:21] (03close) 10damian: random change [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/144 [01:54:59] (03update) 10damian: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) [02:42:01] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] Queue builds when the build queue is full - https://phabricator.wikimedia.org/T402568#11296796 (10Raymond_Ndibe) 05In progress→03Stalled [03:23:34] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [03:24:24] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30040 bytes in 0.258 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:01:34] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:03:32] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 8.941 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:11:28] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [07:31:27] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11297227 (10fgiunchedi) FWIW yesterday while testing the preseed-test fix for many drives (https://gitlab.wikimedia.org/repos/sre/preseed-test/-/merge_requests/6) I was able to install... [07:36:13] (03approved) 10dcaro: toolforge components - add logs-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/286 (https://phabricator.wikimedia.org/T407930) (owner: 10damian) [07:36:17] (03merge) 10dcaro: toolforge components - add logs-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/286 (https://phabricator.wikimedia.org/T407930) (owner: 10damian) [07:36:26] (03update) 10dcaro: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 [07:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:51:16] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node (T405478) [07:51:23] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [08:34:15] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11297418 (10dcaro) I have rebuilt my env many times last week, and I just rebuilt it twice without having this issue, I'm rebuilding it again to recheck. It did happen once that I saw the `automount... [08:36:36] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [08:38:34] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30037 bytes in 6.439 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [08:59:38] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [09:01:05] (03update) 10dcaro: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 [09:01:30] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30036 bytes in 0.374 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [09:06:19] 06cloud-services-team, 10Toolforge: [api-gateway] openapi.json broken - https://phabricator.wikimedia.org/T407930#11297459 (10dcaro) 05Open→03Resolved a:03dcaro This is fixed now yep, thanks for the fix. ` direct-api/direct-api-smoke-tests.bats ✓ get openapi definition [251] ✓ get envvars openapi d... [09:07:14] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11297462 (10Aklapper) "There is `brick` in `Phabrickator` for a reason!" [09:08:55] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11297469 (10dcaro) The rebuild worked, and everything passes the functional tests, noting though that the file has the automount line: ` dcaro@toolslocal$ cat /etc/nsswitch.conf # this is the nsswit... [09:10:17] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11297471 (10dcaro) The lima-kilo I used last was this: ` dcaro@acme$ git log -1 commit 501eb3018060a4b6243abf842f2dabaa54c6d33f (HEAD -> main, origin/main, origin/HEAD) Author: Damian Zaremba 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297473 (10dcaro) [09:18:38] (03approved) 10dcaro: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 [09:18:46] (03merge) 10dcaro: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 [09:19:29] (03update) 10dcaro: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) (owner: 10damian) [09:23:26] (03approved) 10dcaro: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) (owner: 10taavi) [09:25:10] (03merge) 10taavi: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) [09:25:12] (03update) 10taavi: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) [09:37:52] 10Toolforge (Toolforge iteration 24): [api-gateway] can't retrieve openapi.json - https://phabricator.wikimedia.org/T407957 (10dcaro) 03NEW [09:38:57] (03approved) 10dcaro: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) (owner: 10taavi) [09:46:28] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:49:33] (03open) 10olatundeawo: Pagination t399974 [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/140 [09:50:53] (03open) 10aradhya: Add link to SPARQL query on works list page (T397554) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/141 [09:52:42] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11297602 (10Olatundeawo) Hey, @Pepe_piton @Nat_WDU I have resolved the issue, here is my MR: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/140 [09:54:24] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11297604 (10Aradhya05) Hi @Pepe_piton, @Nat_WDU, I have completed this task and submitted the merge request: https://gitlab.wikimedia.org/toolforge-repos/pauli... [10:27:39] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [10:52:19] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [10:54:02] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297750 (10DamianZaremba) In generally it builds into a usable state, I rebuilt with that same ref last night and it was fine, but the ref prior was find in... [10:54:32] 10Toolforge (Toolforge iteration 24): [api-gateway] can't retrieve openapi.json - https://phabricator.wikimedia.org/T407957#11297751 (10dcaro) Note that using https://api-docs.toolforge.org/docs (that does the same query internally) works ok [10:57:06] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297754 (10dcaro) What if we add the automount bit to lima-kilo so ansible does not disagree with whatever adds it? Does that give a stabler deployment? (I'l... [11:02:17] (03update) 10damian: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) [11:02:20] (03update) 10damian: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) [11:04:27] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [11:08:04] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297796 (10DamianZaremba) > What if we add the automount bit to lima-kilo so ansible does not disagree with whatever adds it? That might be a solution. > I'... [11:15:54] 06cloud-services-team, 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11297808 (10akosiaris) > In T407296#11274680, @Andrew wrote: > "After building a pilot bare-metal toolforge replacement, how hard was it, and does it seem like something that would be easy to maintain, o... [11:35:41] (03open) 10bahago: Feature t407385/create new generic page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/142 [11:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:40:00] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11297899 (10Raboyusuf2024) Hi @Pepe_piton @Nat_WDU, I'd appreciate your feedback on this: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/142 Add generic "new-page" template for practice This patch creates... [11:45:25] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [11:46:20] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [11:52:04] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [12:01:20] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [12:02:41] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11297959 (10Miiswom) Hi everyone, @Pepe_piton and @Nat_WDU : Please find attached my [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 | Merge Request ]] for this task, I am still in progres... [12:06:04] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297964 (10DamianZaremba) ` sudo -i apt-get update apt install -y auditd systemctl start auditd auditctl -w /etc/ -p wa -k etc_watch ` Managed to replicate... [12:08:52] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297966 (10DamianZaremba) I'll make a MR after lunch to change the ordering of the sssd setup, which should at least sort that flip-flopping out. Unless ther... [12:11:31] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11297983 (10Miiswom) @Sopzy: Yes, it has been resolved! It happened when I was working on the task. Most of the times due to me forgetting a 's' at the end of some variables. [12:12:19] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297987 (10DamianZaremba) So the current thinking TL;DR: - SSSD config is written - SSSD is installed, changing the config - kind gets setup, bind-mounting... [12:12:58] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11297988 (10DamianZaremba) @dcaro could you try running ansible (`./start-devenv.sh` with `n` or the script inside the container) for at least `base_system` a... [12:14:38] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [12:36:03] (03merge) 10taavi: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) [12:38:37] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298105 (10dcaro) >>! In T407931#11297988, @DamianZaremba wrote: > @dcaro could you try running ansible (`./start-devenv.sh` with `n` or the script inside th... [12:46:03] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298121 (10dcaro) Ansible did change the config file: ` TASK [basic_system : Install nsswitch.conf to use sssd] *********************************************... [12:55:24] (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/28 [12:55:29] (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/28 [12:55:49] (03approved) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/28 [12:55:51] (03update) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/28 [12:57:45] (03open) 10taavi: Remove old front proxy instances [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98 (https://phabricator.wikimedia.org/T283948) [12:57:49] (03update) 10taavi: Remove old front proxy instances [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98 (https://phabricator.wikimedia.org/T283948) [12:59:29] (03merge) 10dcaro: readme: standardize deployment docs [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/28 [13:02:04] 10Toolforge (Toolforge iteration 24): [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11298147 (10dcaro) [13:03:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.68-20251022125940-531b41e4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1005 (https://phabricator.wikimedia.org/T407477) [13:04:49] (03open) 10dcaro: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 [13:05:54] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298155 (10DamianZaremba) Great! That took a bit of sluicing but hopefully this sorts it out long term. [13:06:45] (03update) 10dcaro: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 [13:07:54] (03update) 10dcaro: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 [13:11:25] (03open) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [13:11:29] (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [13:14:38] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11298175 (10Marostegui) So my proposal would be to do this considering 8 existing hosts + 4 expansion hosts already racked.... [13:24:38] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install clouddb102[2-5] - https://phabricator.wikimedia.org/T393733#11298201 (10Marostegui) I've expanded their logical volume to use most of the disk as we normally do ` root@clouddb1022:~# pvs PV VG Fmt Attr PS... [13:31:35] 06cloud-services-team, 10Cloud-VPS: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#11298228 (10Andrew) I just encountered a variation on this: new user dpogorzelski was in the bastion project with the proper role, but 'project-bastion' didn't... [13:36:04] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [13:40:18] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [13:41:32] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [tools,nfs,infra] Address tools NFS getting stuck with processes in D state - https://phabricator.wikimedia.org/T404584#11298261 (10fgiunchedi) 05In progress→03Resolved I'm optimistically and cautiously resolving this task sinc... [13:45:42] 10Toolforge (Toolforge iteration 24): [api-gateway] can't retrieve openapi.json - https://phabricator.wikimedia.org/T407957#11298290 (10dcaro) Did a bit of digging, found nothing yet. On the haproxy, going directly to the control nodes works well: ` root@tools-k8s-haproxy-8:~# curl --http2 -v -k -H "Host: api.... [13:53:38] 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-d... - https://phabricator.wikimedia.org/T194332#11298320 [13:55:00] (03open) 10damian: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 [13:55:46] 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-d... - https://phabricator.wikimedia.org/T194332#11298324 [13:56:02] 06cloud-services-team, 10Toolforge: [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298327 (10DamianZaremba) Rebuilding with https://phabricator.wikimedia.org/T407931 [13:58:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298336 (10dcaro) [13:58:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298339 (10dcaro) p:05Triage→03Medium [13:58:38] 10Toolforge (Toolforge iteration 24): [api-gateway] can't retrieve openapi.json - https://phabricator.wikimedia.org/T407957#11298340 (10dcaro) p:05Triage→03Medium [13:58:44] 10Toolforge (Toolforge iteration 24): [infra,logging] prometheus failing to fetch the metrics endpoint - https://phabricator.wikimedia.org/T407852#11298341 (10dcaro) p:05Triage→03High [14:03:50] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11298377 (10Andrew) In any case, it's clear that preseed-test isn't going to help with the actual issue on 2010-dev :/ [14:06:39] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#11298386 (10fgiunchedi) As part of the task above `testlabs` `toolsbeta` and `tools` project have had their NFS servers migrated to Trixie and the dual-... [14:08:29] (03approved) 10dcaro: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496) (owner: 10raymond-ndibe) [14:08:41] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#11298398 (10fgiunchedi) [14:12:04] 06cloud-services-team, 10Toolforge, 07Epic: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice - https://phabricator.wikimedia.org/T388090#11298426 (10dcaro) 05Open→03Invalid I'll close this for now, and start a design document on the parent to discuss and define... [14:12:46] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_deploy_mr.py doesn't handle missing pipelines - https://phabricator.wikimedia.org/T407982 (10DamianZaremba) 03NEW [14:14:52] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_deploy_mr.py fails to find pipeline jobs on fork - https://phabricator.wikimedia.org/T407983 (10DamianZaremba) 03NEW [14:15:31] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11298474 (10dcaro) [14:17:45] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_deploy_mr.py fails to find pipeline jobs on fork - https://phabricator.wikimedia.org/T407983#11298481 (10DamianZaremba) Fork URL: https://gitlab.wikimedia.org/api/v4/projects/3472/pipelines/142600/jobs Broken URL: https://gitlab.wikimedia.org/api/v4/pro... [14:19:44] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [14:23:39] (03open) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [14:30:24] (03approved) 10dcaro: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) (owner: 10damian) [14:32:29] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [14:32:47] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982 https://phabricator.wikimedia.org/T407983) [14:33:10] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [14:34:11] (03open) 10damian: deploy_mr - Use project/job as returned by the API [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/289 (https://phabricator.wikimedia.org/T407983) [14:40:04] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982 https://phabricator.wikimedia.org/T407983) [14:41:27] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [14:47:24] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [lima-kilo] toolforge_deploy_mr.py fails to find pipeline jobs on fork - https://phabricator.wikimedia.org/T407983#11298629 (10DamianZaremba) Deployed https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 with the script and t... [14:49:11] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11298634 (10DamianZaremba) Instance built successfully, second run showed no (related) changes (well sssd permissions, but that's not... [14:56:09] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [14:56:27] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [14:59:57] (03update) 10dcaro: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [15:00:16] (03update) 10dcaro: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [15:00:45] (03update) 10dcaro: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [15:04:54] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:05:49] (03update) 10dcaro: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [15:06:10] (03update) 10dcaro: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [15:06:25] (03approved) 10dcaro: deploy_mr - Use project/job as returned by the API [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/289 (https://phabricator.wikimedia.org/T407983) (owner: 10damian) [15:06:47] (03update) 10dcaro: deploy_mr - Use project/job as returned by the API [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/289 (https://phabricator.wikimedia.org/T407983) (owner: 10damian) [15:07:00] (03merge) 10dcaro: deploy_mr - Use project/job as returned by the API [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/289 (https://phabricator.wikimedia.org/T407983) (owner: 10damian) [15:07:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:13:04] 10Tool-wmf-openapi-linter, 03[MWI] FY2025-26 Q2, 06MW-Interfaces-Team (MWI-Sprint-21 (2025-10-21 to 2025-11-04)), 07OKR-Work: Create a tool to lint OpenAPI spec based on Wikimedia ruleset - https://phabricator.wikimedia.org/T406879#11298766 (10OWresch-WMF) [15:13:12] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [15:13:14] (03update) 10damian: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 [15:13:54] (03update) 10damian: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 (https://phabricator.wikimedia.org/T407931) [15:14:08] (03update) 10taavi: Remove old front proxy instances [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98 (https://phabricator.wikimedia.org/T283948) [15:16:03] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:19:53] 06cloud-services-team, 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11298805 (10fnegri) >>! In T407296#11277258, @dcaro wrote: > I agree with @Andrew, I think that the experiment is not yet well defined. And I agree also with the questions he raises. Maybe we can phrase... [15:24:21] (03approved) 10dcaro: ingress-admission: bump to 0.0.68-20251022125940-531b41e4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1005 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:24:26] (03merge) 10dcaro: ingress-admission: bump to 0.0.68-20251022125940-531b41e4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1005 (https://phabricator.wikimedia.org/T407477) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:40:12] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#11298937 (10taavi) [15:45:26] 06cloud-services-team, 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11298965 (10akosiaris) > Does "we" include every SRE in the SRE department? Or is that only the "tools-infra" team? Or is it "tools-platform" + "tools-infra" teams? **If the PoC pans out**, my hope is t... [15:51:36] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T405478) [15:51:44] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [15:53:13] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:56:02] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry [15:56:02] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3 [15:56:23] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) [15:56:40] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:56:53] (03approved) 10dcaro: Remove old front proxy instances [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98 (https://phabricator.wikimedia.org/T283948) (owner: 10taavi) [15:59:03] 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade ingress-nginx to v1.12.1+ - https://phabricator.wikimedia.org/T383516#11299032 (10taavi) p:05Low→03Medium a:03taavi [15:59:32] (03update) 10dcaro: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 (https://phabricator.wikimedia.org/T407931) (owner: 10damian) [16:00:12] (03update) 10dcaro: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 (https://phabricator.wikimedia.org/T407931) (owner: 10damian) [16:00:34] (03approved) 10dcaro: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 (https://phabricator.wikimedia.org/T407931) (owner: 10damian) [16:00:42] (03merge) 10dcaro: Configure nsswitch.conf after sssd is running [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/287 (https://phabricator.wikimedia.org/T407931) (owner: 10damian) [16:02:02] (03update) 10damian: Fail with friendly errors on missing pipelines [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/288 (https://phabricator.wikimedia.org/T407982) [16:07:03] 06cloud-services-team, 10Horizon, 10Striker, 13Patch-For-Review: Do not page via Icinga for individual cloudweb hosts being down - https://phabricator.wikimedia.org/T407208#11299100 (10taavi) 05Open→03Resolved [16:14:38] 10Toolforge, 06tools-infrastructure-team: Pull toolviews data from Kubernetes HAProxy or ingress-nginx instead of the front nginx - https://phabricator.wikimedia.org/T284558#11299126 (10taavi) 05Open→03Resolved [16:15:41] 06cloud-services-team, 10Toolforge, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T361237#11299131 (10taavi) [16:28:00] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11299190 (10DidiCoronel) >>! In T405621#11287742, @Oluwatumininu.m wrote: >>>! In T405621#11284544, @DidiCoronel wrote: >>>>! In T405621#11261016, @Oluwatumini... [16:29:06] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:30:18] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:31:39] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:34:18] (03open) 10damian: Retry deployments for 5min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 [16:35:15] (03update) 10damian: Retry deployments for 5min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 [16:38:19] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:39:04] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11299223 (10DidiCoronel) >>! In T405621#11283851, @Afanyulionel wrote: > Hello @Pepe_piton , @Nat_WDU Here's the result of my version of the task: [[ https://g... [16:39:50] (03open) 10taavi: ingress-nginx: Update to chart-4.13.3 on lima/toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1007 (https://phabricator.wikimedia.org/T383516) [16:39:52] (03update) 10taavi: ingress-nginx: Update to chart-4.13.3 on lima/toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1007 (https://phabricator.wikimedia.org/T383516) [16:39:59] 06cloud-services-team, 10Toolforge: [lima-kilo] toolforge_deploy_mr.py fails to find pipeline jobs on fork - https://phabricator.wikimedia.org/T407983#11299224 (10DamianZaremba) 05Open→03Resolved a:03DamianZaremba [16:40:12] (03update) 10taavi: ingress-nginx: Update to chart-4.13.3 on lima/toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1007 (https://phabricator.wikimedia.org/T383516) [16:40:25] (03update) 10oluwatumininu: Improve country dropdown for the "country of origin" filter in the advanced search for works. [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/14 [16:40:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [lima-kilo] jobs-api fails to start sometimes due to nsswitch.conf mount - https://phabricator.wikimedia.org/T407931#11299228 (10DamianZaremba) 05Open→03Resolved a:03DamianZaremba [16:42:28] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11299235 (10Oluwatumininu.m) >>! In T405621#11299190, @DidiCoronel wrote: >>>! In T405621#11287742, @Oluwatumininu.m wrote: >>>>! In T405621#11284544, @DidiCor... [16:48:05] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:48:46] 06cloud-services-team, 10Toolforge: functional tests are optimistic about retries/timeouts - https://phabricator.wikimedia.org/T408002 (10DamianZaremba) 03NEW [16:49:09] (03update) 10damian: Retry deployments for 5min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [16:49:48] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [16:52:25] 06cloud-services-team, 10Toolforge: functional tests are optimistic about retries/timeouts - https://phabricator.wikimedia.org/T408002#11299263 (10dcaro) cc. @fnegri [16:54:20] (03update) 10damian: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [16:54:29] (03update) 10damian: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [16:55:00] (03update) 10damian: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [16:56:37] 06cloud-services-team, 10Toolforge: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#11299273 (10fnegri) > they are dependent on the PoC succeeding. What is the definition of "succeeding"? :) I'm not sure if everybody has the same definition in their mind. In other words: what is the "D... [16:57:09] (03update) 10damian: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [16:58:57] (03update) 10damian: Draft: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [17:05:42] 06cloud-services-team, 10Toolforge: functional tests are optimistic about retries/timeouts - https://phabricator.wikimedia.org/T408002#11299332 (10DamianZaremba) Somewhat related but for example `config generate with some existing jobs works, is loadable and creates the same job (slow)` relies on another test... [17:06:19] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [17:07:04] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#11299337 (10Andrew) Having reviewed and reconsidered this, I now vote for a modified version of option 1: "Track all project definitions... [17:09:15] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#11299342 (10Andrew) I'm thinking this over again, and I'm fairly compelled by the fact that resources managed by tofu are in source control with a history. That said, I know that @fg... [17:13:57] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [17:15:05] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [17:17:37] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11299384 (10DidiCoronel) @shreya-bhagat, @Aradhya05, @Dipanshu1223 code looks ok and I was able to see the new dropdown type-to-search behavior on your PRs. [17:18:06] 10Tool-paulina: Proof of concept for adding new statements - https://phabricator.wikimedia.org/T406989#11299388 (10Dipanshu1223) hi @Pepe_piton i made some changes:https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 changes made: - add style dropdown - add Ethnicity in selection Please... [17:31:04] (03approved) 10dcaro: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) (owner: 10damian) [17:36:23] (03open) 10shr0x-ya: Resolve T407558 "/add work with autocomplete" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/143 [17:44:54] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [17:57:29] 10Tool-sal: Migrate sal project from github to gitlab.wikimedia.org + phabricator.wikimedia.org - https://phabricator.wikimedia.org/T408007 (10bd808) 03NEW [18:25:46] (03update) 10shr0x-ya: Resolve T407558 "/add work with autocomplete" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/143 [18:27:11] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:35:36] (03update) 10shr0x-ya: Add Work Entity with Auto-Complete Feature [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/143 [18:37:40] 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11299856 (10shreya-bhagat) @Pepe_piton @Nat_WDU apologies for the delay due to holiday weekend in India. Have completed the task, ready for review! https://gitlab.wikimedia.org/toolforge-repos/paulina/... [18:38:48] (03update) 10shr0x-ya: [WIP] Draft: Add author proof of concept [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/123 [18:52:12] (03open) 10lebogang: docs: add api-call-examples with tested Wikibase REST API examples [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/144 [18:57:12] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11299903 (10Dipanshu1223) >>! In T405621#11299384, @DidiCoronel wrote: > @shreya-bhagat, @Aradhya05, @Dipanshu1223 code looks ok and I was able to see the new... [19:05:13] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [19:10:32] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [19:11:58] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [19:15:05] (03CR) 10Andrew Bogott: "shall we merge this?" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [19:15:33] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11299968 (10Lebogang) ‎@Nat_WDU @Pepe_piton Completed the microtask and submitted a merge request adding api-call-examples.md, which includes tested examples for creating and... [20:02:31] (03open) 10dipanshu1223: T399974:"Pagination feature" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/145 [20:03:32] (03update) 10dipanshu1223: T399974:"Pagination feature" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/145 [20:03:59] (03update) 10dipanshu1223: T399974:"Pagination feature" [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/145 [20:05:23] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11300160 (10Dipanshu1223) Good evening @Pepe_piton and @Nat_WDU i made a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/145 Changes made:- - Added JavaScript to implement clie... [20:29:04] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11300199 (10Andrew) I don't know what a healthy grub run looks like, but I'm not loving this: ` Oct 22 19:46:10 grub-installer: info: Running chroot /target grub-install --for ce "/d... [20:45:16] 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604#11300256 (10Pepe_piton) @Aradhya05 Nice work! It works really well. I also liked that you used code like `request.args.get('nationality', country_qid)` so that, when the URL... [21:01:46] (03open) 10raymond-ndibe: [storage.kubernetes] remove _remove_none_values method [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/145 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [21:02:17] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11300347 (10Andrew) Here is the equivalent for bookworm (which works): ` Oct 22 20:45:11 grub-installer: info: Running chroot /target grub-install --for ce "/dev/sdd" Oct 22 20:45:11... [21:02:43] (03update) 10raymond-ndibe: [storage.kubernetes] remove _remove_none_values method [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/145 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [21:28:22] 06cloud-services-team, 10Toolforge: [components-api] tries to decode server errors as json - https://phabricator.wikimedia.org/T408028 (10DamianZaremba) 03NEW [21:29:09] 06cloud-services-team, 10Toolforge: [components-api] tries to decode server errors as json - https://phabricator.wikimedia.org/T408028#11300442 (10DamianZaremba) /cc @dcaro from discussion on https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139#note_170947 [21:34:48] 06cloud-services-team, 10Toolforge: [components-api] tries to decode server errors as json - https://phabricator.wikimedia.org/T408028#11300481 (10DamianZaremba) I think something to consider is when we leak internal details e.g. "got exception" vs wrapping in human messages e.g. warnings (jobs-api). If the `... [21:38:52] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300495 (10Raymond_Ndibe) **Job version upgrade email draft:** ` [Cloud-announce] Old... [21:39:30] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300497 (10Raymond_Ndibe) **Affected tools:** ` actrial adamant admin ahechtbot air75... [21:44:10] 06cloud-services-team, 10Toolforge: toolforge jobs dump prints "WARNING: unexpected key 'job_type' in job, omitting it from dump" - https://phabricator.wikimedia.org/T408033 (10taavi) 03NEW [21:45:13] 06cloud-services-team, 10Toolforge: toolforge jobs dump includes booleans as strings - https://phabricator.wikimedia.org/T408034 (10taavi) 03NEW [21:47:15] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300577 (10taavi) >>! In T359649#11300495, @Raymond_Ndibe wrote: > **Job version upgr... [21:54:38] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300621 (10Raymond_Ndibe) **Upgrade notification to individual maintainer draft** ` U... [21:57:23] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300633 (10Raymond_Ndibe) >>! In T359649#11300595, @Raymond_Ndibe wrote: >>>! In T359... [22:00:41] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300639 (10taavi) The original message draft talked about the "v2 job spec", which is... [22:19:08] 06cloud-services-team, 10Toolforge: toolforge jobs dump prints "WARNING: unexpected key 'job_type' in job, omitting it from dump" - https://phabricator.wikimedia.org/T408033#11300709 (10DamianZaremba) Related https://phabricator.wikimedia.org/T405786 [22:37:30] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11300754 (10Raymond_Ndibe) >>! In T359649#11300639, @taavi wrote: > The original messa... [22:42:13] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [22:52:28] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11300811 (10DidiCoronel) >>! In T399974#11263576, @shreya-bhagat wrote: > @Nat_WDU @Pepe_piton kindly review! > > Changes made: > > 1. Creation of Previous and Next button. Both on Author and Works Results page. On the first page, "... [23:01:09] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11300867 (10DidiCoronel) >>! In T399974#11263752, @System625 wrote: > Hello @Pepe_piton > > I've completed this task and created a merge request: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/30 > > The impl... [23:14:23] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11300929 (10Pepe_piton) @Oluwatumininu.m @Dipanshu1223 @Nurah_Wakili Perfect! @Chumzine Very good! And very nice that you added the link on the site menu. One detail: if you set the variable `wikidata_link` with Jinja directly o... [23:17:30] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [23:56:25] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127