[01:54:28] (03update) 10raymond-ndibe: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [03:13:12] (03approved) 10raymond-ndibe: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 (owner: 10dcaro) [03:13:14] (03update) 10raymond-ndibe: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 (owner: 10dcaro) [03:14:51] (03update) 10raymond-ndibe: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 (owner: 10dcaro) [03:14:53] (03approved) 10raymond-ndibe: README: update with deploy and local deploy instructions [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/4 (owner: 10dcaro) [03:22:53] 10Tool-wsindex, 10Wikisource Reader App: Add an option to sort list of books alphabetically - https://phabricator.wikimedia.org/T407311#11305122 (10Bodhisattwa) >>! In T407311#11301684, @System625 wrote: > Hello @Saiphani02 , I've created a PR concerning the logic update: https://codeberg.org/ph4ni/wsindex/pul... [03:24:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:29:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:38:51] 10Tool-wsindex, 10Wikisource Reader App: Add the image of authors to generate thumbnails for the books in the list of books section - https://phabricator.wikimedia.org/T407639#11305127 (10Bodhisattwa) 05Open→03Resolved @Saiphani02 has merged the PR. Closing the issue. @Muguro, Thanks for the contribution. [03:39:56] 10Tool-wsindex, 10Developer-Outreach, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11305130 (10Bodhisattwa) [03:46:13] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11305131 (10Afanyulionel) >>! In T407385#11304937, @Pepe_piton wrote: > @System625 Perfect! And thanks for detecting and coding a fix for the JavaScript errors! > > @shreya-bhagat @Blegodwin @Atlas0007 @Raboyusuf2024 Excellent!... [03:48:12] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [04:03:21] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [04:17:07] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [04:17:51] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [04:23:57] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30036 bytes in 0.846 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [04:35:14] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [04:46:56] 10Tool-wsindex, 10Developer-Outreach, 10Wikisource Reader App, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11305160 (10Bodhisattwa) [06:09:07] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:10:57] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30032 bytes in 0.982 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:22:29] (03open) 10aradhya: Proof of concept: Add author form (T407551) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/148 [06:22:58] (03update) 10aradhya: Proof of concept: Add author form (T407551) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/148 [06:25:15] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11305216 (10Aradhya05) Hi @Pepe_piton @Nat_WDU, I have completed this task and submitted MR : https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/148 Kindly review Thank you [07:08:19] (03open) 10sopzy: Proof of concept for editing statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/149 [07:10:33] 10Tool-paulina: Proof of concept for editing statements in Paulina - https://phabricator.wikimedia.org/T405412#11305265 (10Sopzy) Hi @Pepe_piton @Nat_WDU I'd appreciate your feedback on this task. Thanks! PR link: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/149 [07:42:00] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11305298 (10System625) >>! In T407385#11304937, @Pepe_piton wrote: > @System625 Perfect! And thanks for detecting and coding a fix for the JavaScript errors! > > @shreya-bhagat @Blegodwin @Atlas0007 @Raboyusuf2024 Excellent! >... [08:03:44] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to an-redacteddb1001 - https://phabricator.wikimedia.org/T407485#11305329 (10JAllemandou) The impact on DE seems small IMO. I'm not an expert but my uinderstanding is that some databases/tabl... [08:04:15] (03update) 10dcaro: get_job_logs: return a message when no logs gotten [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/3 [08:21:05] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (T405478) [08:21:13] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [08:33:02] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1198317 (owner: 10L10n-bot) [08:34:54] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1198315 (owner: 10L10n-bot) [08:43:44] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11305467 (10elukey) I've worked with Filippo this morning, we flipped the host to UEFI via provision cookbook and updated the partman partitions. Same error as T407586#... [08:51:16] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11305486 (10Sopzy) >>! In T407385#11304937, @Pepe_piton wrote: > @System625 Perfect! And thanks for detecting and coding a fix for the JavaScript errors! > > @shreya-bhagat @Blegodwin @Atlas0007 @Raboyusuf2024 Excellent! > > @S... [08:54:03] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11305493 (10elukey) ` grub rescue> set fw_path='(hd3,gpt2)/EFI/debian' prefix='(lvmid/XSJkyY-vEFM-dddW-stYw-b7co-lCY3-xPhPjs/mz1h8d-DR5U-bbX5-Ii1P-CcHG-eaxf-yo6lqG)/boo... [08:56:23] (03open) 10taavi: ingress-nginx: Update to chart-4.13.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1011 (https://phabricator.wikimedia.org/T383516) [08:56:25] (03update) 10taavi: ingress-nginx: Update to chart-4.13.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1011 (https://phabricator.wikimedia.org/T383516) [09:01:05] (03update) 10taavi: ingress-nginx: Update to chart-4.13.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1011 (https://phabricator.wikimedia.org/T383516) [09:52:34] 10Tool-sal, 10GitLab (Project Migration): Migrate sal project from github to gitlab.wikimedia.org + phabricator.wikimedia.org - https://phabricator.wikimedia.org/T408007#11305594 (10A_smart_kitten) [09:55:26] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#11305611 (10fnegri) > deployments (codfw1dev and eqiad1) are currently coupled, so if tofu can't apply in codfw1dev then we also can't change things in eqiad1; I'm pretty sure that ne... [10:05:29] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#11305633 (10fnegri) [10:07:28] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#11305638 (10fnegri) I modified the description to clarify that Option 1 would only apply to projects in the default domain, as I think th... [10:52:44] (03open) 10sopzy: Add field inception to show only when available in the work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/150 [10:54:51] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11305698 (10Sopzy) >>! In T406949#11264837, @Pepe_piton wrote: > @Sopzy Hi! Good questions. > >> Can both publication date and inception date appear on the same work’s page? > > Generally not. But there is another mi... [10:56:16] (03update) 10sopzy: Add field inception to show only when available in the work page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/150 [11:14:19] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node (T405478) [11:14:26] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [12:49:18] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11306051 (10fgiunchedi) Today with @elukey's help I upgraded the idrac and bios on cloudcontrol2010-dev, not that I expected it to make a difference (and it didn't) `... [13:18:41] (03update) 10piracalamina: Add link to SPARQL query on works list page (T397554) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/141 (owner: 10aradhya) [13:23:54] (03PS4) 10Majavah: vps: Add cookbook to delete a project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) [13:25:09] (03CR) 10Majavah: "PS4 is a rebase." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [13:29:11] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11306168 (10Nat_WDU) Hello, @Leobgang! I think that you forgot to add the line that makes the actual request for creating a new author. Other than that. all good! [13:31:03] (03CR) 10FNegri: [C:03+1] vps: Add cookbook to delete a project (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [13:32:57] PROBLEM - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/dumps - 324 bytes in 60.013 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [13:33:42] (03CR) 10Majavah: [C:03+2] vps: Add cookbook to delete a project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [13:33:50] (03update) 10nurahwakili: T407558- Proof of concept add work form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/139 [13:35:02] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11306178 (10fgiunchedi) Ok so I did a test with `grub` at version `2.14~git20250718.0e36779-1` (from experimental) and that changed nothing. I'll be trying with bookwor... [13:35:27] RECOVERY - toolschecker: Make sure enwiki dumps are not empty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 23.718 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [13:37:32] (03Merged) 10jenkins-bot: vps: Add cookbook to delete a project [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1139027 (https://phabricator.wikimedia.org/T391836) (owner: 10Majavah) [13:39:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: wmcs-cookbooks: write a cookbook to delete an openstack project - https://phabricator.wikimedia.org/T391836#11306188 (10taavi) 05Open→03Resolved [13:51:36] (03update) 10piracalamina: Author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/103 (owner: 10afanyulionel) [13:52:47] (03update) 10piracalamina: Proof of concept: Add author form (T407551) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/104 (owner: 10oluwatumininu) [13:56:19] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11306224 (10Nat_WDU) Hello, @Afanyulionel let me ask you a tricky question... how would a Paulina user obtain the Wikidata link to a new author? Let me know if you spot why it would not be ok to ask for this field. [14:05:02] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11306292 (10shreya-bhagat) @Nat_WDU I had submitted my MR link but found some responsiveness issues with it. fixing them and reattaching the MR asap! [14:44:06] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11306474 (10fgiunchedi) I got so far as getting `heaptrack` to give me a call trace for peak allocations ` PEAK MEMORY CONSUMERS 8.59G peak memory consumed over 9 call... [14:44:23] (03update) 10nurahwakili: T407551-Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/124 [14:45:32] (03update) 10nurahwakili: T407551-Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/124 [14:50:33] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11306510 (10Nurah_Wakili) Hello @Nat_WDU, Just a heads-up — I pushed a few changes I was working on to my MR in case you’ve already reviewed it. Thanks! [14:58:18] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [15:09:41] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-10-24) - https://phabricator.wikimedia.org/T408218 (10SomeRandomDeveloper) 03NEW [15:13:42] 10VPS-project-Codesearch: Codesearch down/unavailable (2025-10-24) - https://phabricator.wikimedia.org/T408218#11306620 (10SomeRandomDeveloper) [15:18:18] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11306683 (10Raboyusuf2024) @Nat_WDU I removed the external links section and its related JavaScript logic after thinking it through. You raised a good point; if I understand correctly, a Paulina user wouldn’t have a str... [15:19:49] 10VPS-Projects, 06Product Safety and Integrity, 10Temporary accounts: Prepare UTRS for temporary accounts - https://phabricator.wikimedia.org/T351028#11306713 (10OKryva-WMF) [15:33:11] 10VPS-project-Codesearch: CodeSearch seems to be out of disk space - https://phabricator.wikimedia.org/T408221 (10DAlangi_WMF) 03NEW [15:34:57] 10VPS-project-Codesearch: CodeSearch seems to be out of disk space - https://phabricator.wikimedia.org/T408221#11306937 (10A_smart_kitten) Seems like the same as/similar to {T408218}, though I'm tempted to merge that here as this task description has more information as to why Codesearch is down. [15:43:50] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [15:44:17] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [15:48:56] (03open) 10dipanshu1223: T406949:Add date inception in work's page [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/151 [15:50:53] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [15:51:17] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [15:53:10] 10Tool-paulina: Add field "inception" to work page - https://phabricator.wikimedia.org/T406949#11307068 (10Dipanshu1223) Good evening @Pepe_piton and @Nat_WDU i made a merge request related this issue:-https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/151 changes made:- - Added inception_d... [16:19:28] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103 [16:19:38] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103 [16:20:47] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 [16:21:54] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 [16:22:40] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106 [16:24:47] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106 [16:32:12] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to [16:32:12] ols-k8s-worker-nfs-17 [16:41:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:46:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:47:58] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:52:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:57:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-2 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:02:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-3 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:03:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet' [17:05:58] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:10:44] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2005-dev.codfw.wmnet' [17:10:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:14:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet' [17:15:50] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2005-dev.codfw.wmnet' [17:15:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:20:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:25:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:28:13] RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:29:57] (03update) 10lebogang: docs: add api-call-examples with tested Wikibase REST API examples [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/144 [17:36:17] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k [17:36:18] 8s-worker-nfs-16, tools-k8s-worker-nfs-17 [17:49:07] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11307661 (10Lebogang) >>! In T405635#11306168, @Nat_WDU wrote: > Hello, @Leobgang! I think that you forgot to add the line that makes the actual request for creating a new aut... [17:54:58] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 [17:55:15] (03update) 10miiswom: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [18:04:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-19 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:09:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-19 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:21:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-19 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:26:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-22 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:29:50] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 [18:49:49] (03open) 10sopzy: API Call Examples for Creating and Editing Wikidata Author and Work Items [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/152 [18:54:36] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11307932 (10Sopzy) Wow! @Pepe_piton @Nat_WDU This really helped me get a much clearer understanding of the property codes and 'Q' item identifiers I spent a couple of hour... [18:59:29] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs [18:59:29] -45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61 [19:06:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:11:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:12:31] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [19:12:56] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [19:17:58] FIRING: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:18:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet' [19:19:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' [19:22:58] RESOLVED: [2x] InstanceDown: Project tools instance tools-k8s-worker-nfs-32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:30:34] FIRING: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [19:50:34] RESOLVED: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [19:52:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-40 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:56:13] RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-40 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:57:31] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [20:21:28] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-54 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:22:33] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [20:23:03] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [20:25:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:25:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [20:25:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [20:25:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [20:25:49] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [20:26:28] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-54 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:27:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [20:30:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [20:32:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet' [20:35:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:37:58] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-54 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:39:45] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t [20:39:45] ools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61 [20:42:58] RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-nfs-54 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:55:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' [20:56:23] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30038 bytes in 0.270 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:17:34] FIRING: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [21:22:34] RESOLVED: TooManyCloudvirtsDown: Reduced availability for CloudVPS codfw - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TooManyCloudvirtsDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DTooManyCloudvirtsDown [21:51:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [21:52:01] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [21:52:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [21:52:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [21:52:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [21:52:54] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [21:54:00] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [21:54:05] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) [21:55:04] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [21:55:07] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on codfw1dev, with recreate False, for hosts list: ['cloudvirt2004-dev', 'cloudvirt2005-dev', 'cloudvirt2006-dev'] [22:20:42] (03open) 10miiswom: Draft: Proof of concept for adding new statements [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/153 [22:41:38] 10Wikibugs, 07affects-translatewiki.net: #translatewiki still getting notifs weeks after merging config change - https://phabricator.wikimedia.org/T408139#11308523 (10bd808) 05Open→03In progress p:05Triage→03High a:03bd808 >>! In T408139#11303757, @bd808 wrote: > Did that whole system break down for... [23:11:00] 10Wikibugs, 13Patch-For-Review: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#11308588 (10bd808) test [23:11:26] (03CR) 10BryanDavis: [C:04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [23:11:52] (03reopen) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) [23:12:00] (03close) 10bd808: This is a test of the gitlab irc reporter [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/33 (https://phabricator.wikimedia.org/T90594) [23:13:42] 10Wikibugs, 07affects-translatewiki.net: #translatewiki still getting notifs weeks after merging config change - https://phabricator.wikimedia.org/T408139#11308594 (10bd808) Testing to make sure this note //does not// end up posted to #translatewiki. It should show up in #wikimedia-dev and #mediawiki-feed. [23:17:14] 10Wikibugs, 07affects-translatewiki.net: #translatewiki still getting notifs weeks after merging config change - https://phabricator.wikimedia.org/T408139#11308595 (10bd808) >>! In T408139#11308594, @bd808 wrote: > Testing to make sure this note //does not// end up posted to #translatewiki. It should show up i... [23:20:13] 10Wikibugs, 07affects-translatewiki.net: #translatewiki still getting notifs weeks after merging config change - https://phabricator.wikimedia.org/T408139#11308596 (10bd808) 05In progress→03Resolved Thanks for reporting that things were weird @Nikerabbit! I had noticed some confusing things about chang... [23:22:57] 10Tool-wsindex, 10Wikisource Reader App: Add an option to sort list of books alphabetically - https://phabricator.wikimedia.org/T407311#11308600 (10System625) Hi @Saiphani02 @Bodhisattwa , I've implemented the sort functionality in the Android app UI and it's passing the correct sort parameters to the API (ver... [23:31:35] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11308606 (10bd808) 05Open→03In progress p:05Low→03Medium a:03bd808 [23:57:53] 10Wikibugs, 10Phabricator: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T402454#11308638 (10bd808) The user.query -> user.search switch is pretty easy. The other two are going to take a lot of staring at code and API responses to fi...