[00:03:00] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146#10446529 (10bd808) [00:09:59] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146#10446534 (10bd808) Striker lives in the production realm today because it handles Developer account passwords and has write access to the LDAP directory... [00:39:16] 10VPS-Projects, 06Content-Transform-Team-WIP, 10Parsoid, 07Essential-Work, 10Parsoid-Read-Views (Phase 1 - DiscussionTools support): upgrade nodejs on parsing-qa-02 - https://phabricator.wikimedia.org/T349941#10446573 (10ssastry) a:03ssastry [00:39:18] 10VPS-Projects, 06Content-Transform-Team-WIP, 10Parsoid, 07Essential-Work, 10Parsoid-Read-Views (Phase 1 - DiscussionTools support): Create a bookwork-imaged VM ctt-qa-03 to replace parsing-qa-03 - https://phabricator.wikimedia.org/T383249#10446574 (10ssastry) a:03ssastry [02:46:41] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Icebox: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#10447048 (10Ottomata) [02:52:32] 06cloud-services-team, 10Cloud-VPS, 06Data-Engineering, 06Data-Engineering-Icebox: Report page views for labs instances - https://phabricator.wikimedia.org/T103726#10447127 (10Ottomata) [03:49:57] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10447188 (10Huji) 05Resolved→03Open @Andrew you did not make a mistake, but we do have a problem. No matter which of the 4 OS choices I pick, the choices for the configuration only allow me to pick up to... [09:17:50] (03update) 10sstefanova: metrics: add prometheus instrumentation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T381249) [09:23:50] (03update) 10sstefanova: metrics: add prometheus instrumentation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T381249) [09:24:07] (03merge) 10sstefanova: metrics: add prometheus instrumentation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T381249) [09:26:34] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.74-20250106230052-21087bd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/653 [09:26:38] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: components-api: bump to 0.0.74-20250106230052-21087bd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/653 (https://phabricator.wikimedia.org/T381249) [09:28:18] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [10:40:48] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 10Data-Platform-SRE (2025.01.11 - 2025.01.31): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10447537 (10Gehel) [13:28:00] (03open) 10sstefanova: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 [14:15:23] 10PAWS: Automate deploy, or move away from nfs paws - https://phabricator.wikimedia.org/T383403 (10rook) 03NEW [14:24:23] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [14:26:33] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406 (10Atrawog) 03NEW [14:27:36] 10Tool-global-search, 06Discovery-Search: 400 - Bad Request on any Global Search - https://phabricator.wikimedia.org/T358541#10448245 (10Gehel) [14:27:51] 10Cloud-VPS (Quota-requests), 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10448248 (10Gehel) [14:31:39] 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10448286 (10Gehel) [14:34:02] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Experimentation Lab, 06Trust and Safety Product Team: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10448326 (10Gehel) [14:34:12] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Remove AAAA records from an-redacteddb1001 and allow connection from cumin - https://phabricator.wikimedia.org/T368220#10448328 (10Gehel) [14:34:34] 06cloud-services-team, 10Cloud-VPS: Decom cloudvirt-wdqs servers - https://phabricator.wikimedia.org/T367770#10448330 (10Gehel) [14:35:52] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406#10448351 (10rook) I believe you will first need a developer account https://www.mediawiki.org/wiki/Developer_account Let me check if there is a tag for getting project access after that [14:36:18] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: Introducing and testing Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10448364 (10Sarai-WMF) [14:36:22] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10448367 (10Sarai-WMF) [14:52:12] 10PAWS: Automate deploy, or move away from nfs paws - https://phabricator.wikimedia.org/T383403#10448403 (10Atrawog) What could be an option is to use a native Kubernetes storage provider like [[ https://longhorn.io/ | Longhorn ]] or [[ https://rook.io | Rook ]] that can be deployed in a helms chart alongside Ju... [15:02:22] (03update) 10sstefanova: cli: Improve deploy-token command UX and safety [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T380706) [15:11:37] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406#10448475 (10Atrawog) I've already created my developer account is https://www.mediawiki.org/wiki/User:Atrawog [15:19:09] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406#10448516 (10joanna_borun) Approved [15:19:54] 10Cloud-VPS (Quota-requests), 10Wikidata, 10Wikidata-Query-Service: Object storage quota increase request for search project - https://phabricator.wikimedia.org/T382601#10448518 (10Gehel) [15:27:08] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406#10448570 (10rook) ` openstack role add --project paws --user atrawog member openstack role add --project paws --user atrawog reader ` Done, you should have access via ssh on the bastion (bastion.paws.eqiad1.wikim... [15:27:26] 10PAWS: Access to PAWS bastion host and horizon access. - https://phabricator.wikimedia.org/T383406#10448572 (10rook) 05Open→03Resolved [15:30:03] 10Cloud-VPS (Project-requests), 10Wikidata, 10Wikidata-Query-Service: Request creation of wikiqlever VPS project - https://phabricator.wikimedia.org/T377655#10448610 (10Gehel) [15:42:58] 06cloud-services-team, 10Cloud-VPS: Decom cloudvirt-wdqs servers - https://phabricator.wikimedia.org/T367770#10448752 (10Gehel) [15:43:14] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Remove AAAA records from an-redacteddb1001 and allow connection from cumin - https://phabricator.wikimedia.org/T368220#10448753 (10Gehel) [15:44:28] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Experimentation Lab, 06Trust and Safety Product Team: Hide rows in the globalblocks table when the associated globaluser row has gu_hidden_level as not 0 - https://phabricator.wikimedia.org/T371488#10448760 (10Gehel) [15:48:08] 10Tool-global-search, 06Discovery-Search: 400 - Bad Request on any Global Search - https://phabricator.wikimedia.org/T358541#10448794 (10Gehel) [15:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:14:31] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10448937 (10Andrew) That is a completely good point :) I will add the flavors, I imagine you won't be the first to run into this issue. [16:19:37] (03open) 10andrew: Add two more flavors, g4.cores4.ram16.disk20 and g4.cores4.ram32.disk20 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/151 (https://phabricator.wikimedia.org/T383020) [16:27:01] (03update) 10andrew: Add two more flavors, g4.cores4.ram16.disk20 and g4.cores4.ram32.disk20 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/151 (https://phabricator.wikimedia.org/T383020) [16:28:39] (03update) 10fnegri: Add two more flavors, g4.cores4.ram16.disk20 and g4.cores4.ram32.disk20 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/151 (https://phabricator.wikimedia.org/T383020) (owner: 10andrew) [16:28:45] (03approved) 10fnegri: Add two more flavors, g4.cores4.ram16.disk20 and g4.cores4.ram32.disk20 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/151 (https://phabricator.wikimedia.org/T383020) (owner: 10andrew) [16:29:49] (03merge) 10andrew: Add two more flavors, g4.cores4.ram16.disk20 and g4.cores4.ram32.disk20 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/151 (https://phabricator.wikimedia.org/T383020) [16:30:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:30:29] 06cloud-services-team, 10Data-Services: move cloudelastic behind cloudlb - https://phabricator.wikimedia.org/T346946#10449061 (10Gehel) [16:31:15] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:31:47] 10Cloud-VPS (Quota-requests): Higher RAM quota for fa-wp VPSs - https://phabricator.wikimedia.org/T383020#10449082 (10Andrew) 05Open→03Resolved new flavors should appear in horizon now if you reload. [16:35:47] 10PAWS: Automate deploy, or move away from nfs paws - https://phabricator.wikimedia.org/T383403#10449123 (10rook) Both would likely be good options. Longhorn is probably a little better if we were on bare metal, probably leaning in the direction of Rook, in particular as we already are running ceph. Though we wo... [16:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:03:59] 06cloud-services-team, 10[DEPRECATED] wdwb-tech, 10API Platform, 06Content-Transform-Team, and 15 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953#10449301 (10Gehel) [17:04:20] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449306 (10Ladsgroup) An example of the error: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawik... [17:05:27] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Data-Services, 06DBA: Prepare and check storage layer for idwikivoyage - https://phabricator.wikimedia.org/T381079#10449321 (10Gehel) [17:13:57] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 06serviceops, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449347 (10Ladsgroup) I want to double check in a mw-api-ext pod but curl doesn't exist there ` root@deploy2... [17:30:14] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 06serviceops, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449378 (10CDanis) https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Exec_into_a_pod_and_run_co... [17:38:10] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 06serviceops, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449389 (10Ladsgroup) Thanks! Now I confirm that it's a firewall issue on the whole pod: ` root@wikikube-wo... [17:39:48] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 06serviceops, 07Wikimedia-production-error: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449390 (10taavi) AIUI MediaWiki should be sending that request directly to itself instead of going through... [17:46:35] 10wikitech.wikimedia.org, 06Growth-Team, 10Notifications, 06serviceops, and 2 others: Wikitech notifications failing to load cross-wiki - https://phabricator.wikimedia.org/T376305#10449423 (10Ladsgroup) This should fix it. [17:57:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:58:00] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2006-dev:9100 - https://phabricator.wikimedia.org/T383432 (10phaultfinder) 03NEW [18:02:22] 06cloud-services-team: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#10449456 (10valerio.bozzolan) > How much disk, RAM, CPU might be needed? Can we meet those needs with our existing hardware? >Are GPUs required? If so, how many? How would access be controlled? Maybe F... [18:06:36] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#10449462 (10bd808) I poked Matt on-wiki a couple of years ago: * 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#10449475 (10bd808) I was checking Matt's activity status as part of filling out an adoption request and noticed he had some edits on enwiki in December. I bumped my pri... [18:17:21] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#10449477 (10bd808) [18:37:48] RESOLVED: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:53:04] 10Tool-toolwatch, 06Toolforge-standards-committee, 07Privacy: toolwatch loads third party resources - https://phabricator.wikimedia.org/T378901#10449541 (10LucasWerkmeister) I’m afraid it’s still loading from a third-party site: `lang=shell-session $ curl -s https://tool-watch.toolforge.org/tools/1 | grep -F... [20:19:26] vivian-rook closed https://github.com/toolforge/paws/pull/480 [21:35:15] 06cloud-services-team: Supporting AI, LLM, and data models on WMCS - https://phabricator.wikimedia.org/T336905#10449924 (10bd808) > Given the lack of clear licensing and/or incompatible with OSI licensing, what can users run on WMCS? The OSI has published [[https://opensource.org/ai/open-source-ai-definition|The... [22:57:30] (03open) 10bd808: requests: Configure a retry handler for idempotent requests [toolforge-repos/gitlab-account-approval] - 10https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/15 [23:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks