[00:12:32] 10Tool-wikitermbase: Arabic technical terms containing Latin characters are missing - https://phabricator.wikimedia.org/T437827 (10Gerges) 03NEW [00:13:34] 10Tool-wikitermbase: Arabic technical terms containing Latin characters are missing - https://phabricator.wikimedia.org/T437827#12315584 (10Gerges) This task moved from [[ https://github.com/forzagreen/wikitermbase/issues/3 | issue 3 ]]. [00:13:58] (03CR) 10CI reject: [V:04-1] build: Updating eslint-config-wikimedia to 0.32.6 [labs/striker] - 10https://gerrit.wikimedia.org/r/1340693 (owner: 10Libraryupgrader) [00:15:43] 10Tool-wikitermbase: Arabic technical terms containing Latin characters are missing - https://phabricator.wikimedia.org/T437827#12315585 (10Gerges) [00:16:13] 10Tool-wikitermbase: Arabic technical terms containing Latin characters are missing - https://phabricator.wikimedia.org/T437827#12315587 (10Gerges) [03:17:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [04:13:19] 10Tool-adminscore, 06Moderator-Tools-Team, 10XTools: Redirect XTools adminscore URLs to adminscore.toolforge.org - https://phabricator.wikimedia.org/T431880#12315705 (10MusikAnimal) p:05Triage→03Medium [04:32:34] (03Abandoned) 10MusikAnimal: tests: skip tests with app.is_wmf instead of returning early [labs/xtools] - 10https://gerrit.wikimedia.org/r/1310607 (owner: 10MusikAnimal) [04:36:27] 10Tool-projo: Address performance issues of NPP review page - https://phabricator.wikimedia.org/T437673#12315739 (10Audiodude) 05Open→03Resolved a:03Audiodude [04:45:08] (03PS1) 10MusikAnimal: docker: Do not start the asset watcher automatically [labs/xtools] - 10https://gerrit.wikimedia.org/r/1340887 [07:25:24] 10Tool-wikiloves: The statistics counter for the Polish-language edition of the Wiki Loves Monuments contest is not working - https://phabricator.wikimedia.org/T437822#12315916 (10Superchilum) It's not just for Polish WLM, it's for the whole tool. [07:55:35] 10Tool-wikiloves: The statistics counter for the Polish-language edition of the Wiki Loves Monuments contest is not working - https://phabricator.wikimedia.org/T437822#12316002 (10JeanFred) 05Open→03Stalled Yes, as @Nemoralis said, this is blocked on {T431309}. [07:57:25] 10Tool-wikiloves: The statistics counter for the Polish-language edition of the Wiki Loves Monuments contest is not working - https://phabricator.wikimedia.org/T437822#12316012 (10JeanFred) (Not a perfect one, but https://ptools.toolforge.org/uploadersincat.php?category=Images_from_Wiki_Loves_Monuments_2026_in_P... [08:05:54] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12316020 (10JeanFred) [08:08:46] 10Data-Services, 06tools-platform-team, 13Patch-For-Review, 07User-notice: Update user-facing Wiki Replicas features and documentation for x4 - https://phabricator.wikimedia.org/T437208#12316026 (10taavi) [08:14:27] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12316049 (10Superchilum) @JeanFred I think that also that the last link you provided is not updated, unfortunately [08:25:56] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [08:27:20] (03update) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [08:33:53] (03open) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 [08:43:48] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 [08:48:00] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 [08:53:03] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 [08:55:35] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 [08:56:19] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T434206) [08:56:20] (03update) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T434206) [08:59:31] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316488 (10dcaro) 05Open→03In progress [08:59:40] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316500 (10dcaro) a:03dcaro [09:07:07] (03approved) 10tlepage: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T434206) (owner: 10dcaro) [09:08:29] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316578 (10dcaro) I'll go with number 2. > Have a field that's a concatenation of job names, like component_names=",component1,compon... [09:08:44] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12316582 (10taavi) created views and pooled clouddb1024/5:x4 [09:13:15] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316616 (10dcaro) [09:15:06] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316632 (10dcaro) a:05dcaro→03None Waiting for the read endpoint to be ready, as it will require creating datastructures that depe... [09:15:17] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12316634 (10dcaro) [09:15:29] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12316635 (10dcaro) [09:15:36] 10Tool-ghanasupremecases1, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316660 (10Sunkanmi12) [09:19:53] 10Tool-ghanasupremecases1, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316668 (10Sunkanmi12) [09:20:48] 10Tool-ghanasupremecases1, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316670 (10Sunkanmi12) Thanks @Edgars2007, I've tagged the tool and added the repo link [09:21:36] 10Toolforge, 06tools-platform-team: ssh to login.toolforge.org hangs with "pledge: filesystem" - https://phabricator.wikimedia.org/T437781#12316673 (10taavi) 05Open→03Resolved a:03taavi Killed your sessions, you should be able to log in now: `lang=shell-session taavi@tools-bastion-15:~ $ sudo loginct... [09:48:25] 10Toolforge, 06tools-platform-team: ssh to login.toolforge.org hangs with "pledge: filesystem" - https://phabricator.wikimedia.org/T437781#12316807 (10jberkel) Yes, I need to convert that specific task to run in a pod, I forgot about it. Thanks! [09:59:55] 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12316848 (10dcaro) >>! In T437573#12311252, @aputhin wrote: > Yep, tho I don't think this needs to be a "decision request"? It's more formalizing our internal discussions. It started as... [10:05:17] (03approved) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T434206) [10:05:23] (03merge) 10dcaro: First commit [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T434206) [10:23:53] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316935 (10aputhin) [10:31:39] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316986 (10taavi) >>! In T437785#12316668, @Sunkanmi12 wrote: > Thanks @Edgars2007, I've tagged the tool a... [10:34:26] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316992 (10Sunkanmi12) @taavi we are trying to deploy the update version to toolforge but facing the afore... [10:35:03] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12316995 (10SincerelySaeed5000) let FastAPI serve the built React files [11:01:46] (03open) 10chimaifeanyi29: T437765: Add check for Category filter [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/15 [11:27:18] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12317209 (10BTullis) [11:47:38] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c) [11:47:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [12:02:00] PROBLEM - mysqld processes on an-redacteddb1001 is CRITICAL: PROCS CRITICAL: 8 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [12:12:05] (03PS1) 10Ayounsi: Add mock secrets for Arelion API [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) [12:15:09] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12317356 (10fgiunchedi) FWIW I agree with the proposal! Is there any reason why we went with hw raid1 for the ssd/nvme on these hosts... [12:20:10] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:20:47] (03CR) 10Ayounsi: [V:03+2 C:03+2] Add mock secrets for Arelion API [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:23:17] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:20] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:23:25] T435921: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 [12:23:26] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:29] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:23:32] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:35] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:23:38] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:41] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:23:44] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:47] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:23:50] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435921) [12:23:53] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435921) [12:24:12] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435923) [12:24:15] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435923) [12:24:28] T435923: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923 [12:31:10] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12317406 (10fgiunchedi) 05In progress→03Resolved Hosts are back in service [12:31:22] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12317411 (10fgiunchedi) 05Open→03Resolved Host is back in service [12:37:00] RECOVERY - mysqld processes on an-redacteddb1001 is OK: PROCS OK: 9 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [12:57:31] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, 07Documentation: Provide documentation for toolforge users to request access to unexposed data through WikiReplicas - https://phabricator.wikimedia.org/T209992#12317534 (10fnegri) [13:09:16] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] create the new cli/repo/skeleton - https://phabricator.wikimedia.org/T434206#12317651 (10dcaro) 05In progress→03Resolved [13:11:41] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12317656 (10Andrew) >>! In T419892#12317356, @fgiunchedi wrote: > FWIW I agree with the proposal! Is there any reason why we went wit... [13:13:09] 10Toolforge, 06tools-platform-team: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12317664 (10dcaro) @CCiufo-WMF This is currentnly blocking me on working on the cli. I'll start implementing a minimal version of this using `toolforge log... [13:15:36] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12317670 (10komla) @Jdlrobson-WMF thanks! [13:18:59] (03open) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T434093) [13:19:39] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T434093) [13:24:33] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T434093) [13:24:41] 10Tool-inteGraality: Syntax-check SPARQL queries before sending them to the endpoint - https://phabricator.wikimedia.org/T437894 (10JeanFred) 03NEW [13:24:52] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [13:40:31] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [13:46:44] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12317850 (10taavi) Fair enough. We have been trying to hint at it at the relevant docs ([[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#%22receiving_more_traffic_tha... [13:49:40] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12317872 (10fgiunchedi) +1 LGTM [13:51:04] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12317881 (10Andrew) Hello @Peter. I may be missing the obvious, but does your table include how long it took... [13:51:29] (03update) 10raymond-ndibe: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [13:52:50] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12317900 (10Andrew) I'm also wondering about the first run on a new worker vs. subsequent runs. I'd expect do... [13:53:14] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate tools for Push-to-Deploy - https://phabricator.wikimedia.org/T437897 (10komla) 03NEW [13:54:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate maintainers - https://phabricator.wikimedia.org/T437898 (10komla) 03NEW [13:54:44] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate tools for Push-to-Deploy - https://phabricator.wikimedia.org/T437897#12317936 (10komla) 05Open→03Resolved [13:54:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Identify candidate maintainers - https://phabricator.wikimedia.org/T437898#12317940 (10komla) 05Open→03Resolved [13:57:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create Hello World deployment - https://phabricator.wikimedia.org/T437899 (10komla) 03NEW [13:58:17] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12317978 (10komla) [13:59:50] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900 (10komla) 03NEW [14:02:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903 (10komla) 03NEW [14:02:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12318099 (10komla) [14:03:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12318115 (10komla) [14:03:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12318122 (10komla) [14:03:59] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12318129 (10komla) [14:04:29] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318133 (10komla) [14:04:40] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Identify candidate maintainers - https://phabricator.wikimedia.org/T437898#12318134 (10komla) [14:04:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Identify candidate tools for Push-to-Deploy - https://phabricator.wikimedia.org/T437897#12318140 (10komla) [14:09:52] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12318173 (10komla) 05Open→03In progress [14:09:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12318176 (10komla) 05Open→03In progress [14:10:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create code change and redeploy tutorial - https://phabricator.wikimedia.org/T437900#12318180 (10komla) 05Open→03In progress [14:11:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12318195 (10Raymond_Ndibe) 05In progress→03Resolved [14:11:57] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12318196 (10komla) [14:12:20] 06tools-platform-team: jobs-api: job with failing healthcheck should not be marked running in the new status implementation - https://phabricator.wikimedia.org/T436726#12318212 (10Raymond_Ndibe) 05In progress→03Resolved [14:12:45] (03approved) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/188 (https://phabricator.wikimedia.org/T436485) [14:13:16] (03merge) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/188 (https://phabricator.wikimedia.org/T436485) [14:14:35] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [14:22:23] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [14:27:00] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318311 (10dcaro) You can reuse https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images/My_first_Buildpack_Python_tool I think [14:41:54] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12318344 (10dcaro) a:05CCiufo-WMF→03dca... [14:50:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:54:27] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: More granular 'wmcs' clusters for better observability - https://phabricator.wikimedia.org/T437272#12318429 (10fgiunchedi) [14:54:40] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12318430 (10Peter) [14:55:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:58:19] (03open) 10fnegri: Add ruff and mypy to dev dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/2 [14:58:34] (03open) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] (fran-T432865-883c4b2c) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) [15:06:58] 10Tool-leximap: Add Malayalam language to Leximap - https://phabricator.wikimedia.org/T433956#12318450 (10Mohammed_Sadat_WMDE) @Dagmawi-M Please take a look [15:08:33] (03approved) 10tlepage: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [15:11:31] (03update) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] (fran-T432865-883c4b2c) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) [15:11:34] (03update) 10fnegri: Add ruff and mypy to dev dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/2 [15:14:42] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [15:14:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [15:14:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [15:15:02] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [15:15:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [15:15:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [15:15:20] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [15:15:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [15:15:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [15:16:03] (03approved) 10dcaro: Add ruff and mypy to dev dependencies [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/2 (owner: 10fnegri) [15:17:08] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [15:17:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [15:17:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [15:17:29] (03update) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [15:17:34] (03approved) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [15:17:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [15:17:40] (03merge) 10dcaro: global: implement fetching all events for a tool [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/2 (https://phabricator.wikimedia.org/T432572) [15:17:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [15:18:21] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [15:18:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [15:18:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [15:19:10] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12318484 (10fgiunchedi) a:05fgiunchedi→03Jclark-ctr >>! In T419892#12317656, @Andrew wrote: >>>! In T419892#12317356, @fgiunchedi... [15:19:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [15:19:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [15:20:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [15:20:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [15:20:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [15:20:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [15:20:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [15:22:02] (03PS1) 10David Caro: deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) [15:24:24] (03CR) 10David Caro: deploy,gitlab: add logs-cli (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [15:24:42] (03CR) 10CI reject: [V:04-1] deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [15:31:06] (03PS2) 10David Caro: deploy,gitlab: add logs-cli [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) [15:31:28] (03CR) 10David Caro: deploy,gitlab: add logs-cli (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341237 (https://phabricator.wikimedia.org/T432572) (owner: 10David Caro) [15:32:26] (03update) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] (fran-T432865-883c4b2c) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 [15:32:35] (03open) 10dcaro: global: add logs-cli support [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/342 (https://phabricator.wikimedia.org/T432572) [15:33:53] (03open) 10dcaro: d/changelog: bump to 0.0.2 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/3 [15:35:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [15:35:47] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-cli [15:36:44] (03open) 10dcaro: toolforge_get_versions: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) [15:41:01] (03update) 10dcaro: toolforge_get_versions: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) [15:41:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [15:41:08] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-cli [15:42:12] (03update) 10dcaro: toolforge_get_versions,components: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) [15:43:35] (03open) 10fnegri: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) [15:47:57] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318641 (10dcaro) @komla can you specify what do you need reviews of? [15:48:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12318644 (10dcaro) @komla same here, what are the needed? [15:55:52] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12318715 (10dcaro) [15:55:53] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12318716 (10dcaro) [15:56:07] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12318717 (10Peter) Thanks for the questions! Yes the effect on the build is in the description, just below... [15:57:04] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12318728 (10dcaro) a:03dcaro [15:57:35] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12318750 (10dcaro) >>! In T432571#12316632, @dcaro wrote: > Waiting for the read endpoint to be ready, as it will require creating data... [15:57:43] (03update) 10fnegri: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] (fran-T432865-883c4b2c) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) [16:09:16] 10wikitech.wikimedia.org, 06Content-Platform-Team (Work In Progress): Parsoid Read Views: Deploy to wikitech - https://phabricator.wikimedia.org/T437916 (10ssastry) 03NEW [16:14:46] 10wikitech.wikimedia.org, 10Parsoid, 10Parsoid-Read-Views, 06Content-Platform-Team (Work In Progress): Parsoid Read Views: Deploy to wikitech - https://phabricator.wikimedia.org/T437916#12318849 (10ssastry) [16:15:26] (03approved) 10fnegri: toolforge_get_versions,components: add logs-cli [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1418 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [16:15:46] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318856 (10komla) [16:16:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318859 (10komla) @dcaro it's in draft on my page. I will make it Python specific(as opposed to the PHP one above). I've added the link now. [16:19:28] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12318897 (10dcaro) >... [16:20:14] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Establish design principles for parity between interfaces (CLI vs REST API) - https://phabricator.wikimedia.org/T434093#12318899 (10dcaro) fyi. we made one decision for the new `logs-cli` only to go for parity: https://phabricator.wikimedia.org/T432572#... [16:20:53] 10Toolforge, 06tools-platform-team: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12318902 (10dcaro) We decided for API parity: https://phabricator.wikimedia.org/T432572#12318897 [16:21:15] 10Toolforge, 06tools-platform-team: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12318904 (10dcaro) p:05Triage→03High [16:21:24] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12318906 (10dcaro) [16:22:10] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12318907 (10dcaro) p:05Low→03High We decided to go for API parity: https://phabricator.wikimedia.org/T432572#12318897 [16:22:40] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12318914 (10dcaro) [16:22:42] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12318915 (10dcaro) [16:22:50] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12318917 (10dcaro) [16:22:53] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12318918 (10dcaro) [16:28:02] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12318926 (10dcaro) >>! In T437899#12318856, @komla wrote: > @dcaro it's in draft on my page so just someone to take a look/review. I will make it Python specific(as o... [16:29:15] 10wikitech.wikimedia.org, 10Parsoid, 10Parsoid-Read-Views, 06Content-Platform-Team (Work In Progress), 13Patch-For-Review: Parsoid Read Views: Deploy to wikitech - https://phabricator.wikimedia.org/T437916#12318932 (10ssastry) [16:38:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12319003 (10dcaro) Some early feedback: > We will deploy a Python application called recent-changes. The application runs every five minutes as a scheduled job. We... [16:43:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:46:01] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12319031 (10dcaro) 05Open→03In progress [16:48:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:50:10] (03PS2) 10MusikAnimal: docker: Do not start the asset watcher automatically [labs/xtools] - 10https://gerrit.wikimedia.org/r/1340887 [16:50:57] (03CR) 10MusikAnimal: [C:03+2] docker: Do not start the asset watcher automatically [labs/xtools] - 10https://gerrit.wikimedia.org/r/1340887 (owner: 10MusikAnimal) [16:52:07] (03Merged) 10jenkins-bot: docker: Do not start the asset watcher automatically [labs/xtools] - 10https://gerrit.wikimedia.org/r/1340887 (owner: 10MusikAnimal) [17:14:17] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [17:14:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [17:14:25] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [17:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [17:14:34] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [17:14:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [17:15:32] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [17:15:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [17:15:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [17:16:14] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [17:16:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [17:16:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [17:17:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [17:17:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [17:17:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [17:54:31] (03CR) 10Chuiimuii_ofc: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1302134 (https://phabricator.wikimedia.org/T354920) (owner: 10Chuiimuii_ofc) [17:59:34] (03open) 10taavi: eqiad1: cloudinfra: Create security group for etcd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/353 (https://phabricator.wikimedia.org/T410721) [17:59:46] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/353 [18:00:09] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/353 [18:01:28] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [18:01:31] (03merge) 10taavi: eqiad1: cloudinfra: Create security group for etcd [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/353 (https://phabricator.wikimedia.org/T410721) [18:02:07] (03PS2) 10Majavah: Migrate etcd cookbooks to their own hierarchy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337604 (https://phabricator.wikimedia.org/T437219) [18:02:07] (03PS2) 10Majavah: etcd: Use existing Enc wrapper instead of direct wmcs-enc-cli calls [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337605 (https://phabricator.wikimedia.org/T437219) [18:02:07] (03PS1) 10Majavah: etcd: Inline Hiera operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341302 (https://phabricator.wikimedia.org/T437219) [18:02:09] (03PS1) 10Majavah: etcd: Make Hiera keys configurable [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341303 (https://phabricator.wikimedia.org/T437219) [18:02:12] (03PS1) 10Majavah: inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) [18:03:14] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [18:03:54] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:04:05] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [18:05:09] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:05:17] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [18:08:08] (03PS2) 10Majavah: inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) [18:08:09] (03PS1) 10Majavah: etcd: Allow specifying network for a new etcd node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341306 (https://phabricator.wikimedia.org/T437219) [18:08:42] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:15:18] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [18:18:16] 10Toolforge, 06tools-platform-team: maintain-dbusers gives unhelpful error when forgetting to pass --account-type=tool - https://phabricator.wikimedia.org/T437934 (10taavi) 03NEW [18:19:05] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-etcd-1 [18:19:22] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-etcd-1 [18:21:40] 10Toolforge, 06tools-platform-team: maintain-dbusers 500 with no further context when deleting a user - https://phabricator.wikimedia.org/T437935 (10taavi) 03NEW [18:22:29] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:31:34] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [18:32:10] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-etcd-1 [18:32:33] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-etcd-1 [18:34:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:36:50] (03PS2) 10Majavah: etcd: Inline Hiera operations [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341302 (https://phabricator.wikimedia.org/T437219) [18:36:50] (03PS2) 10Majavah: etcd: Make Hiera keys configurable [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341303 (https://phabricator.wikimedia.org/T437219) [18:36:51] (03PS2) 10Majavah: etcd: Allow specifying network for a new etcd node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341306 (https://phabricator.wikimedia.org/T437219) [18:36:51] (03PS3) 10Majavah: inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) [18:36:52] (03PS1) 10Majavah: etcd: Allow bootstrapping new clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341316 (https://phabricator.wikimedia.org/T437219) [18:36:58] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:45:00] !log taavi@cloudcumin1001 cloudinfra END (FAIL) - Cookbook wmcs.etcd.add_cluster_node (exit_code=99) [18:45:05] (03CR) 10Majavah: [C:03+2] Treat etcd clusters as separate concepts from K8s clusters (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337603 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:46:00] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-etcd-1 [18:46:48] (03CR) 10Majavah: [C:03+2] Migrate etcd cookbooks to their own hierarchy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337604 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:46:53] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-etcd-1 [18:46:54] (03CR) 10Majavah: [C:03+2] etcd: Use existing Enc wrapper instead of direct wmcs-enc-cli calls [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337605 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:48:45] (03Merged) 10jenkins-bot: Treat etcd clusters as separate concepts from K8s clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337603 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:50:07] (03Merged) 10jenkins-bot: Migrate etcd cookbooks to their own hierarchy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337604 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:50:09] (03Merged) 10jenkins-bot: etcd: Use existing Enc wrapper instead of direct wmcs-enc-cli calls [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337605 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [18:50:24] (03PS2) 10Majavah: etcd: Allow bootstrapping new clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341316 (https://phabricator.wikimedia.org/T437219) [18:50:24] (03PS4) 10Majavah: inventory: Add cloudinfra etcd cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341304 (https://phabricator.wikimedia.org/T410721) [18:50:36] !log taavi@cloudcumin1001 cloudinfra START - Cookbook wmcs.etcd.add_cluster_node [18:58:58] !log taavi@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.etcd.add_cluster_node (exit_code=0) [19:09:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:38:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:31:32] 10Tool-Pageviews, 07Regression: Massviews Analysis fails due to 414 Request-URI Too Long - https://phabricator.wikimedia.org/T437947 (10Fuzzy) 03NEW [20:33:02] 10Tool-Pageviews, 07Regression: Massviews Analysis fails due to 414 Request-URI Too Long - https://phabricator.wikimedia.org/T437947#12319967 (10Fuzzy) [20:45:51] (03update) 10tmoney: Bound concurrent searches and add request metadata logging [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/4 [20:45:52] (03open) 10tmoney: Bound concurrent searches and add request metadata logging [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/4 [20:46:26] (03merge) 10tmoney: Bound concurrent searches and add request metadata logging [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/4 [20:53:05] 10Tool-findlinkfast: findlinkfast: Keep batch worklists and progress updates usable under search overload - https://phabricator.wikimedia.org/T437949 (10Audiodude) 03NEW [21:28:32] 10Tool-wikinewsie: Default time window on Wikinewsie proportionate to edits/day on the language and portal - https://phabricator.wikimedia.org/T437956 (10Pharos) 03NEW [21:29:50] 10Tool-wikinewsie: Default time windows on Wikinewsie proportionate to edits/day on the language and portal - https://phabricator.wikimedia.org/T437956#12320168 (10Pharos) [21:40:09] 10Tool-wikiloves: Wikiloves statistics are not being updated - https://phabricator.wikimedia.org/T437822#12320228 (10Nemoralis) https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/message/GPMPEOZYXIM2VBR277FYSBCPIVHMEVHQ/ > The new x4 database section is now live on the Wiki Replicas. Tools up... [21:54:00] 10Tool-wikinewsie, 13Patch-For-Review: Wikinewsie shouldn't use year categories for Wikipedia languages where they don't exist - https://phabricator.wikimedia.org/T437765#12320250 (10Pharos) I think I actually overestimated how many Wikipedia languages lack 2026 categories. In fact uzwiki does have such a cate... [22:07:30] 10Tool-Pageviews, 07Regression: Massviews Analysis fails due to 414 Request-URI Too Long - https://phabricator.wikimedia.org/T437947#12320269 (10MusikAnimal) 05Open→03In progress a:03MusikAnimal Thanks for reporting! I'll get this fixed. [22:18:38] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.215-20260914141925-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) [22:20:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [22:20:34] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [22:20:54] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [22:21:01] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [22:25:54] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [22:26:01] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [22:30:13] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.215-20260914222705-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) [22:30:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.215-20260914222705-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) [22:30:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [22:35:30] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [22:42:50] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [22:48:07] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [22:48:28] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12320355 (10Southparkfan) NOTE: TL;DR: Beta Clu... [22:50:56] (03update) 10raymond-ndibe: components-api: bump to 0.0.215-20260914222705-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:51:01] (03approved) 10raymond-ndibe: components-api: bump to 0.0.215-20260914222705-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:51:15] (03merge) 10raymond-ndibe: components-api: bump to 0.0.215-20260914222705-5d6e6ba7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1420 (https://phabricator.wikimedia.org/T436485) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [22:52:02] 06tools-platform-team, 13Patch-For-Review: components-api: update toolforge models - https://phabricator.wikimedia.org/T436485#12320364 (10Raymond_Ndibe) 05In progress→03Resolved [23:05:50] (03PS1) 10MusikAnimal: Add support for Commons x4 shard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341400 (https://phabricator.wikimedia.org/T437830) [23:14:00] (03PS1) 10MusikAnimal: 3.26.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341402 [23:15:04] (03CR) 10MusikAnimal: [C:03+2] 3.26.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341402 (owner: 10MusikAnimal) [23:16:03] (03Merged) 10jenkins-bot: 3.26.0 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1341402 (owner: 10MusikAnimal) [23:35:46] 10Tool-wikitermbase, 07good first task: Replace deprecated document.execCommand('copy') with navigator.clipboard.writeText() - https://phabricator.wikimedia.org/T437964 (10Gerges) 03NEW Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things... [23:54:41] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320526 (10BCornwall) deployment-cp-text09 and deployment-cp-upload09 are provisioned and ready for...