[00:08:57] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:01:20] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Send follow up messages to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T433334#12169944 (10komla) [03:51:59] (03close) 10countcount: Switch API_BASE to the multiuserinfo production deployment [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/13 [03:59:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:04:20] (03open) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/14 [04:04:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [04:08:57] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:10:20] (03update) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/14 [04:12:13] (03update) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/14 [04:12:39] (03merge) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/14 [04:40:29] (03open) 10countcount: Log elapsed time when loading locally disabled global blocks [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/68 [04:43:09] (03update) 10countcount: Trim background-pass logging: elapsed time on the local-disable refresh, quiet per-wiki prune output [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/68 [04:43:09] (03update) 10countcount: Trim background-pass logging: elapsed time on the local-disable refresh, quiet per-wiki prune output [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/68 [04:49:28] (03merge) 10countcount: Trim background-pass logging: elapsed time on the local-disable refresh, quiet per-wiki prune output [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/68 [05:14:50] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/2 [05:14:54] (03open) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/2 [05:15:07] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/15 [05:15:18] (03open) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/15 [05:19:51] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [07:18:23] 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12170140 (10MusikAnimal) 05In progress→03Stalled So, the `pageviews-test` Toolforge account already exists (`pageviews` has a bajillion incoming links; not good for testin... [07:29:55] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/30522988957 (https://github.com/cluebotng/component-configs/commits/e38d48bc554d39c0874c954226583b005318f76f) [07:29:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [07:55:58] 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye depreciatin extension - https://phabricator.wikimedia.org/T433570 (10Leaderboard) 03NEW [07:56:21] 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye depreciation extension - https://phabricator.wikimedia.org/T433570#12170204 (10Leaderboard) [08:05:52] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12170226 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1... [08:08:57] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:10:02] 06cloud-services-team, 10PAWS, 06tools-platform-team, 05WMF-NDA: [paws] automate the current account locking scripts as a first step - https://phabricator.wikimedia.org/T433572 (10dcaro) 03NEW [08:12:36] 10PAWS, 06tools-platform-team: [paws] automate the current account locking scripts as a first step - https://phabricator.wikimedia.org/T433572#12170249 (10JJMC89) [08:14:01] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/2 [08:14:52] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/15 [08:15:11] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 [08:19:06] (03update) 10renovatebot: Update pnpm to v11.17.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 [08:25:28] 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12170344 (10Aklapper) [08:41:18] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12170395 (10JJMC89) This deprecation has been not been properly communicated. - Nothing posted on cloud-announce - Timeline is [[ https://wikitech.wikimedia.or... [08:49:35] 10Tool-lexeme-forms, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): l10n-bot cannot create GitLab merge requests due to gitlab-ssh migration - https://phabricator.wikimedia.org/T432838#12170409 (10abi_) [08:53:44] (03update) 10dcaro: toolforge-common.yaml.j2: add publish_root_domain and publish_proto [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [08:57:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-cli] support publishing continuous job to the internet - https://phabricator.wikimedia.org/T423410#12170431 (10dcaro) [08:58:03] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12170435 (10dcaro) [08:58:25] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] support exposing continuous jobs to the internet - https://phabricator.wikimedia.org/T423408#12170436 (10dcaro) [09:02:04] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12170442 (10dcaro) [09:03:22] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add webservice support - https://phabricator.wikimedia.org/T362077#12170449 (10dcaro) [09:04:16] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] add job publish support - https://phabricator.wikimedia.org/T433355#12170454 (10dcaro) [09:04:38] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add webservice support - https://phabricator.wikimedia.org/T362077#12170455 (10dcaro) [09:16:51] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 [09:16:57] !log filippo@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 [09:18:56] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 [09:23:46] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 [09:24:32] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 [09:25:33] 10Tool-upload-workbench: [Bug] Hello! I noticed the photos on my browser (you can see it from the logs) are … - https://phabricator.wikimedia.org/T433584 (10Sciking) 03NEW [09:29:02] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 [09:38:29] (03open) 10countcount: Unify mirror update jobs and remove completed migrations [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/69 [09:40:16] 10PAWS, 06tools-platform-team: [paws] automate the current account locking scripts as a first step - https://phabricator.wikimedia.org/T433572#12170577 (10dcaro) p:05Triage→03High [09:41:02] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) [09:47:10] (03update) 10tlepage: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) [09:52:17] (03merge) 10countcount: Update pnpm to v11.18.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/24 (owner: 10renovatebot) [09:52:35] (03merge) 10countcount: Update pnpm to v11.18.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/44 (owner: 10renovatebot) [09:52:37] (03merge) 10countcount: Update pnpm to v11.18.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/9 (owner: 10renovatebot) [09:52:43] (03merge) 10countcount: Update pnpm to v11.18.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/15 (owner: 10renovatebot) [09:54:16] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [09:54:29] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12170653 (10fnegri) > Do we want users to be able to look at a list of alerts somewhere other than their email inbox, like through the CLI? I think we'll get this for free... [09:58:19] (03update) 10countcount: Unify mirror update jobs and remove completed migrations [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/69 [09:58:28] (03update) 10countcount: Unify mirror update jobs and remove completed migrations [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/69 [10:02:12] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [10:02:36] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [10:02:59] (03merge) 10countcount: Unify mirror update jobs and remove completed migrations [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/69 [10:04:19] (03open) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:04:37] (03open) 10countcount: Update Node dependencies and lockfile [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/3 [10:05:58] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:06:21] (03merge) 10countcount: Update Node dependencies and lockfile [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/3 [10:12:32] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:12:35] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:13:17] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:14:07] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12170702 (10fgiunchedi) Change to default to dumps-nfs.w.o is deployed ! There are a few stragglers that didn't switch: These hosts run `docker` and bind-mount the dumps directories. Fo... [10:14:32] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [10:14:37] (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/4 [10:14:37] 10Cloud-VPS, 06tools-infrastructure-team: Migrate to dumps-nfs.w.o in Cloud VPS - https://phabricator.wikimedia.org/T432587#12170703 (10fgiunchedi) [10:14:41] (03open) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/4 [10:14:50] (03update) 10renovatebot: Update pnpm to v11.18.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/2 [10:14:58] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:15:09] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:17:43] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:17:49] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:19:56] (03update) 10renovatebot: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 [10:20:47] (03update) 10renovatebot: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 [10:26:11] (03update) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:26:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10Continuous-Integration-Infrastructure: Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592 (10Jdforrester-WMF) 03NEW [10:27:05] (03merge) 10countcount: Update dependency jsdom to v30 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/27 (owner: 10renovatebot) [10:27:20] (03merge) 10countcount: Update Rust crate serial_test to v4 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/61 (owner: 10renovatebot) [10:27:28] (03merge) 10countcount: Show where global blocks are locally disabled [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/16 [10:28:32] (03merge) 10countcount: Update pnpm to v11.18.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/2 (owner: 10renovatebot) [10:29:55] (03open) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/17 [10:31:09] (03close) 10countcount: Repair missing mirrored block expiry values [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/60 [10:32:16] (03merge) 10countcount: Use the production multiuserinfo API [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/17 [10:33:23] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 (owner: 10renovatebot) [10:48:18] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [10:50:35] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_root_domain and publish_proto [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [10:50:38] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_root_domain and publish_proto [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [10:50:39] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_root_domain and publish_proto [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [10:50:52] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [10:53:18] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [10:53:37] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [10:54:19] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [11:14:35] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [11:15:57] (03open) 10dcaro: global: rename toolname -> tool_name [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/165 [11:17:07] (03update) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [11:17:41] (03update) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [11:17:43] (03approved) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [11:20:35] (03approved) 10dcaro: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [11:20:37] (03update) 10dcaro: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [11:25:20] (03Abandoned) 10Urbanecm: [WIP] Add basic tests to Map of Monuments [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/467917 (owner: 10Urbanecm) [11:25:39] (03Abandoned) 10Urbanecm: Add missed login message [labs/tools/articles-needing-links] - 10https://gerrit.wikimedia.org/r/587295 (owner: 10Urbanecm) [11:25:43] (03Abandoned) 10Urbanecm: Add anon-intro [labs/tools/articles-needing-links] - 10https://gerrit.wikimedia.org/r/587297 (owner: 10Urbanecm) [11:25:47] (03Abandoned) 10Urbanecm: Add logged-in-intro [labs/tools/articles-needing-links] - 10https://gerrit.wikimedia.org/r/587296 (owner: 10Urbanecm) [11:26:00] (03Abandoned) 10Urbanecm: Account for page length in score calculation [labs/tools/articles-needing-links] - 10https://gerrit.wikimedia.org/r/588730 (owner: 10Urbanecm) [11:29:31] (03update) 10raymond-ndibe: Draft: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [11:30:07] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [11:31:10] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [11:31:43] (03merge) 10dcaro: envvars-admission: Update open/elasticsearch related vars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1351 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [11:40:46] 10PAWS, 06tools-platform-team: [paws] automate the current account locking scripts as a first step - https://phabricator.wikimedia.org/T433572#12170985 (10dcaro) Updated the scripts we use and the readme, left them running in a screen session on the nfs server. [11:42:33] (03approved) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [11:42:38] (03update) 10dcaro: Paginate JobsLogsResponse [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/28 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [11:52:09] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12171016 (10dcaro) p:05High→03Medium [11:54:00] 06cloud-services-team, 06tools-platform-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#12171030 (10dcaro) @aputhin is this still relevant? [11:54:05] 06cloud-services-team, 06tools-platform-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#12171031 (10dcaro) [11:56:56] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [12:08:57] FIRING: CloudVPSDesignateLeaks: Detected 9 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:12:00] (03open) 10countcount: Mirror per-wiki global-block local-status history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [12:14:19] (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/4 [12:19:11] (03open) 10countcount: Limit historic wide ranges in IP block history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/71 [12:24:49] (03update) 10countcount: Add current-limit filtering for IP block history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/71 [12:24:55] (03update) 10countcount: Add current-limit filtering for IP block history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/71 [12:27:55] (03merge) 10countcount: Add current-limit filtering for IP block history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/71 [12:28:06] (03merge) 10countcount: Update dependency @napi-rs/wasm-runtime to v1.2.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/4 (owner: 10renovatebot) [12:35:23] (03update) 10countcount: Mirror per-wiki global-block local-status history in ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [12:35:50] (03update) 10countcount: Mirror and expose per-wiki global-block local-status history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [12:51:12] 10Tool-wikiloves: Add Portugal to WLE 2025 - https://phabricator.wikimedia.org/T395761#12171195 (10Nemoralis) 05Open→03Invalid https://wikiloves.toolforge.org/earth/2025/Portugal [12:51:18] 10Tool-wikiloves: Add Portugal to WLE 2025 - https://phabricator.wikimedia.org/T395761#12171197 (10Nemoralis) a:05JeanFred→03None [12:53:05] 10Tool-translatetagger, 06Indic-TechCom, 10MediaWiki-extensions-Translate, 07Community-collaboration, and 4 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12171200 (10abi_) 05Open→03Resolved I pull... [12:53:16] 10Tool-translatetagger, 06Indic-TechCom, 10MediaWiki-extensions-Translate, 07Community-collaboration, and 4 others: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12171204 (10abi_) [13:01:49] 10Tool-wikiloves: Wiki Loves Africa 2025 Stats on Toolforge - https://phabricator.wikimedia.org/T391617#12171231 (10Nemoralis) 05Open→03Invalid https://wikiloves.toolforge.org/africa/2025 [13:04:53] 10Tool-wikiloves: WLE in the Democratic Republic of the Congo - https://phabricator.wikimedia.org/T372166#12171248 (10Nemoralis) →14Duplicate dup:03T365064 [13:04:54] 10Tool-wikiloves: Wiki Loves Earth in Democratic Republic of the Congo - https://phabricator.wikimedia.org/T365064#12171250 (10Nemoralis) [13:05:59] 10Tool-wikiloves: Wiki Loves Food 2022 in the Philippines - https://phabricator.wikimedia.org/T336756#12171254 (10Nemoralis) 05Open→03Invalid https://wikiloves.toolforge.org/food/2022/Philippines [13:08:49] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12171261 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt... [13:09:12] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12171263 (10Andrew) p:05Medium→03Unbreak! > Nothing posted on cloud-announce I think this is because communication has been in targeted per-project emails rat... [13:13:02] (03unapproved) 10dcaro: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [13:13:21] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [13:13:29] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [13:13:44] (03approved) 10raymond-ndibe: global: rename toolname -> tool_name [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/165 (owner: 10dcaro) [13:15:30] (03update) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [13:15:39] (03approved) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [13:15:49] (03approved) 10dcaro: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [13:15:56] (03approved) 10dcaro: global: rename toolname -> tool_name [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/165 [13:16:00] (03merge) 10dcaro: global: rename toolname -> tool_name [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/165 [13:16:03] (03update) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [13:26:13] (03CR) 10Majavah: "> I do have some documentation changes and config tweaks I think could be useful as well, since I ended up running this all under rootless" [labs/striker] - 10https://gerrit.wikimedia.org/r/1319174 (owner: 10Perryprog) [13:35:15] (03update) 10countcount: Mirror and expose per-wiki global-block local-status history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [13:35:44] (03update) 10countcount: Mirror and expose per-wiki global-block local-status history [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/70 [13:37:28] (03approved) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [13:37:30] (03update) 10dcaro: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) (owner: 10raymond-ndibe) [13:38:15] (03approved) 10dcaro: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) (owner: 10raymond-ndibe) [13:44:12] (03update) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [13:45:14] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [13:45:37] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [13:45:56] (03merge) 10raymond-ndibe: kubernetes.py: detect conflict with httproutes created by jobs-api [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/114 (https://phabricator.wikimedia.org/T431417) [13:50:23] (03update) 10fnegri: Draft: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [13:51:02] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [13:52:30] (03open) 10dcaro: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 [13:53:02] (03merge) 10raymond-ndibe: toolforge-common.yaml.j2: add publish_url_template [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/332 (https://phabricator.wikimedia.org/T423410) [13:53:20] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [13:54:20] (03update) 10dcaro: cli: fix BrokenPipeError when tailing logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/166 [13:54:25] (03open) 10raymond-ndibe: d/changelog: bump to 0.103.26 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/116 (https://phabricator.wikimedia.org/T431417) [13:56:05] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [13:56:40] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12171455 (10dcaro) >>! In T432860#12170653, @fnegri wrote: >> Do we want users to be able to look at a list of alerts somewhere other than their email inbox, like through th... [13:57:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [13:58:50] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [13:59:28] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli [14:00:04] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [14:00:45] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [14:01:23] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [14:02:04] (03approved) 10raymond-ndibe: d/changelog: bump to 0.103.26 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/116 (https://phabricator.wikimedia.org/T431417) [14:02:18] (03merge) 10raymond-ndibe: d/changelog: bump to 0.103.26 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/116 (https://phabricator.wikimedia.org/T431417) [14:03:51] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:04:16] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [14:04:38] (03merge) 10raymond-ndibe: global: add publish option to expose job to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T423408) [14:04:43] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [14:07:58] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.554-20260730140452-efcd2c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1352 (https://phabricator.wikimedia.org/T423408) [14:08:02] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.554-20260730140452-efcd2c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1352 (https://phabricator.wikimedia.org/T423408) [14:10:55] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12171512 (10dcaro) a:05dcaro→03None [14:14:10] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:16:06] 06cloud-services-team, 06tools-platform-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#12171534 (10dcaro) a:05dcaro→03None [14:16:38] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:17:06] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12171536 (10apaskulin) >>! In T429490#12083661, @bd808 wrote: > The process from https://wikitech.wikimedia.org/wiki/Developer.wikimedia.org#Demo_ser... [14:18:11] 10Tool-wmf-openapi-linter, 06Content-Platform-Team (Work In Progress), 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12171539 (10MSantos) [14:21:33] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12171548 (10mikez-WMF) @TheDJ @Doc_James - thanks so much for your feedback! This is a tricky wish as it covers so many ideas. At a high level I agree that we s... [14:21:55] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:23:29] 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12171560 (10dcaro) [14:25:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:25:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:28:13] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [14:30:46] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [14:31:34] 10Tools, 06Commons, 07Community-Wishlist: A set of tools for image editing directly in Wikimedia Commons - https://phabricator.wikimedia.org/T428062#12171628 (10TheDJ) i'd say, lets first jut make a spot, where already existing user tools can hook into. That shouldn't have to be a lot of design work. Maybe a... [14:33:56] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:40:03] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12171682 (10komla) @Andrew , yes it is targeted. @JJMC89 a bit of history on this. The task was created last year. On August 13th, a **[Cloud-announce] Debian 1... [14:43:28] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:45:49] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.554-20260730140452-efcd2c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1352 (https://phabricator.wikimedia.org/T423408) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:45:53] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.2 Toolforge Alerting System - https://phabricator.wikimedia.org/T432860#12171702 (10fnegri) > I'd strongly recommend not to direct users to karma/prometheus Fair enough, tooling might change and we don't want Toolforge users to rely on it. Tool... [14:46:01] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.554-20260730140452-efcd2c9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1352 (https://phabricator.wikimedia.org/T423408) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:48:02] (03approved) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [14:48:14] (03merge) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T423410) [14:48:15] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [14:52:55] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.32 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/167 (https://phabricator.wikimedia.org/T423410) [14:54:03] FIRING: InstanceDown: Project tools instance tools-bastion-15 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:55:24] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Verify if we can define alert conditions in Prometheus - https://phabricator.wikimedia.org/T432863#12171728 (10fnegri) The MR at https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 defines the following alerts: *... [14:55:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [14:56:10] (03update) 10fnegri: Add new alerts for tool maintainers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/66 (https://phabricator.wikimedia.org/T432863) [14:56:51] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [14:58:30] (03CR) 10Majavah: [C:03+2] Add a link to Bitu for account management (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1319174 (owner: 10Perryprog) [15:00:34] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:00:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:00:43] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [15:00:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:00:55] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [15:01:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:02:31] (03Merged) 10jenkins-bot: Add a link to Bitu for account management [labs/striker] - 10https://gerrit.wikimedia.org/r/1319174 (owner: 10Perryprog) [15:03:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [15:03:29] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:05:57] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [15:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:06:06] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [15:06:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:06:56] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Define alert conditions in Prometheus (if possible) - https://phabricator.wikimedia.org/T432863#12171777 (10fnegri) [15:08:53] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [15:09:45] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:12:00] 10Cloud-VPS (Project-requests): Request creation of PanoCommons VPS project - https://phabricator.wikimedia.org/T433055#12171788 (10fnegri) +1, sounds reasonable to me. [15:14:03] RESOLVED: InstanceDown: Project tools instance tools-bastion-15 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:15:57] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12171799 (10fnegri) Update: {F96528830} `lang=mysql MariaDB [(none)]> SELECT trx_id, trx_started, TIMESTAMPDIFF(SECOND, trx_started, NOW()) AS age_sec... [15:18:52] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli [15:20:48] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12171838 (10Chippyy) maps-warper5 configured. set up mapwarper with apache webserver horizon proxy is now set so it should be serving warper.wmflabs.o... [15:28:33] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12171854 (10JJMC89) The Trixie availability email mentions that deprecation would start in the future and does not constitute an announcement of the deprecation it... [15:30:12] 10Toolforge, 06tools-infrastructure-team: Improve detection of failing ssh to toolforge bastions - https://phabricator.wikimedia.org/T404054#12171861 (10fgiunchedi) bastion-15 was in trouble for ~1h and then something started exhausting all memory, we rebooted it after ~45m after user report of ssh failing. un... [15:33:02] 10Toolforge, 06tools-infrastructure-team: Improve detection of failing ssh to toolforge bastions - https://phabricator.wikimedia.org/T404054#12171877 (10fgiunchedi) Also of note, we should also get per-systemd-unit memory usage breakdowns via cadvisor for example to know what was using all the memory [16:00:48] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12171989 (10komla) @JJMC89 well noted. Thanks for the feedback. [16:08:57] FIRING: CloudVPSDesignateLeaks: Detected 7 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:14:43] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12172025 (10Arnoldokoth) I had a chat with @dcaro and re-created the `phabricator.wmcloud.org` proxy and pointed it to the new host. It does looks li... [16:22:35] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12172043 (10Dzahn) @Arnoldokoth I think you need a fresh scap deployment. There is some config in puppet and some in deployment repo and then scap lo... [16:23:33] 06cloud-services-team, 06tools-platform-team, 07Documentation: Update list of upstream dependencies on mw:Upstream_projects - https://phabricator.wikimedia.org/T328345#12172058 (10aputhin) a:03aputhin Looking at the history, it's still being maintained / updated, so I think it makes sense to take another p... [16:43:42] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Engineering, and 2 others: Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema - https://phabricator.wikimedia.org/T215858#12172165 (10Piastu) [16:43:53] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12172167 (10Arnoldokoth) @Dzahn Oooh. That was indeed the problem. It works fine now. :) I can now delete the old host. [16:46:11] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12172185 (10Dzahn) :) yay, great to hear [16:50:28] (03approved) 10dcaro: Support logs-api pagination [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/163 (https://phabricator.wikimedia.org/T428253) (owner: 10tlepage) [16:58:59] 10Cloud-VPS (Debian Bullseye Deprecation), 06collaboration-services: shutdown / replace bullseye machines in devtools - https://phabricator.wikimedia.org/T429494#12172211 (10Arnoldokoth) [17:10:49] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.32 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/167 (https://phabricator.wikimedia.org/T423410) [17:10:55] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.32 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/167 (https://phabricator.wikimedia.org/T423410) [17:33:04] 10Cloud-VPS (Debian Bullseye Deprecation), 10wikimaps-warper: Migrate wikimaps warper from Debian Bullseye to Trixie - https://phabricator.wikimedia.org/T431805#12172352 (10Chippyy) 05Open→03In progress there's a bug with image thumbnails which I intend to fix but its not completely dependent on the migrat... [18:00:10] (03CR) 10Krinkle: [C:03+2] frontend: Clarify placeholder that regexp must not be wrapped in slashes [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [18:02:00] (03Merged) 10jenkins-bot: frontend: Clarify placeholder that regexp must not be wrapped in slashes [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [18:22:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:26:57] 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch search query format is unclear - https://phabricator.wikimedia.org/T371191#12172475 (10Krinkle) 05Open→03Resolved a:03Krinkle It's not ideal design-wise, as I think a good interface wouldn't require an always-visible textual hint. But, I don't... [18:27:06] 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch search query format is unclear - https://phabricator.wikimedia.org/T371191#12172478 (10Krinkle) a:05Krinkle→03simon04 [18:27:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:50:43] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMIT-Infrastructure: Cloud VPS wlmitvisual01: upgrade Debian bullseye -> trixie - https://phabricator.wikimedia.org/T429723#12172741 (10Ysogo) 05Open→03Resolved I've just erased wlmitvisual01 [20:12:59] FIRING: CloudVPSDesignateLeaks: Detected 9 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:15:17] 10Striker, 06tools-platform-team: Striker dev env gitlab root credentials do not work - https://phabricator.wikimedia.org/T355525#12172785 (10Perryprog) a:03Perryprog Turns out it's because the initial root password has to meet the default complexity requirements (which also can't be configured in the OMNIBU... [20:39:14] (03open) 10musikanimal: channels.yaml: remove #wikimedia-commtech [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/77 [20:45:06] (03merge) 10taavi: channels.yaml: remove #wikimedia-commtech [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/77 (owner: 10musikanimal) [21:31:04] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: gitlab-runners-puppetmaster-01.gitlab-runners.eqiad1.wikimedia.cloud is about to expire in 24d 23h 58m 21s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [21:59:47] 10Tool-toolviews: Unique visitors data only sporadically available - https://phabricator.wikimedia.org/T338053#12173040 (10Husky) Maybe this feature should just be removed or at least get an indication it's broken? This bug has been open for three years and it seems this endpoint is not very useful if it's usual... [22:26:32] (03CR) 10Krinkle: "(Marking as unresolved)" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306100 (https://phabricator.wikimedia.org/T376732) (owner: 10Simon04) [22:48:25] 06tools-infrastructure-team: Create alert for https://public-paws.wmcloud.org - https://phabricator.wikimedia.org/T433650 (10BLiviero-WMF) 03NEW [23:20:10] (03PS1) 10Perryprog: Update dev environment and its documentation [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) [23:23:52] 10Tool-wiki-gender-stats, 13Patch-For-Review: [TF] Migrate to ToolsDB - https://phabricator.wikimedia.org/T432053#12173158 (10Danya) [23:29:24] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12173165 (10Danya) 05Open→03Stalled This issue will be solved by T433199. [23:29:36] 10Tool-wiki-gender-stats: [VERSION] 0.4.0 - https://phabricator.wikimedia.org/T432521#12173171 (10Danya) [23:29:37] 10Tool-wiki-gender-stats: Non-Wikipedia wikis categorized as Wikipedia language codes - https://phabricator.wikimedia.org/T432474#12173170 (10Danya) [23:31:40] (03CR) 10Perryprog: Update dev environment and its documentation (035 comments) [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) (owner: 10Perryprog) [23:36:31] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [23:42:19] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [23:46:15] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92 [23:47:53] (03update) 10danyya: Draft: Resolve T432053 "[TF] Migrate to ToolsDB" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/92