[00:00:53] 10VPS-project-Codesearch, 07Regression: Codesearch suddenly uses the default Hound UI - https://phabricator.wikimedia.org/T429967#12048645 (10Daimona) Guessing that it might be fallout from this task, while making any search I now get: ` TypeError: Failed to fetch ` And in the browser console, e.g. for h... [04:13:53] 06cloud-services-team, 10Cloud-VPS: Project members cannot ssh into newly created deployment-prep instances - https://phabricator.wikimedia.org/T429978#12048826 (10BLiviero-WMF) Hi! with T429542 being resolved, can you confirm whether this problem is also fixed? thank you! [04:34:40] 10VPS-Projects: Redirect wsexport.wmflabs.org to ws-export.wmcloud.org - https://phabricator.wikimedia.org/T430013 (10Samwilson) 03NEW [04:35:02] 10VPS-Projects, 10WS Export: Redirect wsexport.wmflabs.org to ws-export.wmcloud.org - https://phabricator.wikimedia.org/T430013#12048839 (10Samwilson) [05:29:25] 10Tool-wiktlexbot: Use Wiktionary page title dump to retrieve the list of existing entries - https://phabricator.wikimedia.org/T419317#12048870 (10Redmin) 05Stalled→03Open a:03Redmin [05:44:34] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.878% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:27:44] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018 (10Volans) 03NEW p:05Triage→03High [06:32:16] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12048985 (10Volans) [06:32:39] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12048986 (10Volans) [06:51:35] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12049010 (10Volans) The backup process on cloudbackup1004 is also taking much more than one day to run: ` $ journalctl -u backup_vms -g 'backup_vms.service' -n 30 Jun 23... [07:07:43] (03open) 10dcaro: update deps [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/8 [07:08:00] (03approved) 10dcaro: readme: fix link [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/7 (owner: 10lucaswerkmeister) [07:08:19] (03approved) 10dcaro: update deps [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/8 [07:08:45] (03merge) 10dcaro: update deps [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/8 [07:14:57] 06tools-platform-team: [sample-complex-app] currently failing to start both api and worker - https://phabricator.wikimedia.org/T429981#12049053 (10dcaro) 05Open→03Resolved p:05Triage→03Medium a:03dcaro Updated the code to python >3.13, fixes some type annotations, and added a minimal set of tests,... [07:16:40] (03update) 10dcaro: readme: fix link [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/7 (owner: 10lucaswerkmeister) [07:17:48] (03merge) 10dcaro: readme: fix link [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/7 (owner: 10lucaswerkmeister) [08:16:15] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12049179 (10Marostegui) @fnegri clouddb1026-1033 arrived. I am going to start working on them, so I will be depooling hosts (one at the time) to... [08:19:51] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12049187 (10Marostegui) [08:24:15] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12049195 (10Marostegui) [08:24:38] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#12049196 (10dcaro) →14Duplicate dup:03T348755 [08:24:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12049197 (10dcaro) [08:25:11] (03update) 10dcaro: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [08:25:18] (03update) 10dcaro: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [09:11:15] 06cloud-services-team, 10Cloud-VPS: Request creation of wiki-community-metrics-service VPS project - https://phabricator.wikimedia.org/T429983#12049316 (10Aklapper) This sounds interesting. Have there been on-wiki discussions about this proposed service? Which wiki(s) is this targeting? What does "analyzes wik... [09:14:16] 06cloud-services-team, 10Cloud-VPS: Request creation of wiki-community-metrics-service VPS project - https://phabricator.wikimedia.org/T429983#12049327 (10fgiunchedi) In addition to what @Aklapper mentioned, I would like to better understand each blocking requirement for Toolforge and what we could do for this... [09:16:05] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Define a process for OAD resource synchronization - https://phabricator.wikimedia.org/T422927#12049330 (10KBach) 05Open→03In progress [09:23:09] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12049373 (10fgiunchedi) >>! In T429729#12046663, @vitaly-zdanevich wrote: > Ok, thanks, I never used Toolforge - according to the instruction - I sent the Toolforge... [09:30:35] (03open) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [09:31:52] (03open) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [09:38:40] (03update) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [09:38:43] (03update) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [09:39:40] (03update) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [09:44:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.271% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:57:58] (03open) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [09:59:49] (03update) 10dcaro: create_job: call get_resolved_core_job() to validate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) (owner: 10fnegri) [10:16:39] (03open) 10kbach: Draft: Link to rule reference [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/25 [10:18:02] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki isvwiki - https://phabricator.wikimedia.org/T429938#12049620 (10fnegri) a:03fnegri [10:28:01] (03update) 10fnegri: create_job: call get_resolved_core_job() to validate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:28:17] (03update) 10fnegri: create_job: call get_resolved_core_job() to validate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:30:10] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:36:20] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12049648 (10Aklapper) Yeah, another ticket makes most sense (I don't know the workflow or the required group membership from the top of my head). Thanks! [10:37:29] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:37:33] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:40:45] (03merge) 10fnegri: api_get_jobs: compute correct value for missing fields [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/316 (https://phabricator.wikimedia.org/T428354) [10:42:14] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [10:44:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.514-20260624104056-5c46dda2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354) [10:44:26] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.514-20260624104056-5c46dda2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354) [10:53:00] 10VPS-project-Phabricator: Metatron33 - https://phabricator.wikimedia.org/T430031 (10Metatron110) 03NEW [10:57:15] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [11:02:48] (03update) 10fnegri: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 (owner: 10dcaro) [11:03:14] (03update) 10fnegri: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 (owner: 10dcaro) [11:05:59] (03update) 10fnegri: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 (owner: 10dcaro) [11:06:00] (03update) 10fnegri: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 (owner: 10dcaro) [11:07:49] (03approved) 10fnegri: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 (owner: 10dcaro) [11:11:13] 10Tool-wsindex, 10Wikisource Reader App (Android): Analytics table should have book title in original language - https://phabricator.wikimedia.org/T417148#12049765 (10Saiphani02) 05Open→03Resolved [11:15:47] (03update) 10fnegri: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 (owner: 10dcaro) [11:30:55] 10VPS-project-Codesearch: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033 (10Tgr) 03NEW [11:31:28] 10VPS-project-Codesearch, 07Regression: Codesearch suddenly uses the default Hound UI - https://phabricator.wikimedia.org/T429967#12049824 (10Tgr) Filed as {T430033} (the DNS entry existed yesterday so probably not related to this task?) [11:36:49] 10VPS-project-Codesearch: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033#12049837 (10Tgr) (Would be nice to mention that domain in the docs at https://www.mediawiki.org/wiki/Codesearch/Admin BTW) [11:44:47] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [11:45:28] (03update) 10dcaro: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 [11:45:30] (03update) 10dcaro: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 [11:45:41] (03update) 10dcaro: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 [11:47:05] (03approved) 10dcaro: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) (owner: 10fnegri) [11:50:57] (03approved) 10dcaro: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T428354) (owner: 10fnegri) [11:51:14] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12049866 (10Marostegui) ` [11:49:36] marostegui@cumin1003:~$ sudo confctl select name=clouddb1026.eqiad.wmnet set/weight=100 eqiad/wikireplica-d... [11:54:03] (03merge) 10dcaro: config: update the default builder/runner [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/164 [11:54:08] (03update) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [11:58:12] (03unapproved) 10dcaro: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) (owner: 10fnegri) [12:05:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 [12:05:52] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 [12:10:05] (03open) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [12:20:30] 10VPS-Projects, 10WS Export: Redirect wsexport.wmflabs.org to ws-export.wmcloud.org - https://phabricator.wikimedia.org/T430013#12049936 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi {{done}} [12:20:30] (03update) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [12:28:13] (03open) 10samwilson: Remove wmflabs.org config from Apache conf [toolforge-repos/wsexport-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T430013) [12:28:39] (03merge) 10samwilson: Remove wmflabs.org config from Apache conf [toolforge-repos/wsexport-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport-config/-/merge_requests/1 (https://phabricator.wikimedia.org/T430013) [12:41:00] (03update) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [12:42:17] (03open) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [12:53:38] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [12:56:42] (03update) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [12:57:01] (03update) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [12:58:57] (03update) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [13:44:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 5.655% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:01:24] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28103726388 (https://github.com/cluebotng/component-configs/commits/3ae736249e7b8cae3c57261c55b221a4d62296f9) [14:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:06:31] 10VPS-project-Codesearch, 07Regression: Codesearch suddenly uses the default Hound UI - https://phabricator.wikimedia.org/T429967#12050589 (10Dzahn) >>! In T429967#12048645, @Daimona wrote: > Guessing that it might be fallout from this task Can't confirm. I never touched any DNS records whatsoever and thi... [14:11:41] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057 (10Jacob_WMDE) 03NEW [14:12:20] 10Data-Services, 06tools-platform-team, 06Data-Persistence: CPU spikes in clouddb1014@s2 - https://phabricator.wikimedia.org/T429892#12050623 (10aputhin) p:05Triage→03Low [14:12:28] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [14:13:20] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki isvwiki - https://phabricator.wikimedia.org/T429938#12050629 (10aputhin) p:05Triage→03Medium [14:13:26] 06cloud-services-team, 10Data-Services, 06tools-platform-team: [wikireplicas] Create views for new wiki magwiki - https://phabricator.wikimedia.org/T428282#12050630 (10aputhin) p:05Triage→03Medium [14:14:19] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Create Cloud VPS VM Migration Guide - https://phabricator.wikimedia.org/T429635#12050641 (10aputhin) 05Open→03In progress [14:14:49] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Create Cloud VPS VM Migration Guide - https://phabricator.wikimedia.org/T429635#12050642 (10aputhin) p:05Triage→03High [14:18:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#12050650 (10aputhin) p:05High→03Medium [14:24:31] 10Toolforge, 06tools-platform-team: [logs-api] failing to return logs for job - https://phabricator.wikimedia.org/T429265#12050700 (10aputhin) 05Open→03Declined p:05Triage→03Low Closing for now as it (apparently) didn't happen again. Feel free to reopen if it does. [14:28:11] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12050725 (10aputhin) p:05Triage→03Low Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not b... [14:33:27] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12050754 (10Pppery) You need someone in https://meta.wikimedia.org/wiki/Special:ListUsers?username=&group=oauthadmin&wpsubmit=&wpFormIdentifier=mw-listusers-form&lim... [14:35:22] 10Toolforge, 06tools-platform-team: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12050761 (10aputhin) We can think of adding pagination to the params and improving the UX of the existing params. `--last` is not very intuitive on its own. [14:37:07] 10Tool-phab-ban: Add a button to phab-ban that turns on manual account approval - https://phabricator.wikimedia.org/T420251#12050767 (10Pppery) @Aklapper You should probably also change the edit policy. [14:39:23] 06cloud-services-team, 10Toolforge, 06tools-platform-team: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214#12050774 (10aputhin) 05Open→03Resolved Resolving for now. If it happens again we can dig deeper, as we would also have more data available from logs. [14:39:40] 10Toolforge, 06tools-platform-team: [logs-api] failing to return logs for job - https://phabricator.wikimedia.org/T429265#12050778 (10DamianZaremba) @aputhin it would be good to get to the bottom of the transient errors as it causes hassle for maintainers every time it happens, which goes through phases of... [14:59:29] 06cloud-services-team, 10Cloud-VPS: Project members cannot ssh into newly created deployment-prep instances - https://phabricator.wikimedia.org/T429978#12050813 (10dancy) 05Open→03Resolved a:03dancy >>! In T429978#12048826, @BLiviero-WMF wrote: > Hi! with T429542 being resolved, can you confirm wheth... [15:02:04] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12050824 (10Jclark-ctr) a:03Andrew Is there a ticket tracking the rebalancing of CloudVirts for power to make room? Has any decision been made regarding... [15:05:13] 10Toolforge, 06tools-platform-team: [logs-api] failing to return logs for job - https://phabricator.wikimedia.org/T429265#12050840 (10aputhin) >>! In T429265#12050778, @DamianZaremba wrote: > @aputhin it would be good to get to the bottom of the transient errors as it causes hassle for maintainers every ti... [15:21:20] (03open) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [15:21:39] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [15:22:08] (03update) 10fnegri: create_job: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [15:22:15] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [15:25:51] (03update) 10fnegri: core/core.py: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [15:26:22] (03update) 10fnegri: core/core.py: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [15:39:33] (03update) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [15:42:42] 10VPS-project-Wikistats: Add isvwiki to wikistats - https://phabricator.wikimedia.org/T429940#12051006 (10Dzahn) a:03Dzahn [15:52:42] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12051057 (10fgiunchedi) Yes the cloudvirt rebalancing task is {T424658} and the last table in the description lists the host moves to get to balance. I'll... [15:53:58] (03update) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [15:57:27] (03approved) 10dcaro: core/core.py: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) (owner: 10fnegri) [16:05:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [16:10:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [16:28:32] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [16:29:16] (03update) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [16:31:41] (03update) 10fnegri: core/core.py: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [16:31:48] (03merge) 10fnegri: core/core.py: call get_resolved_core_job() early [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/319 (https://phabricator.wikimedia.org/T429907) [16:31:59] (03update) 10fnegri: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T428354) [16:32:23] (03update) 10fnegri: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T428354) [16:32:32] (03merge) 10fnegri: Do not require --filelog if --filelog-std(out|err) is used [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/158 (https://phabricator.wikimedia.org/T428354) [16:34:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [16:34:31] 10Toolforge, 06tools-platform-team: [logs-cli] `since` + `until` limited by `last` - https://phabricator.wikimedia.org/T428253#12051236 (10dcaro) As a note, currently our loki deployment limits the number of logs per query to 5000 (that's why the default value of last is 5000): https://grafana.com/docs/loki/la... [16:34:41] (03approved) 10fnegri: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:34:41] (03update) 10fnegri: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:35:00] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [16:35:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.515-20260624163159-bc15c172 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354 https://phabricator.wikimedia.org/T429907) [16:35:23] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.515-20260624163159-bc15c172 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354 https://phabricator.wikimedia.org/T429907) [16:36:37] (03approved) 10dcaro: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:36:42] (03merge) 10dcaro: builds-api: bump to 0.0.217-20260624115416-b2b5f9d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1306 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:37:05] (03update) 10fnegri: jobs-api: bump to 0.0.515-20260624163159-bc15c172 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1305 (https://phabricator.wikimedia.org/T428354 https://phabricator.wikimedia.org/T429907) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:37:41] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [16:39:24] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T429907) [16:39:29] T429907: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907 [16:44:41] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] Refactor k8s object creation to have per-job-type functions - https://phabricator.wikimedia.org/T430071 (10dcaro) 03NEW [16:45:02] (03update) 10dcaro: k8s: use job_type instead of custom string for labels [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/321 [16:45:09] (03update) 10dcaro: k8s: when deleting a job, use it's own function [repos/cloud/toolforge/jobs-api] (use_job_type_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/322 [16:45:17] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [16:50:42] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) [16:50:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:51:13] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T429907) [16:51:17] T429907: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907 [17:00:53] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] Refactor k8s object creation to have per-job-type functions - https://phabricator.wikimedia.org/T430071#12051380 (10dcaro) 05Open→03In progress [17:00:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-api] Refactor k8s object creation to have per-job-type functions - https://phabricator.wikimedia.org/T430071#12051382 (10dcaro) 05In progress→03Open p:05Triage→03High [17:01:05] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) [17:01:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:03:11] (03update) 10dcaro: k8s: use per-job-type function to delete all jobs [repos/cloud/toolforge/jobs-api] (delete_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/325 [17:04:21] (03open) 10dcaro: k8s: move get k8s objects per job_type [repos/cloud/toolforge/jobs-api] (delete_all_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/326 [17:12:25] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) [17:12:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:15:12] 06tools-platform-team, 13Patch-For-Review: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907#12051455 (10fnegri) Deployed to toolsbeta and verified it works correctly there: ` lang=shell-session toolsbeta.admin@toolsbeta-bastion-7:~$ too... [17:29:54] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01) [17:29:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:38:03] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) [17:38:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:39:55] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117368498 (https://github.com/cluebotng/component-configs/commits/2a2ca3b7a9ed1a932fd98d3ae6694fc8b386db5d) [17:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:44:49] FIRING: DiskSpace: Disk space cloudbackup1004:9100:/srv 4.941% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:52:37] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) [17:52:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:53:56] 10VPS-project-Codesearch: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033#12051602 (10Dzahn) I created a second webproxy codesearch-backend and pointed it to http://172.16.21.36:3002 I think it works again now. [17:54:34] 10VPS-project-Codesearch: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033#12051605 (10Dzahn) The reason is probably that webproxies get deleted along with an instance they are assigned to and it took 24 (or 48?) hours to expire the old one? [17:56:34] 10VPS-project-Codesearch, 07Regression: Codesearch suddenly uses the default Hound UI - https://phabricator.wikimedia.org/T429967#12051614 (10Dzahn) I created a second webproxy "codesearch-backend" and pointed it to port 3002. I think it's working again now. So it was a DNS record but not in the DNS zone... [18:02:07] 10VPS-project-Codesearch, 06collaboration-services, 07Regression: Codesearch suddenly uses the default Hound UI - https://phabricator.wikimedia.org/T429967#12051627 (10Dzahn) [18:02:47] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b) [18:02:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:03:53] 10VPS-project-Codesearch, 06collaboration-services: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033#12051631 (10Dzahn) [18:08:45] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12051662 (10DamianZaremba) There is a similar issue with `components deployment list` where it exits with a failure code (2) when there are no depl... [18:10:32] 10VPS-project-Wikistats: Add isvwiki to wikistats - https://phabricator.wikimedia.org/T429940#12051679 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ('isv','Interslavic','medžuslovja... [18:13:01] 10Cloud-VPS (Project-requests): Request creation of wikimedia_commons_uploader_telegram_bot VPS project - https://phabricator.wikimedia.org/T429729#12051687 (10Bugreporter) >>! In T429729#12050754, @Pppery wrote: > You need someone in https://meta.wikimedia.org/wiki/Special:ListUsers?username=&group=oauthadmin&w... [18:13:02] (03CR) 10Dzahn: [C:03+2] "@dziewonski@fastmail.fm the repo has been added btw: https://codesearch.wmcloud.org/search/?q=+Charts+targeted+towards&files=&excludeFile" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1305224 (owner: 10Dzahn) [18:19:12] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06Data-Engineering (Q4 FS25/26 April 1st - June 30st), 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Check Editor Counts - https://phabricator.wikimedia.org/T427548#12051707 (10pmiazga) 05Open→03Resolved Ok, so I did some testin... [18:20:07] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12051711 (10Buxtang) a:03Buxtang [18:25:13] 10VPS-project-Codesearch, 06collaboration-services: Codesearch broken due to missing DNS record for codesearch-backend.wmcloud.org - https://phabricator.wikimedia.org/T430033#12051740 (10Dzahn) 05Open→03Resolved a:03Dzahn [18:48:06] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12051798 (10ayounsi) {T424871} are the switch replacement tracking task to support 25G. But even after they arrive some time will be needed to configure/au... [19:40:47] (03CR) 10Bartosz Dziewoński: "Cool, thanks!" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1305224 (owner: 10Dzahn) [20:32:24] 10Tool-bash: Bash: Slime application error on login attempt - https://phabricator.wikimedia.org/T430088#12051978 (10Peachey88) [20:49:30] 10Tool-global-search: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092 (10cscott) 03NEW [20:55:07] 10Tool-global-search: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092#12052070 (10cscott) On IRC it was said: > if its a oauth1 user, it appears there was a bit of a surprise breaking change > bawolfs comment in -tech So maybe that's a hint? [21:04:34] RESOLVED: DiskSpace: Disk space cloudbackup1004:9100:/srv 6.724% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:05:45] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12052087 (10Volans) As I wasn't sure that the current backup run could finish without filling the whole partition I've manually forced the execution of `/usr/local/sbin/w... [21:10:22] 10Tool-global-search: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092#12052096 (10cscott) global search uses wikimedia/toolforge-bundle which uses mediawiki/oauthclient. And mediawiki/oauthclient uses format=json wouldn't you know. [21:13:08] 10Tool-global-search: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092#12052098 (10Bawolff) Looks like it is in the dependency https://github.com/wikimedia/mediawiki-oauthclient-php/blob/master/src/Client.php#L158 [21:23:03] 10Cloud-VPS, 06tools-infrastructure-team: Rebalance WMCS instance backups between the two hosts - https://phabricator.wikimedia.org/T430018#12052115 (10Volans) The cleanup has completed and now we're at: ` $ df -hT /srv/ Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg0-srv ext4 15T... [21:48:23] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092#12052160 (10cscott) [21:51:44] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in - https://phabricator.wikimedia.org/T430092#12052163 (10cscott) p:05Triage→03Unbreak! Elevating priority because this breaks 3rd party login (including wmf cloud login) to Wikimedia servers for all downstream apps using t... [21:54:36] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauth clients broken) - https://phabricator.wikimedia.org/T430092#12052169 (10cscott) [21:54:58] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052170 (10cscott) [22:37:55] 06cloud-services-team, 06tools-infrastructure-team: Patch demo resource check - https://phabricator.wikimedia.org/T429990#12052259 (10BLiviero-WMF) [22:45:00] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052261 (10Tgr) a:03Tgr [22:46:12] 10Tool-global-search, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052263 (10Tgr) Broken in {fda4f89e49452f387dd3d9683c1b2d113b6e3f0b} Hopefully fixed in {f1d042007df1b5c29044e8019a8cc1093864f2ed} and {b1... [22:59:00] 10Tool-global-search, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052269 (10Tgr) [23:28:20] 10Tool-global-search, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052309 (10Tgr) 05Open→03Resolved [23:28:50] 10Tool-global-search, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052311 (10Tgr) Sorry for causing trouble. [23:32:53] 10Tool-global-search, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Wikimedia Global Search won't let me log in (all mediawiki/oauthclient users broken) - https://phabricator.wikimedia.org/T430092#12052314 (10brennen) Thanks for fix!