[01:24:34] FIRING: [2x] ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesse [04:23:34] (03close) 10bodhisattwa: Add uk.json [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/2 [05:51:39] (03merge) 10mahir256: add de from Lydia [toolforge-repos/sangkalak] - 10https://gitlab.wikimedia.org/toolforge-repos/sangkalak/-/merge_requests/4 (owner: 10bodhisattwa) [06:31:30] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.87%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [07:13:16] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091 (10taavi) 03NEW [07:17:10] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056636 (10taavi) [07:19:03] 06cloud-services-team, 10Cloud-VPS: Use cloud-private network and cfssl certs for instance live migrations - https://phabricator.wikimedia.org/T355145#11056637 (10taavi) [07:28:38] (03merge) 10taavi: Migrate Toolforge deployment to components [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/28 (https://phabricator.wikimedia.org/T397929) [07:35:14] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056645 (10fgiunchedi) [07:42:24] 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T401076#11056650 (10taavi) [07:58:55] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11056660 (10dcaro) [08:08:47] 10PAWS, 10OpenRefine, 10Wikidata: OpenRefine on PAWS, cannot login to Wikidata - https://phabricator.wikimedia.org/T401092 (10Jklamo) 03NEW [08:15:54] (03update) 10dcaro: cli: only send fields that are set [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/112 [08:22:47] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056685 (10joanna_borun) [08:43:01] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11056724 (10dcaro) I think that you should not bind yourself to the execution engine details here, instead of relying on k8s setting the ready or not ready statu... [08:53:11] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056728 (10fnegri) [08:53:21] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056729 (10fnegri) [08:59:09] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [09:04:26] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11056760 (10dcaro) C/C++ support in the buildpack world is quite overlooked yep, I fear that if we want to have support for it we will have to build... [09:07:16] (03approved) 10taavi: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014) (owner: 10bd808) [09:17:02] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [09:19:10] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Add NEL headers to Cloud VPS and Toolforge managed proxies - https://phabricator.wikimedia.org/T400994#11056795 (10taavi) a:03taavi [09:19:55] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11056800 (10Alien333) On the hathi credentials: these are for a terminated api (https://babel.hathitrust.org/cgi/htd/); so sharing them is not much risk. You can remove them if you want. [09:20:53] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [09:34:38] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056858 (10taavi) [09:35:39] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11056859 (10taavi) [09:40:01] 06cloud-services-team, 10Cloud-VPS: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11056875 (10taavi) Is the new proxy meant to be temporary or will it exist permanently? [09:41:16] (03update) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe) [09:43:30] 06cloud-services-team, 10Toolforge: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024#11056898 (10taavi) p:05Triage→03High a:03taavi [09:44:01] (03approved) 10dcaro: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) (owner: 10raymond-ndibe) [09:46:15] 06cloud-services-team, 10Toolforge: Prebuilt webservice containers should not log to NFS by default - https://phabricator.wikimedia.org/T401102 (10taavi) 03NEW p:05Triage→03Medium [09:47:41] 06cloud-services-team, 10Toolforge: Prebuilt webservice containers should not log to NFS by default - https://phabricator.wikimedia.org/T401102#11056932 (10taavi) [09:47:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11056933 (10taavi) [10:04:35] (03update) 10taavi: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) (owner: 10fnegri) [10:07:15] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024#11056989 (10taavi) I didn't catch this while testing, but the above patch doesn't quite work: now the URL is `http://tools-static.wmflabs.o... [10:13:18] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [10:15:01] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024#11056998 (10taavi) 05Open→03Resolved [10:16:25] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [10:38:41] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [11:11:10] (03CR) 10Stevemunene: [C:03+2] Add keytabs for new an-druid100[67] hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1171214 (https://phabricator.wikimedia.org/T397440) (owner: 10Stevemunene) [11:11:24] (03CR) 10Stevemunene: [V:03+2 C:03+2] Add keytabs for new an-druid100[67] hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1171214 (https://phabricator.wikimedia.org/T397440) (owner: 10Stevemunene) [11:33:22] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [11:44:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 [11:47:51] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [11:48:53] (03merge) 10dcaro: runtime: do the diff at the core.models.Job level [repos/cloud/toolforge/jobs-api] (fix_diff_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [11:48:56] (03update) 10dcaro: [runtimes.k8s.runtime] fix bug in diff_with_running_job method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/169 (https://phabricator.wikimedia.org/T394734) (owner: 10raymond-ndibe) [11:56:09] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 [12:07:10] (03update) 10dcaro: [runtimes.k8s.runtime] fix bug in diff_with_running_job method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/169 (https://phabricator.wikimedia.org/T394734) (owner: 10raymond-ndibe) [12:08:23] 06cloud-services-team, 10Toolforge: Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110 (10DamianZaremba) 03NEW [12:12:41] (03approved) 10dcaro: [runtimes.k8s.runtime] fix bug in diff_with_running_job method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/169 (https://phabricator.wikimedia.org/T394734) (owner: 10raymond-ndibe) [12:13:02] (03merge) 10dcaro: [runtimes.k8s.runtime] fix bug in diff_with_running_job method [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/169 (https://phabricator.wikimedia.org/T394734) (owner: 10raymond-ndibe) [12:13:05] (03update) 10dcaro: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) (owner: 10raymond-ndibe) [12:13:34] (03update) 10dcaro: cli: only send fields that are set [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/112 [12:15:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.390-20250804121313-fee7fdea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/892 (https://phabricator.wikimedia.org/T394734) [12:17:31] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11057297 (10fgiunchedi) [12:23:19] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11057330 (10taavi) [12:23:20] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11057326 (10DamianZaremba) The application does expose it's readiness (by `/tmp/container_ready` being created), the issue is how to expose that (given that the... [12:27:33] (03update) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/42 [12:31:55] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11057353 (10DamianZaremba) I did experiment with that sort of concept using a "Python" setup with poetry running a custom "setup.py" which dealt wit... [12:45:51] (03approved) 10fnegri: webservice: fix command [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/78 (owner: 10dcaro) [12:47:05] (03approved) 10dcaro: webservice: fix command [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/78 [12:47:05] (03update) 10dcaro: webservice: fix command [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/78 [12:47:08] (03merge) 10dcaro: webservice: fix command [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/78 [12:47:23] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/79 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:49:53] 06cloud-services-team, 10Toolforge: [jobs-api] Migrate to FastAPI - https://phabricator.wikimedia.org/T401113 (10taavi) 03NEW [12:49:59] 06cloud-services-team, 10Toolforge: [jobs-api] Migrate to FastAPI - https://phabricator.wikimedia.org/T401113#11057395 (10taavi) p:05Triage→03Medium [12:50:22] 06cloud-services-team, 10Toolforge: [jobs-api] Migrate to FastAPI - https://phabricator.wikimedia.org/T401113#11057396 (10taavi) [12:50:26] 06cloud-services-team, 10Toolforge: [jobs-api] Support following logs from Loki - https://phabricator.wikimedia.org/T400916#11057397 (10taavi) [12:54:34] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-2 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [12:56:10] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11057402 (10fgiunchedi) [13:04:53] !log filippo@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'reader' (T401091) [13:04:56] !log filippo@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'reader' (T401091) [13:04:57] T401091: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091 [13:05:11] !log filippo@cloudcumin1001 cloudinfra START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091) [13:05:16] !log filippo@cloudcumin1001 cloudinfra END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091) [13:05:27] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091) [13:05:33] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091) [13:05:42] !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091) [13:05:47] !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091) [13:05:53] !log filippo@cloudcumin1001 paws START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091) [13:05:58] !log filippo@cloudcumin1001 paws END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091) [13:06:07] !log filippo@cloudcumin1001 metricsinfra START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091) [13:06:11] !log filippo@cloudcumin1001 metricsinfra END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091) [13:06:51] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11057433 (10fgiunchedi) [13:11:46] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/79 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:11:53] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [13:11:54] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/79 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:18:09] (03update) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/77 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [13:18:43] (03update) 10dcaro: d/changelog: bump to 0.103.16 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/77 (https://phabricator.wikimedia.org/T360488 https://phabricator.wikimedia.org/T384788) [13:21:55] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11057467 (10dcaro) Just fyi. the logic behind the follow option will be changed relatively soon (from k8s to directly using loki), so the implementation details might be q... [13:22:38] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11057473 (10dcaro) How critical is this feature for you? (if it's very critical, we might want to prioritize implementing it before the loki support for streaming). [13:27:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:37:37] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:38:57] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:53:37] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:55:50] (03open) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 [13:56:11] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11057589 (10dcaro) [13:56:41] (03open) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) [13:58:34] 06cloud-services-team: Onboard Filippo as SRE in Cloud Services - https://phabricator.wikimedia.org/T401091#11057596 (10taavi) [14:01:20] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [components-api] store the config used for the deployment in the deployment themselves - https://phabricator.wikimedia.org/T400064#11057604 (10dcaro) [14:04:35] (03merge) 10dcaro: jobs-api: bump to 0.0.390-20250804121313-fee7fdea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/892 (https://phabricator.wikimedia.org/T394734) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:07:46] !log andrew@cloudcumin1001 paws START - Cookbook wmcs.openstack.quota_increase [14:07:53] !log andrew@cloudcumin1001 paws END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [14:08:08] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [14:11:30] RESOLVED: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.13%) at host paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:11:56] FIRING: PawsNFSDown: No paws nfs services running found - https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsNFSDown [14:11:59] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [14:12:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [14:13:28] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:15:01] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11057669 (10DamianZaremba) I have a workaround in place (checking the container is running via the kubernetes api) for now, so can wait until the loki change lands. [14:18:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:30:58] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [14:31:30] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on toolsbeta-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [14:33:25] RESOLVED: PawsNFSDown: No paws nfs services running found - https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsNFSDown [14:35:26] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [14:41:26] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:49:28] FIRING: InstanceDown: Project tools instance tools-prometheus-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:59:28] RESOLVED: InstanceDown: Project tools instance tools-prometheus-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:08:12] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [15:14:43] (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) [15:19:01] (03update) 10taavi: Draft: Replace Flask with FastAPI [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/192 (https://phabricator.wikimedia.org/T401113) [15:26:43] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [15:32:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 22), 07Kubernetes: Unable to load Toolforge job: ERROR: TjfCliError: Unknown error (403 Client Error: Forbidden for url - https://phabricator.wikimedia.org/T399417#11058008 (10dcaro) 05In progress→03Resolved Closing this as https://gitlab.wikime... [15:32:32] (03update) 10fnegri: Relicense under Apache 2.0 [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/2 [15:45:43] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [15:48:44] 10Toolforge (Toolforge iteration 22), 13Patch-For-Review: [jobs-api] bug in runtime diff_with_running_job function - https://phabricator.wikimedia.org/T394734#11058062 (10dcaro) 05In progress→03Resolved Deployed \o/ [16:08:08] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11058142 (10dcaro) Thanks for the explanation! A couple notes: * If you are just grabbing the logs you don't need to `--follow`, you can poll the `run-edit-set... [16:13:43] (03open) 10anzx: Add new file - Kannada(kn) Language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 [16:14:32] (03update) 10anzx: Add new file - Kannada(kn) Language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 [16:16:20] (03update) 10anzx: Draft: Add new file - Kannada(kn) Language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 [16:17:32] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [16:18:05] (03update) 10anzx: Draft: Add new file - Kannada(kn) Language [toolforge-repos/wstranclude] - 10https://gitlab.wikimedia.org/toolforge-repos/wstranclude/-/merge_requests/1 [16:27:09] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [16:29:42] (03update) 10dcaro: api: Allow querying logs for non-existent jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T400913) (owner: 10taavi) [16:37:26] (03update) 10fnegri: Create .deb package [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/1 (https://phabricator.wikimedia.org/T395266) [16:37:28] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11058249 (10DamianZaremba) My preference would be to "stream" the logs back to the parent, currently when debugging I `kubectl logs` directly against the child s... [16:39:48] FIRING: PuppetFailure: Puppet has failed on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:39:59] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130 (10phaultfinder) 03NEW [16:45:50] (03update) 10vriaa: Draft: Basic banner implementation [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/1 [16:47:35] (03approved) 10dcaro: api: Allow querying logs for non-existent jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T400913) (owner: 10taavi) [16:52:28] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11058308 (10fnegri) The error message is ` /Stage[main]/Dumps::Web::Dumpstatusfiles/File[/usr/local/bin/unpack-dumpstatusfiles.sh] Could not evaluate: Could not retrieve information from en... [16:54:48] FIRING: [2x] PuppetFailure: Puppet has failed on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:55:00] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T401133 (10phaultfinder) 03NEW [16:56:14] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11058330 (10fnegri) > 18:54 that file is not there https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/c3ad9dd1dff81dfec21d3b0ce3fd486d982ce6ad/modules/dumps/files/web/... [16:58:52] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11058335 (10fnegri) 05Open→03In progress [16:59:01] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11058336 (10fnegri) p:05Triage→03Medium [17:00:09] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T401133#11058340 (10fnegri) [17:00:11] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T401130#11058341 (10fnegri) [17:00:50] 06cloud-services-team, 10Cloud-VPS: Request to enable XFF headers for test XTools hostnames - https://phabricator.wikimedia.org/T400964#11058343 (10MusikAnimal) >>! In T400964#11056874, @taavi wrote: > Is the new proxy meant to be temporary or will it exist permanently? I'm asking to keep it permanently, if t... [17:09:53] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11058376 (10bd808) {T363033} would be one way to work out what is needed to build cluebotng-core. [17:29:29] 06cloud-services-team, 10Toolforge: toolforge-static gives 301 redirects to backend server with port 8000 - https://phabricator.wikimedia.org/T401024#11058445 (10bd808) 05Resolved→03Open https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175483/1/modules/profile/templates/toolforge/static/nginx.conf.erb... [17:29:38] (03update) 10raymond-ndibe: Draft: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) [17:30:20] (03update) 10raymond-ndibe: Draft: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) [17:30:25] (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) [17:30:26] (03PS1) 10Andrew Bogott: Added dump capiservicek3s.yaml file [labs/private] - 10https://gerrit.wikimedia.org/r/1175557 (https://phabricator.wikimedia.org/T393782) [17:31:33] (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [17:44:09] (03PS2) 10Andrew Bogott: Added dummy capiservicek3s.yaml file [labs/private] - 10https://gerrit.wikimedia.org/r/1175557 (https://phabricator.wikimedia.org/T393782) [17:44:16] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Added dummy capiservicek3s.yaml file [labs/private] - 10https://gerrit.wikimedia.org/r/1175557 (https://phabricator.wikimedia.org/T393782) (owner: 10Andrew Bogott) [17:44:44] (03update) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014) [17:51:42] (03update) 10bd808: lighttpd: Use "lighttpd" as webservice type [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/80 (https://phabricator.wikimedia.org/T401014) [17:52:30] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11058518 (10MusikAnimal) Thanks @LucasWerkmeister :) I have examined the tool more closely and don't see any problems. After talking with Alien333, we concluded the Hathi credentials aren't needed at all... [17:53:20] (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [17:54:32] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11058519 (10DamianZaremba) I wasn't really planning on doing this today... but after some playing around https://github.com/InfraBits/cmake-buildpac... [18:00:00] (03update) 10raymond-ndibe: [tests] account for warning messages printed to stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/883 (https://phabricator.wikimedia.org/T400390) [18:04:36] 10Tools: Make hatjitsu SD calculation more robust - https://phabricator.wikimedia.org/T401141 (10TJones) 03NEW [18:05:49] (03update) 10raymond-ndibe: [deployment] add config to deployment [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/112 (https://phabricator.wikimedia.org/T400064) [18:06:21] 06cloud-services-team, 10Toolforge, 06Toolforge-standards-committee: Keep track of tools without stated default licenses - https://phabricator.wikimedia.org/T190377#11058598 (10bd808) >>! In T190377#11055957, @Pintoch wrote: > The [[ https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Tool_sweep | Tool swee... [18:25:41] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11058656 (10DamianZaremba) And building what I actually need to build; Source: https://github.com/cluebotng/core/tree/feature/support-running-in-pac... [18:44:20] (03update) 10raymond-ndibe: [maintain-harbor] add tests and configurations for new maintain-harbor jobs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/881 (https://phabricator.wikimedia.org/T360509) [18:48:27] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [19:49:29] (03update) 10raymond-ndibe: api: allow protocol to be specified for ports [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/186 (owner: 10dcaro) [19:50:29] (03update) 10raymond-ndibe: api: allow protocol to be specified for ports [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/186 (owner: 10dcaro) [19:59:12] 06cloud-services-team, 10Toolforge, 06Toolforge-standards-committee: Keep track of tools without stated default licenses - https://phabricator.wikimedia.org/T190377#11058937 (10AntiCompositeNumber) That Toolhub link is probably a significant overcount due to tool authors not writing complete toolinfo.json re... [20:11:31] 06cloud-services-team, 10Toolforge, 06Toolforge-standards-committee: Keep track of tools without stated default licenses - https://phabricator.wikimedia.org/T190377#11058958 (10bd808) >>! In T190377#11058937, @AntiCompositeNumber wrote: > Toolhub doesn't appear to support adding a license not represented in... [20:12:24] (03update) 10raymond-ndibe: [cli] Change port type to allow protocol suffix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/115 (https://phabricator.wikimedia.org/T400024) [20:13:05] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11058963 (10DamianZaremba) 05Open→03Declined [20:13:14] (03update) 10raymond-ndibe: [cli] Change port type to allow protocol suffix [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/115 (https://phabricator.wikimedia.org/T400024) [20:13:42] 06cloud-services-team, 10Toolforge: Allow health checks (readinessProbe) to be specified for one-off jobs - https://phabricator.wikimedia.org/T401071#11058964 (10DamianZaremba) Closing in favour of T401069 [20:17:57] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11058989 (10DamianZaremba) T400913 would also be fine for my use case. I'll leave this open as a record of the current behaviour but I'm happy with it not being fixed [20:18:26] 06cloud-services-team, 10Toolforge: `toolforge jobs logs` returns nothing if started too early. - https://phabricator.wikimedia.org/T401073#11058996 (10DamianZaremba) p:05Triage→03Low [20:20:04] 06cloud-services-team, 10Toolforge: Support `--timeout` for one-off jobs - https://phabricator.wikimedia.org/T401110#11058999 (10DamianZaremba) I've wrapped my calls in gnu `timeout` for now but I believe this would be valuable more generally and also allow cleaner status reporting regarding timeout vs exit code [20:25:19] 06cloud-services-team, 10Toolforge: Toolforge jobs api reports status as running before container is launched - https://phabricator.wikimedia.org/T401069#11059019 (10DamianZaremba) Playing this this a bit more, you can sort of get the container status by parsing the long status string. Maybe the solution here... [20:31:27] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11059050 (10Alien333) Thanks! Could I also get rights on [[ https://gitlab.wikimedia.org/toolforge-repos/pagelister | the gitlab repo ]]? To keep it up to date. [20:31:31] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059055 (10DamianZaremba) >>! In T401075#11058376, @bd808 wrote: > {T363033} would be one way to work out what is needed to build cluebotng-core.... [20:39:20] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059083 (10bd808) >>! In T401075#11059055, @DamianZaremba wrote: >>>! In T401075#11058376, @bd808 wrote: >> {T363033} would be one way to work out... [20:52:40] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059135 (10bd808) > Upstream docs imply that something along the lines of We still have heroku/deb-packages 0.1.3 when using `--use-latest-version... [20:55:03] FIRING: [2x] PuppetFailure: Puppet has failed on clouddumps1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:58:22] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11059167 (10LucasWerkmeister) Ideally yes, but I don’t have permissions to add you :/ maybe ask in [IRC](https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication)? [20:58:28] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11059171 (10bd808) >>! In T398111#11059050, @Alien333 wrote: > Thanks! Could I also get rights on [[ https://gitlab.wikimedia.org/toolforge-repos/pagelister | the gitlab repo ]]? To keep it up to date. {... [20:58:59] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11059175 (10LucasWerkmeister) Thanks @bd808! [21:08:08] 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151 (10DamianZaremba) 03NEW [21:09:32] 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11059225 (10DamianZaremba) Adding taavi as they seem to be doing a lot of work on logging/loki [21:11:38] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059244 (10DamianZaremba) >>! In T401075#11059083, @bd808 wrote: >>>! In T401075#11059055, @DamianZaremba wrote: >>>>! In T401075#11058376, @bd808... [21:14:10] 06cloud-services-team, 10Toolforge: Support installing packages from non-upstream repo and/or build pack for C/C++code - https://phabricator.wikimedia.org/T401075#11059255 (10DamianZaremba) >>! In T401075#11059135, @bd808 wrote: >> Upstream docs imply that something along the lines of > > We still have heroku... [21:27:10] !log andrew@cloudcumin1001 magnum START - Cookbook wmcs.vps.create_project for project magnum in eqiad1 [21:27:12] andrew@cloudcumin1001: Unknown project "magnum" [21:27:47] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/259 [21:29:19] (03merge) 10andrew: projects: added project magnum [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/259 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [21:31:47] !log andrew@cloudcumin1001 magnum END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project magnum in eqiad1 [21:31:47] andrew@cloudcumin1001: Unknown project "magnum" [21:51:51] (03open) 10eliza189: Eliza http errors [toolforge-repos/miss-search] (update-cycle-toolforge-testing) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/11 [22:36:02] 06Toolforge-standards-committee: Adoption request for pagelister - https://phabricator.wikimedia.org/T398111#11059416 (10Alien333) Thanks! All good here, as far as I'm concerned. [23:14:48] FIRING: [3x] PuppetFailure: Puppet has failed on cloudcontrol1011:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:14:58] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T401133#11059426 (10phaultfinder) [23:19:48] FIRING: [5x] PuppetFailure: Puppet has failed on cloudcontrol1007:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [23:19:58] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T401133#11059427 (10phaultfinder)