[00:09:06] andrew@cloudcumin1001 safe_reboot (PID 485336) is awaiting input [00:17:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' [00:30:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [00:36:32] andrew@cloudcumin1001 safe_reboot (PID 496565) is awaiting input [00:44:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [01:02:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:02:31] 06cloud-services-team: JobUnavailable Reduced availability for job rabbitmq in cloud@codfw - https://phabricator.wikimedia.org/T406089 (10phaultfinder) 03NEW [01:06:56] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [01:07:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:09:50] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [01:11:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [01:17:24] andrew@cloudcumin1001 safe_reboot (PID 499595) is awaiting input [01:37:37] andrew@cloudcumin1001 safe_reboot (PID 499595) is awaiting input [01:39:09] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [01:47:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' [01:51:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2005-dev.codfw.wmnet}' [01:52:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [02:04:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt2004-dev.codfw.wmnet}' [05:30:35] (03open) 10samwilson: Migrate CI from GitHub to GitLab [toolforge-repos/wscontest] - 10https://gitlab.wikimedia.org/toolforge-repos/wscontest/-/merge_requests/1 (https://phabricator.wikimedia.org/T324410) [05:32:39] (03update) 10samwilson: Migrate CI from GitHub to GitLab [toolforge-repos/wscontest] - 10https://gitlab.wikimedia.org/toolforge-repos/wscontest/-/merge_requests/1 (https://phabricator.wikimedia.org/T324410) [06:34:10] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/62 [06:34:19] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/62 [06:34:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/136 [06:34:31] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/136 [06:34:38] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/135 [06:34:42] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/135 [06:34:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/77 [06:35:04] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/77 [06:35:13] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/30 [06:35:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/30 [06:35:32] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/137 [06:35:33] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/137 [06:35:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/26 [06:35:37] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/26 [06:35:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/31 [06:35:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/121 [06:35:54] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/31 [06:35:58] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/32 [06:36:06] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/87 [06:36:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/32 [06:36:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/121 [06:36:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/223 [06:36:34] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/224 [06:36:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/93 [06:36:41] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/94 [06:36:54] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/87 [06:36:59] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/26 [06:37:01] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/79 [06:37:04] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/223 [06:37:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/53 [06:37:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/37 [06:37:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/224 [06:37:14] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/65 [06:37:16] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/93 [06:37:16] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/146 [06:37:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/94 [06:37:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/80 [06:37:31] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/74 [06:37:35] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/26 [06:37:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/276 [06:37:43] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/79 [06:37:47] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/37 [06:37:54] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/53 [06:37:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/53 [06:38:02] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/54 [06:38:06] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/65 [06:38:10] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/146 [06:38:14] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/80 [06:38:22] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/74 [06:38:26] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/276 [06:38:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/53 [06:38:34] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/54 [06:46:30] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce) [06:46:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [06:58:10] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11232047 (10A_smart_kitten) >>! In T388022#11231303, @Dzahn wrote: > I don't mind if you do that. That would give us an idea how common it i... [07:09:14] 10tool-wscontest, 13Patch-For-Review: Move wscontest source code to GitLab - https://phabricator.wikimedia.org/T324410#11232078 (10Samwilson) [07:11:10] 10tool-wscontest, 10GitLab (Project Migration), 13Patch-For-Review: Move wscontest source code to GitLab - https://phabricator.wikimedia.org/T324410#11232081 (10A_smart_kitten) [07:34:18] (03merge) 10samwilson: Migrate CI from GitHub to GitLab [toolforge-repos/wscontest] - 10https://gitlab.wikimedia.org/toolforge-repos/wscontest/-/merge_requests/1 (https://phabricator.wikimedia.org/T324410) [08:56:05] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:00:40] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:02:49] (03merge) 10dcaro: build: return error when the given ref is not resolvable [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/145 [09:03:00] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/146 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:12:04] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.200-20251001090302-d6fa5285 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/984 (https://phabricator.wikimedia.org/T405829) [09:12:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.200-20251001090302-d6fa5285 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/984 (https://phabricator.wikimedia.org/T405829) [09:33:20] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:42:36] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [09:48:16] 06cloud-services-team, 10Toolforge: Rate limiting errors should not trigger ToolforgeWebHighErrorRate - https://phabricator.wikimedia.org/T406010#11232566 (10taavi) The HAProxy startup log includes this highly related warning: ` Sep 29 07:41:48 toolsbeta-test-k8s-haproxy-5 haproxy[512020]: [WARNING] (512020)... [09:49:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:55:32] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [10:10:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [10:16:04] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [10:18:24] (03approved) 10dcaro: builds-api: bump to 0.0.200-20251001090302-d6fa5285 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/984 (https://phabricator.wikimedia.org/T405829) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:18:29] (03merge) 10dcaro: builds-api: bump to 0.0.200-20251001090302-d6fa5285 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/984 (https://phabricator.wikimedia.org/T405829) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:19:02] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-api] does not correctly resolve `ref` - builds random things - https://phabricator.wikimedia.org/T405829#11232677 (10dcaro) 05In progress→03Resolved [10:28:49] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [10:35:02] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [11:00:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [builds-api] does not correctly resolve `ref` - builds random things - https://phabricator.wikimedia.org/T405829#11232743 (10DamianZaremba) Thanks [13:09:50] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124 (10itsmoon) 03NEW [13:17:12] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11233132 (10taavi) Fwiw, while debugging an unrelated issue I spotted this in the Cloud VPS outbound mail server logs that looks highly rele... [13:51:21] 06cloud-services-team, 10Cloud-VPS: unable to "apt install helmfile" on CloudVPS debian 13 vm - https://phabricator.wikimedia.org/T405970#11233308 (10taavi) 05Open→03Resolved a:03Andrew [13:51:31] 06cloud-services-team, 10Toolforge: Rate limiting errors should not trigger ToolforgeWebHighErrorRate - https://phabricator.wikimedia.org/T406010#11233312 (10taavi) p:05Triage→03High [13:52:20] 06cloud-services-team: JobUnavailable Reduced availability for job rabbitmq in cloud@codfw - https://phabricator.wikimedia.org/T406089#11233321 (10taavi) 05Open→03Resolved this has resolved itself [13:52:35] 06cloud-services-team: JobUnavailable Reduced availability for job openstack in cloud@eqiad - https://phabricator.wikimedia.org/T405851#11233324 (10taavi) 05Open→03Resolved since resolved [13:52:59] 06cloud-services-team: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T405740#11233327 (10taavi) 05Open→03Resolved a:03taavi [13:54:34] FIRING: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 6.986% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:03:21] 06cloud-services-team: Cloud VPS requests review - https://phabricator.wikimedia.org/T405864#11233365 (10taavi) p:05Triage→03High [14:03:53] 06cloud-services-team: Cloud VPS requests review - https://phabricator.wikimedia.org/T405864#11233367 (10taavi) [14:04:22] 06cloud-services-team, 10Toolforge: When a job runs out of memory, no message in the error file can be seen - https://phabricator.wikimedia.org/T405854#11233371 (10taavi) p:05Triage→03Low [14:04:34] RESOLVED: DiskSpace: Disk space cloudcontrol2006-dev:9100:/ 6.965% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudcontrol2006-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:05:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 07Regression: Validation error for CommonJob: file logging is only available with --mount=all - https://phabricator.wikimedia.org/T405828#11233376 (10taavi) p:05Triage→03High [14:06:39] 06cloud-services-team, 10Toolforge: [jobs-cli] `job_type` not handled in `dump`, generates warning per job - https://phabricator.wikimedia.org/T405786#11233391 (10taavi) p:05Triage→03Medium [14:07:06] 06cloud-services-team, 10Tool-spacemedia, 10Toolforge: [Build service] latest builder has old Java - https://phabricator.wikimedia.org/T405415#11233395 (10taavi) p:05Triage→03Low [14:07:25] 06cloud-services-team, 10Cloud-VPS, 10Ceph: cloudceph: automate nic names in profile::cloudceph::osd::hosts - https://phabricator.wikimedia.org/T405374#11233397 (10taavi) p:05Triage→03Low [14:08:10] 06cloud-services-team, 10Toolforge: Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#11233401 (10taavi) p:05Triage→03High [14:08:37] 06cloud-services-team, 10Toolforge: [jobs-api] allow port configuration on non-continuous jobs - https://phabricator.wikimedia.org/T405036#11233415 (10taavi) p:05Triage→03Low [14:09:25] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [envvars] ease revealing a secret - https://phabricator.wikimedia.org/T405024#11233417 (10taavi) p:05Triage→03Low [14:10:10] 06cloud-services-team, 10Toolforge: [envvars] scope to jobs/components - https://phabricator.wikimedia.org/T405022#11233434 (10taavi) p:05Triage→03Medium [14:11:16] 06cloud-services-team, 10Toolforge: Allow to remap/rename secrets from envvars when running specific toolforge job - https://phabricator.wikimedia.org/T376849#11233436 (10taavi) 05Open→03Declined We discussed this and agreed that {T405022} should be implemented instead. [14:12:28] 06cloud-services-team, 10Toolforge: [envvars] only mask secrets - https://phabricator.wikimedia.org/T405018#11233444 (10taavi) p:05Triage→03Low [14:13:48] 06cloud-services-team, 10Toolforge: [jobs-api] non-existant job not complying to kubernete's naming returns 400 error - https://phabricator.wikimedia.org/T404852#11233453 (10taavi) p:05Triage→03Low [14:15:27] 06cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 06DC-Ops, 06SRE, 13Patch-For-Review: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478#11233469 (10taavi) p:05Triage→03Medium [14:41:46] (03open) 10taavi: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) [14:41:48] (03update) 10taavi: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) [14:43:42] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Rate limiting errors should not trigger ToolforgeWebHighErrorRate - https://phabricator.wikimedia.org/T406010#11233554 (10taavi) a:03taavi [14:51:27] (03update) 10taavi: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) [14:51:29] (03update) 10taavi: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) [14:59:35] (03approved) 10dcaro: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) (owner: 10taavi) [14:59:57] (03merge) 10taavi: kubernetes: Fixup web error rate alert metric [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/42 (https://phabricator.wikimedia.org/T406010) [15:00:22] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Rate limiting errors should not trigger ToolforgeWebHighErrorRate - https://phabricator.wikimedia.org/T406010#11233600 (10taavi) 05Open→03Resolved [15:07:25] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [15:07:54] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/54 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:07:57] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/54 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:08:37] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/53 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:08:41] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/53 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:10:06] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/53 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:10:25] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/276 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:10:48] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/276 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:10:56] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/94 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:10:59] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/94 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:11:27] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/93 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:11:51] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/93 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:02] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:09] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/87 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:17] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/121 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:22] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/121 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:27] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/135 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:30] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/135 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:35] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:39] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:13:30] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/93 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:13:38] (03update) 10dcaro: logs: use logs-api for logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/121 [15:23:14] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11233681 (10Pppery) How hard would it be to support sending mail from phabricator.wmcloud.org on the cloud-vps side? That seems like it woul... [16:01:03] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [16:02:03] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [16:29:22] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11234003 (10Dzahn) >>! In T388022#11231344, @Pppery wrote: > It hasn't been fixed. I asked phabricator.wmcloud.org to resend me a verificati... [16:32:32] 10VPS-project-Phabricator, 06collaboration-services: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11234012 (10taavi) >>! In T388022#11234003, @Dzahn wrote: > Also I would like to add some general comments about this ticket. We do like it... [16:33:32] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124#11234020 (10Dzahn) Hey @itsmoon when trying to verify I got: `Usage Exception: No email exists with address..` I used the address I see on https://l... [16:48:02] 06cloud-services-team, 10Openstack-Magnum: ssh to cloud-vps 'utility' nodes (magnum, trove, octavia) - https://phabricator.wikimedia.org/T402317#11234133 (10Andrew) The only debian-based nodes that don't already have cumin keys are potential 'unmanaged user VMs' using Debian. With the current setup, users wou... [17:14:25] (03update) 10dcaro: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [17:27:33] (03CR) 10Hashar: Fix typo in success messages for sending labels and descriptions (031 comment) [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1189385 (https://phabricator.wikimedia.org/T201491) (owner: 10Harsh_Kushwaha07) [17:27:37] (03PS2) 10Harsh_Kushwaha07: Fix typo in success messages for sending labels and descriptions [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1189385 (https://phabricator.wikimedia.org/T201491) [17:28:58] (03CR) 10Hashar: [C:03+1] "@murbanec@wikimedia.org is weapon-of-mass-description still of any use?" [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1189385 (https://phabricator.wikimedia.org/T201491) (owner: 10Harsh_Kushwaha07) [17:34:15] (03CR) 10Harsh_Kushwaha07: [C:03+1] Fix typo in success messages for sending labels and descriptions (031 comment) [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1189385 (https://phabricator.wikimedia.org/T201491) (owner: 10Harsh_Kushwaha07) [17:59:26] (03PS1) 10Majavah: Fix mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192943 [17:59:26] (03PS1) 10Majavah: t3: Ignore oathauth-twofactorauth group [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192944 [17:59:29] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124#11234537 (10A_smart_kitten) (I'm guessing that the email address can probably be looked up from the test instance database directly in a similar way to... [18:01:24] (03CR) 10CI reject: [V:04-1] Fix mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192943 (owner: 10Majavah) [18:01:36] (03CR) 10CI reject: [V:04-1] t3: Ignore oathauth-twofactorauth group [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192944 (owner: 10Majavah) [18:02:30] (03PS2) 10Majavah: Fix mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192943 [18:02:30] (03PS2) 10Majavah: t3: Ignore oathauth-twofactorauth group [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192944 [18:04:40] (03CR) 10Majavah: [C:03+2] Fix mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192943 (owner: 10Majavah) [18:04:44] (03CR) 10Majavah: [C:03+2] t3: Ignore oathauth-twofactorauth group [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192944 (owner: 10Majavah) [18:06:42] (03Merged) 10jenkins-bot: Fix mypy errors [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192943 (owner: 10Majavah) [18:06:47] (03Merged) 10jenkins-bot: t3: Ignore oathauth-twofactorauth group [labs/tools/majavah-bot] - 10https://gerrit.wikimedia.org/r/1192944 (owner: 10Majavah) [19:31:35] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124#11234835 (10Dzahn) Done as suggested. The email address has been verified. [19:41:22] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124#11234869 (10itsmoon) Thanks! [19:45:18] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org account verification request: itsmoon - https://phabricator.wikimedia.org/T406124#11234879 (10A_smart_kitten) 05Open→03Resolved a:03Dzahn [20:16:57] 10VPS-project-Wikistats: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#11234966 (10Dzahn) made a new cinder volume `backup-trixie` and attached it to new trixie instance.. trying to fix errors on initial puppet run related to mounting cinder volume and git cloning [23:34:01] 10VPS-project-Wikistats, 13Patch-For-Review: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#11235385 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1192986 [23:34:12] 10VPS-project-Wikistats, 13Patch-For-Review: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#11235386 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1192985 [23:48:31] 10Tool-paulina: Public domain calculator module - https://phabricator.wikimedia.org/T375247#11235398 (10Pepe_piton) I made improvements to the public domain calculation of authors. Previously, Paulina calculated public domain status of authors only based on the author's date of death. Now, it first checks whethe... [23:48:53] 10Tool-paulina: Public domain calculator module - https://phabricator.wikimedia.org/T375247#11235399 (10Pepe_piton) 05Open→03In progress [23:51:30] 10VPS-project-Wikistats, 13Patch-For-Review: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#11235401 (10Dzahn) After the puppet changes above and some manual unmounting / mounting and fixing /etc/fstab on the old instance; puppet runs are now without errors on both in... [23:52:19] 10VPS-project-Wikistats, 13Patch-For-Review: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#11235402 (10Dzahn) https://wikistats-trixie.wmcloud.org/ we will see tomorrow if the timers / jobs ran