[00:35:17] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps hosts to Debian Trixie - https://phabricator.wikimedia.org/T409579#11418062 (10Andrew) Just now I ran into this error during reimage: > > RuntimeError: Host is in BIOS mode but needs to be UEFI as it is connected to a Nokia switch > > Is that right? D... [06:29:57] 06cloud-services-team, 10Toolforge: Can the ruby-dev and libyaml-dev packages be made available on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11418203 (10Ijon) 05Resolved→03Invalid [06:33:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/8 [06:33:43] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/69 [06:33:51] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/69 [06:33:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/150 [06:34:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/141 [06:34:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/9 [06:34:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/141 [06:34:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/150 [06:34:33] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/81 [06:34:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/9 [06:34:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/81 [06:34:53] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/34 [06:35:00] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/70 [06:35:16] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/33 [06:35:18] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/70 [06:35:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/33 [06:35:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/36 [06:35:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/126 [06:35:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/91 [06:35:46] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/151 [06:35:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/36 [06:35:51] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/102 [06:35:53] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/256 [06:35:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/126 [06:36:03] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/37 [06:36:07] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/151 [06:36:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/29 [06:36:09] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/42 [06:36:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/91 [06:36:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/68 [06:36:23] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/256 [06:36:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/151 [06:36:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/85 [06:36:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/102 [06:36:31] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/37 [06:36:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/299 [06:36:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/79 [06:36:43] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/29 [06:36:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/59 [06:36:51] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/59 [06:36:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/42 [06:36:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/68 [06:37:03] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/151 [06:37:07] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/85 [06:37:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/79 [06:37:15] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/299 [06:37:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/59 [06:37:23] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/59 [08:43:28] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b) [08:43:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [08:47:44] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b) [08:47:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [11:41:03] 10Cloud-VPS, 06tools-infrastructure-team: Improve how virt networks are configured in cloudgw - https://phabricator.wikimedia.org/T411081#11418914 (10taavi) p:05Triage→03High [12:30:47] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/24 [12:38:34] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1213447 (owner: 10L10n-bot) [12:52:42] !log taavi@cloudcumin1001 codesearch START - Cookbook wmcs.openstack.quota_increase (T411062) [12:52:46] T411062: request to increase disk/volume quota for codesearch project - https://phabricator.wikimedia.org/T411062 [12:52:49] !log taavi@cloudcumin1001 codesearch END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T411062) [12:53:13] 06cloud-services-team, 10Cloud-VPS (Quota-requests): request to increase disk/volume quota for codesearch project - https://phabricator.wikimedia.org/T411062#11419164 (10taavi) 05Open→03Resolved a:03taavi [12:53:55] 06cloud-services-team, 10Cloud-VPS: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846#11419167 (10taavi) p:05Triage→03High [12:54:17] 06cloud-services-team, 10Cloud-VPS: Add default Cloud VPS project alerts for low disk space and low inode count - https://phabricator.wikimedia.org/T337422#11419170 (10taavi) p:05Triage→03Low [12:54:26] 06cloud-services-team, 10Toolforge: [tofu-provisioning] Allow running manual "tofu state" commands - https://phabricator.wikimedia.org/T410720#11419171 (10taavi) p:05Triage→03Medium [12:54:43] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Support nftables in Cloud VPS - https://phabricator.wikimedia.org/T411089#11419172 (10taavi) p:05Triage→03Medium [12:56:13] 06cloud-services-team, 10Data-Services: wmcs-wikireplica-dns is horribly inefficient - https://phabricator.wikimedia.org/T411192#11419180 (10taavi) p:05Triage→03Medium [12:56:31] 06cloud-services-team, 10Cloud-VPS: Standardize on opentofu management of all projects in the default keystone domain - https://phabricator.wikimedia.org/T410659#11419182 (10taavi) p:05Triage→03Medium [12:56:47] 06cloud-services-team, 10Cloud-VPS: [tofu-infra] Remove vendored openstack provider binary - https://phabricator.wikimedia.org/T411087#11419183 (10taavi) p:05Triage→03Low [12:59:54] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/24 (owner: 10l10n-bot) [13:00:16] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/24 (owner: 10l10n-bot) [13:43:51] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11419322 (10Aklapper) This still seems not to be deployed [14:24:17] 06cloud-services-team, 10Cloud-VPS: wmcs Trixie kernel reboots - https://phabricator.wikimedia.org/T410846#11419506 (10Andrew) The remaining reboots are blocked by VMs that can't be drained. I hope to have that resolved tomorrow when mass reboots are scheduled. [14:29:38] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps hosts to Debian Trixie - https://phabricator.wikimedia.org/T409579#11419519 (10Andrew) >>! In T409579#11418062, @Andrew wrote: > Just now I ran into this error during reimage: > > >> >> RuntimeError: Host is in BIOS mode but needs to be UEFI as it is con... [14:54:35] 06cloud-services-team, 06Wikimedia Enterprise, 10Wikimedia Enterprise Volunteer Request: Toolforge no longer has IP-based access to Wikimedia Enterprise - https://phabricator.wikimedia.org/T410994#11419654 (10HShaikh) Hey folks, I have raised this to our SRE team and they are looking into it. Will keep this... [15:02:47] FIRING: NodeDown: Node cloudweb1004 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:36:26] 06cloud-services-team: Audit and standardize on UTC timezone for grafana.wmcloud.org dashboards - https://phabricator.wikimedia.org/T411274#11419843 (10fgiunchedi) ALSO: make sure grafana's default is UTC on new dashboards [15:51:46] (03open) 10volans: infra-tracing: extend loki retention to 60 days [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1092 (https://phabricator.wikimedia.org/T399313) [15:53:00] (03approved) 10taavi: infra-tracing: extend loki retention to 60 days [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1092 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [15:54:27] (03open) 10hakimi97: Revert "Edit app.js" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/4 [15:55:00] (03approved) 10andrew: infra-tracing: extend loki retention to 60 days [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1092 (https://phabricator.wikimedia.org/T399313) (owner: 10volans) [15:55:11] (03open) 10hakimi97: Revert "Edit index.html" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/5 [15:55:54] (03open) 10hakimi97: Revert "Edit style.css" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/6 [15:57:01] (03merge) 10hakimi97: Revert "Edit app.js" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/4 [15:57:12] (03merge) 10hakimi97: Revert "Edit index.html" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/5 [15:57:27] (03merge) 10hakimi97: Revert "Edit style.css" [toolforge-repos/jawi] - 10https://gitlab.wikimedia.org/toolforge-repos/jawi/-/merge_requests/6 [15:59:06] 10VPS-project-Wikistats: Add tokwiki to wikistats - https://phabricator.wikimedia.org/T404572#11419933 (10Dzahn) a:03Dzahn [16:04:41] !log volans@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing [16:22:02] !log volans@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing [16:25:09] (03open) 10andrew: codfw1dev: Add two projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/284 (https://phabricator.wikimedia.org/T410659) [16:25:13] (03update) 10andrew: codfw1dev: Add two projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/284 (https://phabricator.wikimedia.org/T410659) [16:25:40] (03update) 10andrew: secgroups: open ipv6 access to codfw1dev ntp server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/261 (https://phabricator.wikimedia.org/T408387) [16:25:50] (03update) 10andrew: secgroups: open ipv6 access to codfw1dev ntp server [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/261 (https://phabricator.wikimedia.org/T408387) [16:26:01] (03update) 10andrew: codfw1dev: Add two projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/284 (https://phabricator.wikimedia.org/T410659) [16:26:12] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing [16:27:26] (03CR) 10Andrew Bogott: [C:03+1] "this seems necessary in order to allow codfw1dev to be truly useful for test/dev" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1211722 (https://phabricator.wikimedia.org/T411090) (owner: 10FNegri) [16:46:22] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing [16:55:24] (03CR) 10FNegri: [C:03+2] tofu: Allow running apply on a single cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1211722 (https://phabricator.wikimedia.org/T411090) (owner: 10FNegri) [16:56:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:56:22] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-public-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:59:30] (03Merged) 10jenkins-bot: tofu: Allow running apply on a single cluster [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1211722 (https://phabricator.wikimedia.org/T411090) (owner: 10FNegri) [17:00:55] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 13Patch-For-Review: [tofu-infra] [wmcs-cookbooks] Allow running "tofu apply" on a single cluster - https://phabricator.wikimedia.org/T411090#11420249 (10fnegri) 05In progress→03Resolved This is done, although the alert for codfw is currently not w... [17:01:05] (03merge) 10volans: infra-tracing: extend loki retention to 60 days [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1092 (https://phabricator.wikimedia.org/T399313) [17:01:17] FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:01:22] FIRING: [4x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:01:22] FIRING: [2x] HAProxyServiceUnavailable: HAProxy service keystone-public-api_backend has no available backends on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyServiceUnavailable [17:04:56] FIRING: SystemdUnitDown: The service unit nova-fullstack.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:06:17] FIRING: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:06:22] FIRING: [6x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:06:22] RESOLVED: [3x] HAProxyServiceUnavailable: HAProxy service keystone-admin-api_backend has no available backends on cloudlb1001:9900 - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyServiceUnavailable [17:11:22] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [17:16:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:23:15] FIRING: MetricsinfraMaintainProjectsBroken: metricsinfra failing to update projects - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraMaintainProjectsBroken - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMetricsinfraMaintainProjectsBroken [17:25:26] RESOLVED: SystemdUnitDown: The service unit nova-fullstack.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:28:15] RESOLVED: MetricsinfraMaintainProjectsBroken: metricsinfra failing to update projects - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraMaintainProjectsBroken - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMetricsinfraMaintainProjectsBroken [17:39:47] FIRING: NodeDown: Node cloudweb1003 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudweb1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [17:47:11] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Enable sitenotice on mobile for Wikitech - https://phabricator.wikimedia.org/T410702#11420557 (10bd808) 05Open→03Resolved a:03bd808 {F70805129,size=full} [17:54:51] 10VPS-project-Wikistats: Add tokwiki to wikistats - https://phabricator.wikimedia.org/T404572#11420586 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ('tok','Toki Pona','toki pona',8); ... root@wikistats-bookworm:/home/dzahn# /usr/bin/ph... [18:28:20] (03PS8) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) [18:29:55] 06cloud-services-team, 10Striker, 10CAS-SSO, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#11420828 (10Arendpieter) >>! In T359554#11374327, @taavi wrote: > > This is pending a response to https://gerrit.wikimedia.org/r/c/labs/striker/+/1189915/comme... [18:56:54] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, and 2 others: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11420883 (10bd808) >>! In T407624#11419322, @Aklapper wrote: > This still seems not to be deployed Deplo... [18:59:39] (03CR) 10Majavah: [C:04-1] "How did you produce this patch? Did you use any LLMs or similar tools?" [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [19:15:37] (03PS3) 10Aklapper: Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) [19:19:21] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, and 2 others: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11420973 (10Aklapper) @bd808 Sorry, I did not want to sound pushy, was more of a note to myself. Plus thi... [19:25:48] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, and 2 others: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11420995 (10taavi) >>! In T407624#11420973, @Aklapper wrote: > Plus this likely the patch for T409341 sho... [19:27:51] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, and 2 others: Striker: Always expose observed external URI when creating Diffusion repositories - https://phabricator.wikimedia.org/T407624#11421003 (10Aklapper) Yeah, I'm not sure how / where I'm supposed to test, sorry. [20:30:57] (03PS3) 10Wandji collins: Adding gerrit review to the project [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1212620 (https://phabricator.wikimedia.org/T391419) [20:38:46] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, and 2 others: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#11421226 (10bd808) >>! In T407624#11421003, @Aklapper wrote: > Yeah, I'm not sure how... [21:36:24] 06cloud-services-team, 07Surveys: 2023 Cloud Services Survey - https://phabricator.wikimedia.org/T411419 (10komla) 03NEW [21:51:39] 06cloud-services-team, 07Surveys: 2023 Cloud Services Survey - https://phabricator.wikimedia.org/T411419#11421507 (10komla) [21:56:28] 06cloud-services-team, 07Surveys: 2024 Cloud Services Survey - https://phabricator.wikimedia.org/T411420 (10komla) 03NEW [21:56:41] 06cloud-services-team, 07Surveys: 2023 Cloud Services Survey - https://phabricator.wikimedia.org/T411419#11421533 (10komla) 05Open→03Resolved [21:56:53] 06cloud-services-team, 07Surveys: 2024 Cloud Services Survey - https://phabricator.wikimedia.org/T411420#11421534 (10komla) 05Open→03Resolved [22:01:28] 06cloud-services-team, 07Surveys: 2025 Cloud Services Survey - https://phabricator.wikimedia.org/T411421 (10komla) 03NEW [22:01:39] 06cloud-services-team, 07Surveys: 2024 Cloud Services Survey - https://phabricator.wikimedia.org/T411420#11421560 (10komla) [22:03:39] 06cloud-services-team, 07Surveys: 2023 Cloud Services Survey - https://phabricator.wikimedia.org/T411419#11421561 (10komla) [22:30:37] !log volans@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 (T399313) [22:30:37] !log volans@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 (T399313) [22:30:43] T399313: Add tracing to understand Toolforge and CloudVPS usage and dependencies - https://phabricator.wikimedia.org/T399313 [22:31:20] !log volans@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 (T399313)